Image processing method, device and system

Abstract

The invention provides an image processing method, device and system, the method adopts a specific division mode that an overlapping region exists between adjacent image blocks when an image is divided, and based on the mode, an image processing mode for reducing the calculation scale is adopted when the image blocks are subjected to image processing, so that the image processing efficiency is improved. The processed image blocks obtained by processing are partial regions of the original unprocessed image blocks, and the processed image blocks of the finally obtained multiple image blocks can be spliced into a complete image, namely the processed image. Therefore, according to the image processing method provided by the invention, the calculation scale of image processing (such as high-resolution rendering, anti-aliasing, de-noising and other image processing with relatively high calculation load for changing the image quality characteristics of the image) is reduced, so that the processing can also be realized by computer equipment with relatively low processor calculation power.

Description

Technical Field

[0001] The present application relates to the field of media, and in particular to image processing methods, devices and systems. Background Art

[0002] Image rendering is used in media technology to simulate visual effects such as lighting, shadows, materials, and textures to produce realistic images. Image rendering can involve high-resolution rendering, anti-aliasing, and denoising. These processes require a high computational load and are difficult to implement on computer equipment.

[0003] Currently, the server's GPU can be used to perform high-resolution rendering, anti-aliasing, denoising and other processing based on machine learning models based on machine learning frameworks (such as machine learning frameworks based on Tensorflow, Pytorch, etc.) and computing libraries (such as Compute Unified Device Architecture, abbreviated as CUDA, etc.). This method can reduce the computational load of image rendering. However, since the implementation of the solution depends on the deployment of specific software components (such as machine learning frameworks and computing libraries) on the server, it is difficult to apply widely. In addition, performing high-resolution rendering, anti-aliasing, denoising and other processing based on machine learning models based on machine learning frameworks and computing libraries requires high computing power from the server GPU, while the computing power of the GPUs on some devices (such as mobile phones, laptops and other terminals) is generally low, making it difficult to achieve. Summary of the Invention

[0004] The present application provides an image processing method, device and system that can solve the problem that computer devices with low processor computing power have difficulty in implementing image processing with high computing load (such as high-resolution rendering, anti-aliasing, denoising, etc.).

[0005] In a first aspect, an image processing method is provided, which divides an image to be processed into multiple image blocks, wherein adjacent image blocks in the multiple image blocks have overlapping areas; and then performs image processing on the multiple image blocks, wherein the image processing includes processing for changing image quality characteristics of the image, wherein the image quality characteristics include one or more of resolution, sharpness, noise, and jaggies. After performing image processing on any image block, i.e., a first image block and a second image block adjacent to the first image block, a processed image block of the first image block and a processed image block of the second image block are obtained, wherein the processed image block of the first image block is a partial area in the first image block, and the processed image block of the second image block is a partial area in the second image block; finally, the processed image blocks of the multiple image blocks are spliced ​​to obtain a processed image of the image to be processed.

[0006] The image processing method provided by the present application adopts a specific division method of redundantly dividing the image, that is, dividing adjacent image blocks into overlapping areas. Based on this specific method, when image processing is performed on the image blocks, the processed image blocks obtained are partial areas of the original unprocessed image blocks, that is, partial areas of the original unprocessed image blocks are removed. It can be understood that due to the overlap between adjacent image blocks, the partial areas of the image blocks removed by image processing belong to redundant overlapping areas, so the processed image blocks of the multiple image blocks obtained in the end can still be spliced ​​into a complete image. Therefore, the image processing method of the present application reduces the computational scale of image processing (such as high-resolution rendering, anti-aliasing, denoising, etc., which require a high computational load to change the image quality characteristics of the image), so that computer devices with low processor computing power can also perform complex image processing.

[0007] In one possible implementation, the method inputs a first image block and a second image block adjacent to the first image block into a neural network, and performs image processing on the first image block and the second image block adjacent to the first image block, where the image processing includes convolution processing; wherein the neural network includes one or more convolution layers, and the size of the input image of the convolution layer is larger than the size of the output image of the convolution layer.

[0008] Using neural networks to implement image processing such as high-resolution rendering, anti-aliasing, and denoising can reduce the computational load of these processes, further reducing the computing power requirements of the processors performing these processes. In addition, the neural network used is a convolutional neural network, which includes cascaded convolutional layers. One or more of the convolutional layers in the cascade has an output image size smaller than the input image size. Finally, the image size output by the convolutional neural network is smaller than the image size input to the convolutional neural network, effectively reducing the overall computational scale of the convolutional neural network.

[0009] In another possible implementation, the method is performed by a computer device, and the system of the computer device includes hardware, a hardware interface layer, a framework layer and an application layer, wherein the hardware interface layer is used to provide a hardware and software interface of the hardware, the framework layer is connected to the hardware interface layer, and the framework layer is used to provide a calling interface of the application layer, and the neural network is deployed in the framework layer of the system of the computer device.

[0010] By directly deploying the proposed neural network in the framework layer of the computer device's system, the computer device's processor can smoothly and efficiently perform neural network-based image processing (such as high-resolution rendering, anti-aliasing, denoising, and other image processing that requires a high computational load to change the image quality characteristics of the image) based on the framework layer to obtain the processed image, without the need to additionally deploy machine learning frameworks, computing libraries, etc., thereby reducing the computing power requirements of the computer device's processor.

[0011] In addition, the framework layer of the system of the computer device may include a graphics processing module (such as a graphics synthesizer SurfaceFlinger), and the graphics processing module can be used to process graphics data to obtain an image that can be displayed (such as a rendered image, etc.). For example, a graphics synthesizer can be used to superimpose a first rendered image and a second rendered image to obtain a composite image. A neural network can be deployed in a graphics processing module, such as a graphics synthesizer, so that in the process of the processor of the computer device executing a graphics processing flow based on a graphics processing module such as a graphics synthesizer, it is also implemented to perform image processing based on a neural network, such as performing image processing on a composite image to obtain a processed image. This eliminates the need to create / execute new data processing tasks to achieve high-resolution rendering, anti-aliasing, denoising and other image processing, and the processor has higher processing efficiency.

[0012] In another possible implementation, the data volume of the feature maps output by one or more convolutional layers is smaller than the size of the storage space used to store the output results of the convolutional layers.

[0013] The data volume of the feature map output by the convolution layer is made smaller than (or equal to) the size of the storage space used to store the output results of the convolution layer, so that the intermediate processing results of the convolutional neural network in the image processing process, that is, the feature maps output by each convolution layer, can be successfully matched and stored in different storage spaces (such as the storage space used as a buffer of the processor), which can speed up the data access speed of the entire image processing process and improve the efficiency of image processing.

[0014] In another possible implementation, the data volume of the feature maps output by one or more convolutional layers is determined based on the number of convolution kernels of the convolutional layer, the size of the convolution kernel, and the size of the input image of the convolutional layer.

[0015] By setting or limiting the number of convolution kernels, convolution kernel size, and input image size of the convolution layer, it can be effectively ensured that the amount of data of the feature map output by the convolution layer is smaller than the size of the storage space used to store the output results of the convolution layer.

[0016] In another possible implementation manner, image processing is performed on the first image block and the second image block according to image processing parameters obtained from a plurality of parameters.

[0017] Image processing may include at least one of high-resolution rendering, anti-aliasing, denoising, etc. By having each type of image processing correspond to a parameter, and by directly obtaining the parameters of the required type of image processing from a plurality of parameters, and performing image processing based on the obtained parameters, a variety of different types of image processing can be conveniently implemented, so that the method can be widely applied to a variety of scenarios.

[0018] In another possible implementation, a first rendered image and a second rendered image are obtained, wherein the first rendered image includes an image obtained by low-resolution rendering of the first graphic data, and the second rendered image includes an image obtained by low-resolution rendering of the second graphic data; and the first rendered image and the second rendered image are superimposed to obtain an image to be processed.

[0019] By applying the image processing method provided in this application to the low-resolution image rendering process of a computer device, a high-quality image can be efficiently obtained after at least one of high-resolution rendering, anti-aliasing, and denoising has been performed. Furthermore, in scenarios where rendered images are superimposed, the image processing method provided in this application is configured to perform image processing on the superimposed image after the rendered images are superimposed, further ensuring that a high-quality image can be efficiently obtained after at least one of high-resolution rendering, anti-aliasing, and denoising has been performed in such scenarios.

[0020] In another possible implementation, a neural network training method includes: dividing a sample image into multiple sample image blocks, where adjacent sample image blocks in the multiple sample image blocks have overlapping areas; performing image processing on a first sample image block and a second sample image block adjacent to the first sample image block to obtain a processed image block of the first sample image block and a processed image block of the second sample image block, where the processed image block of the first sample image block is a partial area in the first sample image block, the processed image block of the second sample image block is a partial area in the second sample image block, and the first sample image block is any one of the multiple sample image blocks; splicing the processed image blocks of the multiple sample image blocks to obtain a processed sample image of the sample image to be processed; and adjusting parameters of the neural network according to the processed sample image, the standard image, and a loss function, where the loss function is related to the difference between the processed sample image and the standard image.

[0021] By adopting a specific division method that redundantly partitions the sample image, i.e., one with overlapping areas between adjacent sample image blocks, and based on this specific method, a processing method that reduces the computational scale of the neural network is used when processing the sample image blocks through a neural network. The processed image blocks obtained by processing are partial areas of the original unprocessed sample image blocks, i.e., the partial areas of the original unprocessed sample image blocks are removed. These removed areas are redundant overlapping areas. As a result, the processed image blocks of the multiple sample image blocks obtained can still be spliced ​​into a complete image. As a result, the trained neural network reduces the computational scale of image processing involved in high-resolution rendering, anti-aliasing, denoising, and other image processing, allowing computer devices with lower processor computing power to use the neural network to perform these processes.

[0022] In another possible implementation, the image processing includes at least one of high-resolution rendering, anti-aliasing, and denoising.

[0023] The second aspect provides an image processing method, which is executed by a processing device, including: obtaining at least two rendered images, including a first rendered image obtained by rendering first graphic data and a second rendered image obtained by rendering second graphic data; and running a graphics synthesizer to implement: superimposing the at least two rendered images to obtain a composite image; performing image processing on the composite image through a neural network to obtain a processed image of the composite image, the image processing including at least one of high-resolution rendering, anti-aliasing and denoising; wherein the graphics synthesizer is a program running on the processing device.

[0024] By also deploying a neural network within the graphics compositor, the computer's processor can simultaneously perform high-resolution rendering, anti-aliasing, and denoising on the synthesized image using the neural network, while executing the graphics processing flow based on the graphics compositor. This eliminates the need to create or execute new data processing tasks to achieve high-resolution rendering, anti-aliasing, and denoising, reducing the processor burden and improving processing efficiency.

[0025] In one possible implementation, the neural network includes one or more convolutional layers, and the amount of data of the feature maps output by the one or more convolutional layers is smaller than the size of a buffer used to store the output results of the convolutional layers.

[0026] The neural network used is a convolutional neural network, which includes cascaded convolutional layers, and the data volume of the feature map output by the convolutional layer is smaller than (or equal to) the size of the storage space-buffer used to store the output results of the convolutional layer. This allows the intermediate processing results of the convolutional neural network during image processing, i.e., the feature maps output by each convolutional layer, to be successfully matched and stored in different storage spaces-buffers, respectively. This can speed up the data access speed of the entire image processing process and improve the efficiency of image processing.

[0027] In another possible implementation, the data volume of the feature maps output by one or more convolutional layers is determined based on the number of convolution kernels of the convolutional layer, the size of the convolution kernel, and the size of the input image of the convolutional layer.

[0028] By setting or limiting the number of convolution kernels, convolution kernel size, and input image size of the convolution layer, it can be effectively ensured that the amount of data of the feature map output by the convolution layer is smaller than the size of the storage space used to store the output results of the convolution layer.

[0029] In another possible implementation, the processing device is deployed in the hardware of the computer device, and the system of the computer device includes hardware, a hardware interface layer, a framework layer and an application layer, wherein the hardware interface layer is used to provide a hardware and software interface of the hardware, the framework layer is connected to the hardware interface layer, and the framework layer is used to provide a calling interface of the application layer, and the graphics synthesizer is deployed in the framework layer of the system of the computer device.

[0030] In this way, the processing device of the computer device can smoothly and efficiently perform rendering, rendered image overlay and image processing based on the neural network (such as high-resolution rendering, anti-aliasing, denoising, etc., which require a high computational load to change the image quality characteristics of the image) based on the framework layer, and obtain the processing flow of the processed image without the need to additionally deploy machine learning frameworks, computing libraries, etc., thereby reducing the computing power requirements of the processor of the computer device.

[0031] In a third aspect, an image processing device is provided, including: an image division module, used to divide an image to be processed into multiple image blocks, wherein adjacent image blocks in the multiple image blocks have overlapping areas; an image processing module, used to perform image processing on a first image block and a second image block adjacent to the first image block, to obtain a processed image block of the first image block and a processed image block of the second image block, wherein the processed image block of the first image block is a partial area in the first image block, and the processed image block of the second image block is a partial area in the second image block, wherein the image processing includes processing for changing image quality characteristics of the image, wherein the image quality characteristics include one or more of resolution, sharpness, noise, and jaggies, and the first image block is any one of the multiple image blocks; an image stitching module, used to stitch the processed image blocks of the multiple image blocks to obtain a processed image of the image to be processed.

[0032] In one possible implementation, the image processing module is further used to input the first image block and the second image block adjacent to the first image block into a neural network, and perform image processing on the first image block and the second image block adjacent to the first image block, where the image processing includes convolution processing; wherein the neural network includes one or more convolution layers, and the size of the input image of the convolution layer is larger than the size of the output image of the convolution layer.

[0033] In another possible implementation, the system of a computer device includes hardware, a hardware interface layer, a framework layer, and an application layer, wherein the hardware interface layer is used to provide a hardware and software interface for the hardware, the framework layer is connected to the hardware interface layer, and the framework layer is used to provide a calling interface for the application layer, and the neural network is deployed in the framework layer of the system of the computer device.

[0034] In another possible implementation, the data volume of the feature maps output by one or more convolutional layers is smaller than the size of the storage space used to store the output results of the convolutional layers.

[0035] In another possible implementation, the data volume of the feature maps output by one or more convolutional layers is determined based on the number of convolution kernels of the convolutional layer, the size of the convolution kernel, and the size of the input image of the convolutional layer.

[0036] In another possible implementation manner, the image processing module is further configured to perform image processing on the first image block and the second image block according to image processing parameters obtained from a plurality of parameters.

[0037] In another possible implementation, the image processing module is further used to obtain a first rendered image and a second rendered image, where the first rendered image includes an image obtained by low-resolution rendering of the first graphic data, and the second rendered image includes an image obtained by low-resolution rendering of the second graphic data; and superimpose the first rendered image and the second rendered image to obtain an image to be processed.

[0038] In another possible implementation, the neural network is deployed in the framework layer of the system of the computer device.

[0039] In another possible implementation, the device further includes a neural network training module, which is used to: divide the sample image into multiple sample image blocks, where adjacent sample image blocks in the multiple sample image blocks have overlapping areas; perform image processing on a first sample image block and a second sample image block adjacent to the first sample image block to obtain a processed image block of the first sample image block and a processed image block of the second sample image block, where the processed image block of the first sample image block is a partial area in the first sample image block, the processed image block of the second sample image block is a partial area in the second sample image block, and the first sample image block is any one of the multiple sample image blocks; splice the processed image blocks of the multiple sample image blocks to obtain a processed sample image of the sample image to be processed; and adjust the parameters of the neural network according to the processed sample image, the standard image, and the loss function, where the loss function is related to the difference between the processed sample image and the standard image.

[0040] In another possible implementation, the image processing includes at least one of high-resolution rendering, anti-aliasing, and denoising.

[0041] In a fourth aspect, an image processing device is provided, comprising a graphics synthesizer, the graphics synthesizer comprising a synthesis module and a neural network. The synthesis module is configured to superimpose at least two rendered images to obtain a synthesized image, wherein the at least two rendered images include a first rendered image obtained by rendering first graphics data and a second rendered image obtained by rendering second graphics data; and the neural network is configured to perform image processing on the synthesized image, the image processing including at least one of high-resolution rendering, anti-aliasing, and denoising, to obtain a processed image of the synthesized image.

[0042] In one possible implementation, the neural network includes one or more convolutional layers, and the amount of data of the feature maps output by the one or more convolutional layers is smaller than the size of a buffer used to store the output results of the convolutional layers.

[0043] In another possible implementation, the data volume of the feature maps output by one or more convolutional layers is determined based on the number of convolution kernels of the convolutional layer, the size of the convolution kernel, and the size of the input image of the convolutional layer.

[0044] In another possible implementation, the system of a computer device includes hardware, a hardware interface layer, a framework layer, and an application layer, wherein the hardware interface layer is used to provide a hardware and software interface of the hardware, the framework layer is connected to the hardware interface layer, and the framework layer is used to provide a calling interface of the application layer, and the graphics synthesizer is deployed in the framework layer of the system of the computer device.

[0045] According to a fifth aspect, a processor is provided, wherein the processor is configured to execute the method as described in the first aspect or the second aspect.

[0046] In a sixth aspect, a computer device is provided, comprising a processor and a memory; the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory, so that the computer device executes the method described in the first aspect or the second aspect.

[0047] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed by a computer device, causes the computer device to execute the method as described in the first aspect or the second aspect.

[0048] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0049] The following description includes more details about the implementation methods provided by the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 System architecture of the computer device provided in the embodiment of the present application Figure 1 ;

[0051] Figure 2 System architecture of the computer device provided in the embodiment of the present application Figure 2 ;

[0052] Figure 3 Schematic diagram of the data processing process for image rendering provided in the embodiment of the present application Figure 1 ;

[0053] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of image blocks obtained by division according to an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of the storage space of a computer device provided in an embodiment of the present application;

[0057] Figure 8 Schematic diagram of the data processing process for image rendering provided in the embodiment of the present application Figure 2 ;

[0058] Figure 9 System architecture of the computer device provided in the embodiment of the present application Figure 3 ;

[0059] Figure 10 System architecture of the computer device provided in the embodiment of the present application Figure 4 ;

[0060] Figure 11 A schematic diagram of the training process of a neural network provided in an embodiment of the present application;

[0061] Figure 12 Schematic diagram of the effect of the image to be processed and the processed image provided in the embodiment of the present application;

[0062] Figure 13 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0063] Figure 14 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0064] Figure 15 A schematic diagram of the structure of a computer device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] To facilitate understanding, the terms involved in this application are first introduced.

[0066] Image rendering is the process of converting a 3D model or scene into a 2D image. During the rendering process, a computer device can perform various operations to simulate visual effects such as lighting, shadows, materials, and textures to produce realistic images. For example, an application sends a 3D model or image to a computer device, which then performs a series of processing on the 3D model and image to generate a rendered 2D image.

[0067] High-resolution rendering refers to the image rendering process that produces high-resolution images, which can provide better image clarity and details. In contrast, low-resolution rendering refers to the rendering process that produces lower-resolution images, which produces images with poor clarity and details.

[0068] Anti-aliasing is the process of eliminating or reducing the jagged edges that appear in image rendering. Jagged edges are the jagged patterns that appear in the rendered image, such as those that appear in areas with drastic color changes.

[0069] Image denoising is the process of removing noise from an image. For example, noise around the edges of objects in a rendered image can cause them to appear jagged and rough. Noise refers to unnecessary or redundant interfering information in an image and can include various types, such as salt and pepper noise, speckle noise, and Gaussian noise.

[0070] Image sharpness is an indicator that reflects the clarity of the image plane and the sharpness of the image edges.

[0071] Machine learning frameworks provide scalable computing components for machine learning and can implement various data processing tasks in the field (such as training and / or inference of machine learning models). There are many machine learning frameworks, such as TensorFlow and PyTorch. A machine learning framework can support one or more machine learning models for data processing tasks.

[0072] A computing library is a software library or computing platform built to implement computing tasks. It can be used to provide mathematical computing software to help implement or accelerate computing tasks. For example, Compute Unified Device Architecture (CUDA) is used.

[0073] A graphics processing unit (GPU) is a processor used to perform image and graphics-related computing tasks.

[0074] A buffer (also called a buffer) is a storage space used to store data to buffer data input into or output from memory. For example, when writing data to a disk, the data can be first input into the buffer. When the buffer is full, the data can be written to the disk, which can reduce the number of disk reads and writes. Depending on the type of data stored in the buffer, it can include multiple buffers, such as a primitive buffer (also called a primitive buffer) that can be used to store primitive information, a color buffer (also called a color buffer) that can be used to store pixel color information, a depth buffer (also called a depth buffer) that can store pixel depth information, and a stencil buffer (also called a stencil buffer) that can be used to store pixel stencil value information.

[0075] A machine learning model is a function that can achieve a specific function (mapping) by learning from data. A machine learning model can be used to process input data and generate predictions or inference results.

[0076] Convolutional Neural Network (CNN) is a neural network / model that includes convolution calculations and can be used to analyze visual images.

[0077] The convolution layer is a component of the convolutional neural network and is used to perform convolution operations.

[0078] The convolution layer includes one or more convolution kernels, each of which can perform a convolution operation on the input image of the convolution layer.

[0079] A feature map may refer to an image output after the convolution kernel of a convolution layer convolves the input image of the convolution layer, which contains feature information of the input image.

[0080] An image block refers to a portion of an image area obtained by dividing an image.

[0081] Figure 1 System architecture of the computer device provided in the embodiment of the present application Figure 1 .like Figure 1As shown, the system of a computer device (such as a server 101, a portable computer 102, a handheld terminal device 103 and other terminal devices) can generally include a hardware layer (Hardware Layer) 110, a hardware interface layer (Hardware Abstract Layer) 120, a framework layer (Framework Layer) 130, and an application software layer (Application Layer) 140. Among them, the hardware layer 110 includes the underlying hardware of the computer device, such as a processor (such as a GPU, NPU (Neural Network Processing Unit)), a memory, etc. The hardware interface layer 120 includes an abstract module of the underlying hardware, which is used to provide a hardware and software interface for the operating system, application software, etc. The framework layer 130 includes encapsulated APIs, which can be called to achieve efficient software development and software function implementation. The application software layer 140 includes application software for implementing specific functions, such as browsers, email clients, video players, etc. The framework layer is connected to the hardware interface layer to establish a connection with the hardware, and the framework layer is used to provide a calling interface for the upper application layer.

[0082] Image rendering techniques such as high-resolution rendering, image anti-aliasing, and image denoising can be implemented on a computer device, such as a server. In some embodiments, a processor, such as a GPU, can directly perform high-resolution rendering, anti-aliasing, and denoising on all image pixels. This method has a high computational load and is difficult to implement on a computer device.

[0083] Figure 2 System architecture of the computer device provided in the embodiment of the present application Figure 2 In some embodiments, as Figure 2 As shown, in Figure 2 On this basis, a machine learning framework (e.g., Tensorflow, Pytorch, etc.) 210 and a computing library (e.g., CUDA, etc.) 220 can also be deployed on a computer device such as a server system. After the server obtains the image to be processed, the processor of the computer device, such as a GPU, performs high-resolution rendering, anti-aliasing, denoising, and other processing based on the machine learning model based on the machine learning framework and computing library to obtain a rendered result image.

[0084] Figure 3 Schematic diagram of the data processing process for image rendering provided in the embodiment of the present application Figure 1 .like Figure 3 As shown, more specifically, Figure 2The data processing process of the computer device shown, such as the server, for performing image rendering may include: (the processor of the computer device may include a GPU or other processor capable of performing graphics processing, Figure 3 The GPU is used as an example in the description.)

[0085] In step 310, the GPU performs geometry processing, which includes obtaining graphics data and performing processing such as transformation and rendering on vertex data of the graphics data. In some embodiments, the data required for the GPU to perform the processing in step 310 is stored in the computer device's memory (e.g., random access memory (RAM)), and the result data obtained by the GPU's processing is stored in the computer device's memory (e.g., the first storage space). The GPU performs processing based on data exchange with the computer device's memory, and the data exchange process between the GPU and the computer device's memory is also implemented through a cache (e.g., a cache).

[0086] Step 320: The GPU performs primitive assembly, which includes combining the processed vertices into primitives, such as triangles. In some embodiments, the result data obtained by the GPU processing in step 320 is stored in a buffer (e.g., a primitive buffer).

[0087] At step 330, the GPU performs rasterization, which involves converting primitives into pixels and generating fragments for each pixel (a fragment contains all the data needed to render a pixel). In some embodiments, at step 330, the GPU retrieves the result data from step 320 from a buffer (e.g., a primitive buffer) for processing.

[0088] In step 340, the GPU performs fragment pixel processing (Per-Fragment / Pixel Ops), which may include depth testing, stencil testing, blending, and other processing on the pixels. In some embodiments, part of the data required for the GPU to perform the processing in step 340 is stored in the computer device's memory (e.g., RAM), and the result data obtained by the GPU performing the processing is stored in the computer device's memory (e.g., in the second storage space). The GPU performs the processing based on data exchange with the computer device's memory, and the data exchange process between the GPU and the computer device's memory is also implemented through a cache (e.g., Cache).

[0089] In step 350, the GPU performs fragment processing, which includes calculating the color value of the pixel fragment (which may involve texture mapping and shading), and also includes the GPU performing high-resolution rendering, anti-aliasing, denoising, and other processing based on the machine learning model based on the machine learning framework and computing library, ultimately obtaining a rendered result image. In some embodiments, the portion of data required for the GPU to perform the processing in step 350 is stored in the computer device's memory (e.g., RAM), and the result data obtained by the GPU performing the processing is stored in the computer device's memory (e.g., in a third storage space). The GPU performs processing based on data exchange with the computer device's memory, and the data exchange process between the GPU and the computer device's memory is also implemented through a cache (e.g., a cache).

[0090] The GPU of the computer device performs high-resolution rendering, anti-aliasing, denoising and other processing based on the machine learning model based on the machine learning framework and computing library. Compared with the computer device's GPU directly performing high-resolution rendering, anti-aliasing, denoising and other processing on all image pixels, the computing load is reduced. However, since the machine learning framework and computing library need to be additionally deployed on the computer device, the implementation of the solution relies on specific software components and is difficult to be widely used. In addition, since the GPU requires high computing power to perform high-resolution rendering, anti-aliasing, denoising and other processing based on the machine learning model based on the machine learning framework and computing library, the computing power of the GPU on some devices (such as mobile phones, laptops and other terminal devices) is generally low, making it difficult for devices with low GPU computing power to achieve model-based image rendering.

[0091] In order to solve the problem that the image rendering implementation scheme that requires additional deployment of machine learning frameworks and computing libraries on computer devices relies on specific software components and is difficult to be widely used, and that devices with low GPU computing power (such as mobile phones, laptops and other terminal devices) have difficulty in achieving high-resolution rendering, anti-aliasing, denoising and other image processing, the present application provides an image processing method and system that does not require additional deployment of machine learning frameworks and computing libraries on computer devices. By adopting a specific division method of redundantly dividing the image, that is, there are overlapping areas between adjacent image blocks, a processing method that reduces the computational scale is adopted when performing image processing on the image blocks, thereby reducing the computational scale of high-resolution rendering, anti-aliasing, denoising and other image processing involved in image rendering, so that computer devices with low processor computing power can also achieve these processes to obtain the required result image, that is, the processed image.

[0092] Figure 4 It is a flowchart of the image processing method provided in an embodiment of the present application. Figure 4The method shown can be executed by a processing device such as a processor of a computer device, and the processor can include a GPU or other processor capable of performing graphics processing. Figure 4 As shown, the method includes the following steps.

[0093] Step 410: Divide the image to be processed into a plurality of image blocks, wherein adjacent image blocks in the plurality of image blocks have overlapping areas.

[0094] The image to be processed may be various types of two-dimensional images, such as an image that has been rendered (such as a low-resolution rendering), or other two-dimensional images including scenes / graphics with three-dimensional realistic effects.

[0095] In some embodiments, the image to be processed may have one or more of the following problems: low resolution, low sharpness, aliasing, noise, etc. The image to be processed can be processed by a processor to obtain an image with higher image quality, such as a rendered image with high resolution, improved sharpness, and anti-aliasing and anti-noise.

[0096] The image to be processed may be provided to the processor by application software or other devices or other systems, or may be generated by the processor itself and acquired by the processor.

[0097] In some embodiments, when the image to be processed is an image that has been rendered (such as low-resolution rendering), the image to be processed can be obtained by the following method: a processor such as a GPU obtains graphics data, and the processor renders the graphics data (such as low-resolution rendering) to obtain a rendered image as the image to be processed.

[0098] Graphics data may include various graphics data that can be used for image rendering, such as three-dimensional models and images. Graphics data may be provided to a computer device by application software, other devices, or other systems, or may be generated by the computer device itself for easy access by the computer device.

[0099] For more details on how a processor such as a GPU renders graphics data (e.g., low-resolution rendering) to obtain a rendered image, see Figure 8 、 Figure 9 and its related descriptions.

[0100] In some embodiments, the processor may divide the image to be processed into multiple image blocks by various feasible methods, such as tiling technology. The shape and size of the image block can be set as required, for example, the shape can be rectangular, square, etc., and the size can be 32*32 pixels, 16*16 pixels, etc. In some embodiments, when dividing the image to be processed into image blocks, the image to be processed can be padded as needed (for example, filling the edges), and the padded image can be divided into image blocks so that the processed image obtained by subsequent image processing does not lose the image area of ​​the image to be processed.

[0101] Among the multiple image blocks of the divided image to be processed, any two adjacent image blocks have partial regions overlapping, and the overlapping partial regions can be called overlapping regions, where adjacent can include vertically adjacent, horizontally adjacent, etc.

[0102] As an example, Figure 5 A first image block (which can be any one of the multiple image blocks) among multiple image blocks and a second image block adjacent to the first image block are shown, and the first image block and the second image block partially overlap (i.e., the overlapping area in the figure). Specifically, the first image block includes a first area r1 (a dot-filled portion) and a second area r2 (a portion without a pattern fill), and the second image block includes a third area r3 (a dot-filled portion) and a fourth area r4 (a portion without a pattern fill), wherein r1 and r3 are adjacent, r2 and r1 are adjacent, r4 and r3 are adjacent, r2 and r3 at least partially overlap, and r4 and r1 at least partially overlap. In some embodiments, as Figure 4 As shown, the second region r2 surrounds the first region r1 , and the fourth region r4 surrounds the third region r3 .

[0103] Step 420: Perform image processing on the first image block and a second image block adjacent to the first image block to obtain a processed image block of the first image block and a processed image block of the second image block, where the processed image block of the first image block is a partial area in the first image block, and the processed image block of the second image block is a partial area in the second image block, and the first image block is any one of the multiple image blocks.

[0104] Image processing includes processing for changing image quality characteristics of an image, and image quality characteristics may include one or more of resolution, sharpness, noise, aliasing, etc. In other words, image processing may change one or more of characteristics such as resolution (e.g., increasing or decreasing resolution), sharpness (e.g., increasing or decreasing sharpness), noise (e.g., increasing or eliminating noise), aliasing (e.g., increasing or eliminating aliasing), etc.

[0105] In some embodiments, the image processing may include one of the following processes: high-resolution rendering, anti-aliasing, and denoising, or may further include a superposition of at least two of the processes: high-resolution rendering, anti-aliasing, and denoising.

[0106] In some embodiments, the processor may perform image processing on each image block sequentially or in batches to obtain a processed image block corresponding to each image block. The processor may perform the same image processing on each image block. The following description mainly uses the example of the processor performing image processing on a first image block and a second image block adjacent to the first image block.

[0107] The processor performs image processing on the image block so that: the processed image block obtained after the image block is reduced in part (it can be understood that the size is reduced) compared with the corresponding original image block, that is, the processed image block of the image block is a partial area of ​​the corresponding original image block.

[0108] Taking a first image block and a second image block adjacent to the first image block as an example, in some embodiments, as Figure 5 As shown, the processed image block corresponding to the first image block includes region r1 and no longer includes region r2, and the processed image block corresponding to the second image block includes region r3 and no longer includes region r4. Since regions r1 and r3 are adjacent and do not overlap, it can be concluded that the processed image block corresponding to the first image block and the processed image block corresponding to the second image block have no overlapping areas.

[0109] In some embodiments, the aforementioned multiple image processings correspond to multiple parameters. The processor can determine the parameters corresponding to the image processing, namely the target parameters, from the multiple parameters, and use the target parameters to implement the corresponding image processing.

[0110] Step 430: splice the processed image blocks of the multiple image blocks to obtain a processed image of the image to be processed.

[0111] It is understood that after the processing in step 420, the processed image block corresponding to the first image block and the processed image block corresponding to the second image block can be spliced ​​together, and similarly, the processed image blocks corresponding to other adjacent image blocks can also be spliced ​​together. Thus, after all the processed image blocks are spliced ​​together in step 430, a complete result image can be obtained, and this result image can be the desired processed image.

[0112] Compared with the original image to be processed, the processed image may have improved image quality in at least one of the following aspects: high-resolution rendering effect, elimination or reduction of aliasing, elimination or reduction of noise, etc.

[0113] Figure 12 This is a schematic diagram of the effect of the image to be processed and the processed image provided in the embodiment of the present application. Figure 12 As shown, image A has a low resolution, poor clarity and details, and aliasing (such as the area circled in the circle). After image A is processed by the image processing method provided in this application (superposition of high-resolution rendering and anti-aliasing), image B is obtained. The resolution of image B becomes higher, the clarity and details become better, and the aliasing is removed.

[0114] For example Figure 12 As shown, the resolution of image C is low, the clarity and details are poor, and there is noise at the edge of the image, resulting in unclear edges. After image C is processed by the image processing method provided in this application (superposition of high-resolution rendering and denoising), image D is obtained. The resolution of image D becomes higher, the clarity and details become better, and the noise is removed.

[0115] For example Figure 12 As shown, the resolution of image E is low, and the clarity and details are poor. After image E is processed (high-resolution rendering) by the image processing method provided in this application, image F is obtained, and the resolution of image F becomes higher, and the clarity and details become better.

[0116] In some embodiments, the processor can implement image processing on image blocks based on a model. For example, multiple image blocks can be arranged in sequence to form an image block sequence, which can then be input into the model for processing to obtain processed image blocks for the multiple image blocks. Alternatively, each image block can be input into the model for processing to obtain a processed image for each image block. The model performs the same processing on each image block.

[0117] The model can be implemented by various models / functions that can realize the functions required by this application, for example, the model can include a neural network. This application mainly uses a neural network as an example for explanation.

[0118] In some embodiments, a model such as a neural network can be deployed in a system of a computer device (e.g. Figure 1 The system shown in FIG. 1 ) allows the processor of the computer device to perform image processing based on a model such as a neural network. More details about the method of deploying a model such as a neural network in a system of a computer device can be found in Figure 9 、 Figure 10 Related description.

[0119] In some embodiments, the aforementioned multiple image processings correspond to multiple model parameters. The processor can determine the model parameters corresponding to the image processing, namely the target model parameters, from the multiple model parameters, and implement the corresponding image processing based on the neural network using the target model parameters.

[0120] The model parameters can be obtained by training the neural network and stored in a storage medium. The processor or computer device can obtain the required model parameters from the storage medium. Different image processing corresponds to different training tasks, and thus different model parameters can be obtained. For example, the training task corresponding to high-resolution rendering obtains model parameter P1, the training task corresponding to anti-aliasing obtains model parameter P2, the training task corresponding to denoising obtains model parameter P3, and the training task corresponding to the superposition processing of high-resolution rendering and anti-aliasing obtains model parameter P4, etc. For more details about neural network training, please refer to Figure 10 and its related descriptions.

[0121] The following describes in more detail how to process an image block using a model such as a neural network to obtain a processed image block.

[0122] In some embodiments, the neural network may include a convolutional neural network (CNN). A convolutional neural network may include one or more convolutional layers. It is understood that when image processing is performed on an image block using a convolutional neural network, the image processing is primarily achieved through convolution processing.

[0123] Figure 6 This is a schematic diagram of a neural network provided in an embodiment of the present application. Figure 6 As shown, the neural network includes a convolutional neural network, which includes 3 convolutional layers, and the 3 convolutional layers are a sequentially cascaded architecture.

[0124] In some embodiments, a neural network may include an encoder composed of one or more convolutional layers, a feature extractor composed of one or more convolutional layers, and a decoder composed of one or more convolutional layers. The feature extractor may also be implemented by the convolutional layers of the encoder. The encoder, feature extractor, and decoder form a cascaded pipeline architecture.

[0125] The encoder can be used to process the input image to reduce the spatial dimensions (height, width) of the image while increasing the depth of the image features (i.e. the number of feature channels). The encoder processing helps the network capture more complex, semantic image features.

[0126] A feature extractor can be used to process an input image to identify and extract important features from the image.

[0127] The decoder can be used to process the input image to convert the deep features of the image into higher resolution features, and can also gradually restore the spatial dimensions of the input image through upsampling and convolution operations.

[0128] As an example, Figure 6 As shown in Figure 1, convolutional layer 0 can be used as an encoder, convolutional layer 1 can be used as a feature extractor, and convolutional layer 2 can be used as a decoder.

[0129] It should be noted that in some embodiments, a neural network may not be composed of an encoder, a feature extractor, and a decoder. It may include only one or two of these three components. For example, a neural network may include one or two convolutional layers, and these one or two convolutional layers only constitute one or two of the aforementioned three components. The structure of the neural network, such as the number of convolutional layers and the parameters of the convolutional layers, can be determined based on task requirements (e.g., the image quality of the neural network output image or the requirements of the application scenario).

[0130] For any one of the multiple image blocks, the specific process of the neural network processing the image block may include: when the neural network includes a convolutional layer, processing the image block through the convolutional layer to obtain the processed image block corresponding to the image block. Alternatively, when the neural network includes multiple convolutional layers, the multiple convolutional layers perform processing in order, and the processing performed by the convolutional layer includes processing the input data of the convolutional layer and obtaining a processing result (or output result). The input data of the first convolutional layer to perform processing includes the image block, the input data of the second and subsequent convolutional layers to perform processing include the processing results obtained by the previous convolutional layer to perform processing, and the processing result of the last convolutional layer to perform processing includes the processed image block corresponding to the image block.

[0131] As an example, Figure 6 As shown, the first image block is input into the convolution layer 0 of the neural network for processing and obtains the processing result (i.e., feature map m0), the feature map m0 is further input into the convolution layer 1 for processing and obtains the processing result (i.e., feature map m1), the feature map m1 is further input into the convolution layer 2 for processing and obtains the processing result, which is the processed image block corresponding to the first image block.

[0132] In some embodiments, among the multiple convolutional layers included in the neural network, the feature maps obtained by at least two convolutional layers have different sizes. Moreover, among the at least two convolutional layers, the size of the feature map obtained by the convolutional layer that performs the processing later is smaller than the size of the feature map obtained by the convolutional layer that performs the processing earlier. Therefore, it can be understood that in the process of the neural network processing the image block, the size of the output image (i.e., the generated feature map) will gradually decrease. Alternatively, in some embodiments, among the multiple convolutional layers included in the neural network, the size of the input image of each convolutional layer is larger than the size of the output image of the convolutional layer. Therefore, in the process of the neural network processing the image block, the size of the output image (i.e., the generated feature map) will gradually decrease.

[0133] As an example, Figure 6 As shown, the size of the first image block is 32*32 pixels, the size of the feature map m0 obtained by convolution layer 0 is 24*24 pixels, the size of the feature map obtained by convolution layer 1 is 24*24 pixels, and the size of the feature map obtained by convolution layer 2 is 20*20 pixels. That is, the size of the processed image block corresponding to the first image block is 20*20 pixels. It can be understood that in this embodiment, compared with the first image block, the processed image block corresponding to the first image block removes the region r2 with a width of 6 pixels and only includes the region r1 with a size of 20*20 pixels.

[0134] In some embodiments, the computer device may include one or more storage spaces. In some embodiments, when the computer device includes multiple storage spaces (e.g., multiple buffers), the capacity (or storage space size) of each storage space may be different.

[0135] Figure 7 A schematic diagram of the storage space of a computer device provided in an embodiment of the present application. As an example, Figure 7 As shown, the processor of the computer device includes a storage space 710, and the storage space 710 on the processor includes buffer 0, buffer 1, and buffer 2. Buffer 0 is a color buffer (Color Buffer) with a capacity of 4KB, buffer 1 is a depth buffer (Depth Buffer) with a capacity of 3KB, and buffer 2 is a stencil buffer (Stencil Buffer) with a capacity of 1KB.

[0136] In some embodiments, the processor may store one or more output results obtained by one or more convolutional layers of the neural network in one or more storage spaces of the computer device, such as one or more buffers of the processor. Figure 6 The neural network shown and Figure 7 Taking the memory shown as an example, the feature map m0 obtained by convolution layer 0 can be stored in buffer 0, the feature map m1 obtained by convolution layer 1 can be stored in buffer 1, and the feature map m2 obtained by convolution layer 2 can be stored in buffer 2.

[0137] In the aforementioned embodiment, it can be understood that the data file size (or data volume) of the output result obtained by the convolutional layer of the neural network satisfies the requirement of being less than or equal to the capacity of the corresponding storage space.

[0138] According to the convolution principle of convolutional neural networks, the processing results output by the convolution layer can include one or more feature maps. The number of feature maps output by the convolution layer is determined by the number of convolution kernels in the convolution layer (generally, the number of feature maps equals the number of convolution kernels). The data file size D of the processing results obtained by the convolution layer of the neural network is positively correlated with the data size d of the feature map output by the convolution layer (for example, the data size is 1KB, or 520Byte) and the number of feature maps output by the convolution layer.

[0139] Furthermore, the data size d of the feature map output by the convolutional layer can be calculated based on the image size (OutputShape) of the feature map, and d is positively correlated with the image size of the feature map. Therefore, the data file size of the processing result output by the convolutional layer is positively correlated with the image size of the feature map output by the convolutional layer and the number of convolution kernels in the convolutional layer. In other words, the larger the size of the feature map output by the convolutional layer, the larger the data file size, and the larger the number of convolution kernels in the convolutional layer, the larger the data file size.

[0140] It should be noted that according to the convolution principle of convolutional neural networks, the size of the feature map output by the convolution layer is related to the image size (Input Shape) of the input data of the convolution layer, the parameters of the convolution layer such as the convolution kernel size (Kernel Size), the convolution stride, etc.

[0141] In some embodiments, setting the parameters of the convolution layer, the number of convolution layers and other network parameters according to the convolution principle of the convolutional neural network can achieve the requirements for the neural network in the aforementioned embodiments, for example: the processed image block obtained after the image block is processed by the neural network has a reduced area compared with the corresponding original image block, the feature map sizes output by at least two of the multiple convolution layers included in the neural network are different, the size of the feature map output in the process of the neural network processing the image block gradually decreases, and the data file size of the processing result output by the convolution layer of the neural network meets the requirement of being less than or equal to the capacity of the corresponding storage space.

[0142] For example only, in some embodiments, the network parameters of the neural network satisfy the following constraints: InputShape n -Kernel Size n +1 = Output Shape n , Kernel Number n *d n <=Buffer Size n , InputShape`-2*Guard Size=Output Shape`.

[0143] Among them: Input Shapen Indicates the width / height of the image size of the input data of the nth convolutional layer; KernelSize n Indicates the convolution kernel size of the nth convolution layer; Output Shape n Indicates the width / height of the image size of the processed result output by the nth convolutional layer; Kernel Number n Indicates the number of convolution kernels in the nth convolution layer; d n Indicates the data size of the feature map output by the nth convolutional layer, and d n According to Output Shape n OK; Buffer Size n Indicates the capacity of the buffer corresponding to the processing result output by the nth convolutional layer; Input Shape` indicates the image size (such as width / height) of the image block input to the neural network; Output Shape` indicates the image size (such as width / height) of the processed image block obtained by the neural network; Guard Size indicates the width of the image area removed compared with the image block of the input convolutional layer after the processed image block obtained by the neural network, for example Figure 5 As shown, compared with the first image block, the processed image block corresponding to the first image block removes the area with a Guard Size of 6 pixels surrounding the area r1.

[0144] Table 1 shows the network parameter settings for a neural network. In this embodiment, the neural network includes three convolutional layers: convolutional layer 0, convolutional layer 1, and convolutional layer 2. The image block size input to the neural network (i.e., input to convolutional layer 0) is 32*32 pixels, the guard size is 6 pixels, and the network parameters of the neural network meet the aforementioned constraints.

[0145] Table 1

[0146] Table 2 shows the network parameter settings for another neural network. In this embodiment, the neural network includes two convolutional layers, namely convolutional layer 0 and convolutional layer 1. The image block size input to the neural network (i.e., input to convolutional layer 0) is 32*32 pixels, the guard size is 3 pixels, and the network parameters of the neural network meet the aforementioned constraints.

[0147] Table 2

[0148] Table 3 shows the network parameter settings for another neural network. In this embodiment, the neural network includes one convolutional layer, namely, convolutional layer 0, the image block size input to the neural network (i.e., input to convolutional layer 0) is 32*32 pixels, the guard size is 2 pixels, and the network parameters of the neural network meet the aforementioned constraints.

[0149] Table 3

[0150] Figure 8 Schematic diagram of the data processing process for image rendering provided in the embodiment of the present application Figure 2 . Figure 8 The method shown can be implemented by a processing device such as a processor of a computer device. The processor may include a GPU or other processor capable of graphics processing. Figure 8 The GPU is used as an example for the description.

[0151] In some embodiments, Figure 8 The method shown is Figure 4 The image processing method includes a process of obtaining an image to be processed (i.e., a rendered image) through rendering. Figure 8 The illustrated method includes specific instructions for obtaining a rendered image through a rendering process (eg, low-resolution rendering).

[0152] like Figure 8 As shown, the data processing process of image rendering includes the following steps.

[0153] Step 810: GPU performs geometry processing. Figure 3 The specific content of step 310 is similar, and reference can be made to the relevant description of step 210.

[0154] In step 820, the GPU performs primitive assembly and binning, which includes combining the processed vertices into primitives, such as triangles, determining which image tiles the primitives will appear in, and sorting and storing all primitives based on which image tiles they will appear in. In some embodiments, the resultant data generated by the GPU processing in step 820 is stored in the computer device's memory (e.g., RAM).

[0155] At step 830, the GPU performs rasterization, which includes processing the primitives in each divided image block separately based on the divided image blocks. This processing includes converting the primitives in the image block into pixels and generating fragments for each pixel (a fragment includes all the data required to render a pixel). In some embodiments, at step 830, the GPU retrieves the result data of step 820 from the computer device's memory (e.g., RAM) for processing.

[0156] In step 840, the GPU performs fragment processing, which includes separately processing the pixels of each image block. The processing may include calculating the color values ​​of the pixel fragments in the image block (which may involve texture mapping and shading). Thus, through steps 810-840, the image block can be rendered. All the rendered image blocks can constitute the rendered image (e.g., the rendered image can be obtained by splicing). In some embodiments, part of the data required for the GPU to perform the processing in step 840 is stored in the computer device's memory (e.g., RAM). The GPU performs processing based on data exchange with the computer device's memory, and the data exchange process between the GPU and the computer device's memory is also implemented through a cache (e.g., cache).

[0157] Step 840 also includes the GPU performing image processing (such as high-resolution rendering, anti-aliasing, denoising, etc.) on the rendered image to obtain a plurality of processed image blocks, which can constitute a processed image (such as stitching to obtain a processed image). For details on image processing of the rendered image, please refer to Figure 4 and its related descriptions.

[0158] In some embodiments, the Figure 1 The neural network is deployed in the system of the computer device shown, and the processor of the computer device performs image processing on the rendered image based on the neural network. For more details about the method of deploying a neural network in the system of the computer device, please refer to Figure 9 、 Figure 10 Related description.

[0159] In some embodiments, the result data obtained by the GPU performing the processing in step 840 can be stored through the method of step 850.

[0160] In step 850, the GPU performs a memory operation (also called Tile Load / Store), which includes saving the pixel values ​​obtained in step 840 into the storage space on the processor (also called on-chip storage space) according to the image blocks, and then uniformly storing the pixel values ​​of all image blocks into the memory (e.g., RAM) of the computer device.

[0161] Figure 9 System architecture of the computer device provided in the embodiment of the present application Figure 3 .like Figure 9 As shown in Figure 1 Based on the system architecture of FIG. 1 , the neural network 910 can also be deployed in a system of a computer device such as a terminal device (e.g., Figure 4 and enables the processor of the computer device to implement Figure 4 The image processing method described in Figure 8 The image rendering process described in .

[0162] The neural network 910 can be deployed in the form of a program module, and the functions of the neural network 910 can be implemented when the processor executes the program module.

[0163] In some embodiments, the program module of neural network 910 can be deployed in the software of the computer system. It should be noted that the software here refers to the non-tangible part of the computer system, which can include the programs, documents, and various information required for the development, use, and maintenance of the software in the computer system. For example, the software can include the programs in the application layer 140 (Application Layer), the framework layer 130 (Framework Layer), etc. In some embodiments, the program module of neural network 910 can be deployed in the framework layer 130 (Framework Layer).

[0164] In some embodiments, the software of the computer system (e.g., the framework layer 130 (Framework Layer)) includes a program 920 for implementing graphics processing (e.g., a program for implementing rendering processing), and the program module of the neural network 910 can be included in the program, so that the processor of the computer device, such as a GPU, can implement the graphics processing by executing / running the program 920 for implementing graphics processing. Figure 4 The image processing method described in Figure 8 The image rendering process described in .

[0165] For example, in some embodiments, the program 920 for implementing graphics processing may include a first program and a second program. The first program may be used to obtain graphics data and render the graphics data when a processor of a computer device, such as a GPU, executes the first program to obtain a rendered image. For details on the implementation process of the rendering process, see Figure 8The second program may include a program module of the neural network 910, and the second program may be used to implement image processing of the rendered image by the neural network when the processor of the computer device, such as a GPU, executes the second program to obtain a processed image. For more detailed description of image processing of the rendered image by the neural network, please refer to Figure 8 Step 850 Figure 4 Related description.

[0166] Figure 10 System architecture of the computer device provided in the embodiment of the present application Figure 4 .like Figure 10 As shown, in some embodiments, Figure 9 Based on the system architecture, the application software layer 140 (Application Layer) of the computer device system may include one or more application software, such as a video player, user interface (UI) related application software, a gallery browser, etc. The program 920 for implementing graphics processing may include a graphics synthesizer 1010 (also known as a graphics overlay, such as SurfaceFlinger) for implementing image overlay, and the neural network 910 may be deployed in the graphics synthesizer 1010.

[0167] As an example, a graphics synthesizer (such as SurfaceFlinger) may include a synthesis module 1020 (such as RenderEngine, which is an engine that can be used for graphics processing) for implementing image superposition processing on multiple rendered images to obtain a synthesized image, and may also include a neural network 910.

[0168] In some embodiments, multiple application software can provide graphic data, and it is desired that the graphics or scenes indicated by the multiple graphic data provided by the multiple application software can be reflected in a single image (e.g., a rendered image). As an example, in an actual application scenario, a video player provides image data, and a user provides a 3D model or scene through a related application software of the UI, and the user wishes to display both the provided 3D model or scene and the image data provided by the video player in a single rendered image.

[0169] Alternatively, in some embodiments, an application software may provide multiple graphics data, and these multiple graphics data need to be superimposed and appear in a single image (e.g., a rendered image). As an example, in an actual application scenario, an application provides graphics data for a status bar, a navigation bar, and wallpaper, and needs to display the status bar, navigation bar, and wallpaper on a single rendered image.

[0170] In some embodiments, multiple application software (such as application software a and application software b) of the application software layer (Application Layer) 140 of the computer device can provide graphic data to the processor respectively, or one application software can provide multiple graphic data to the processor. Also, when the processor of the computer device executes the program 920 for implementing graphic processing, it can be achieved that: multiple graphic data (such as first graphic data, second graphic data, etc.) are rendered to obtain multiple rendered images (such as first rendered image, second rendered image, etc.). Then, multiple rendered images are superimposed (for example, two or more rendered images are superimposed) to obtain a superimposed rendered image, that is, a composite image. The superimposed rendered image is further processed by a neural network (such as Figure 4 The GPU may send the processed image to the display for display.

[0171] In some embodiments, the program 920 for implementing graphics processing may also include a program for implementing trapezoidal correction of the rendered image (not shown in the figure). When the processor of the computer device executes the program 920 for implementing graphics processing, in addition to implementing the process described in the previous paragraph, the rendered image after superposition (i.e., the composite image) is further subjected to trapezoidal correction. The rendered image after trapezoidal correction is further subjected to image processing (e.g., image processing) through a neural network. Figure 4 The image processing includes at least one of high-resolution rendering, anti-aliasing, and denoising to obtain a processed image. The trapezoidal correction refers to correcting trapezoidal distortion in the image.

[0172] Figure 11 A schematic diagram of the neural network training process provided in an embodiment of the present application. Figure 11 The neural network described in Figure 4 、 Figure 8 、 Figure 9 、 Figure 10 The neural network used in Figure 11 The training method shown can adjust the model parameters of the neural network, and the adjusted model parameters can be used in the neural network to enable the neural network to achieve Figure 4 、 Figure 8 、 Figure 9 、 Figure 10The functionality of the neural network described in . Figure 11 The method shown can be implemented by a computer device such as a server.

[0173] like Figure 11 As shown, the method includes the following steps.

[0174] Step 1110: Acquire a sample image and a standard image.

[0175] The sample image refers to an image used as a sample, and may be a two-dimensional image containing various graphics / scenes.

[0176] The standard image may be a rendered image with higher image quality, for example, a rendered image having one or more of the following performances: high resolution, high sharpness, no or less aliasing, and no or low noise.

[0177] The sample image may have the same image content as the standard image, but lower image quality than the standard image. The sample image to be processed may have one or more of the following problems: low resolution, low sharpness, jagged edges, noise, etc.

[0178] The standard image can be obtained in various feasible ways, such as obtaining it from a device or system that stores the standard image, or generating it through some equipment that can achieve higher quality rendering (such as Figure 2 Computer equipment through Figure 3 generated using the method shown).

[0179] The sample image can be obtained in various feasible ways, such as downsampling the standard image to reduce the resolution, reduce the sharpness, add aliasing, add noise, etc., to obtain the sample image to be processed, or by generating it through a device that can achieve lower quality rendering (for example, by Figure 1 computer equipment rendering).

[0180] It should be noted that Figure 11 The training method of the proposed method includes using training data (including standard images and sample images to be processed) to adjust the model parameters so that the neural network can realize the image processing. Figure 3 The image processing described above obtains a processed image. As previously mentioned, image processing includes processing for changing image quality characteristics. Image quality characteristics may include one or more of resolution, sharpness, noise, aliasing, etc. For example, image processing may include one of the following processes: high-resolution rendering, anti-aliasing, and denoising, or may also include the superposition of at least two of these processes.

[0181] Therefore, each image processing can correspond to a training task. Figure 11The training method described above uses different image types for training data under different training tasks. For example:

[0182] When the image processing to be implemented in the training task is high-resolution rendering, the standard image is a high-resolution image, and the sample image is a low-resolution image. When the image processing to be implemented in the training task is anti-aliasing, the standard image is an image with no or minimal aliasing, and the sample image is an image with or with significant aliasing. When the image processing to be implemented in the training task is denoising, the standard image is an image with no or minimal noise, and the sample image is an image with or with significant noise. When the image processing to be implemented in the training task is a combination of high-resolution and anti-aliasing, the standard image is an image with high-resolution and no or minimal aliasing, and the sample image is an image with low-resolution and with or with significant aliasing. When the image processing to be implemented in the training task is a combination of high-resolution and denoising, the standard image is an image with high-resolution and no or minimal noise, and the sample image is an image with low-resolution and with or with significant aliasing. When the image processing to be implemented in the training task is the superposition of high resolution, anti-aliasing, and denoising, the standard image is an image with high resolution and no aliasing, or less aliasing, and no noise, or low noise, and the sample image is an image with low resolution and aliasing, or more aliasing, and noisy, or more noise.

[0183] Step 1120: Process the sample image to obtain a processed sample image.

[0184] Processing the sample image includes: dividing the sample image into multiple sample image blocks, where adjacent sample image blocks in the multiple sample image blocks have overlapping areas; performing image processing on a first sample image block and a second sample image block adjacent to the first sample image block to obtain a processed image block of the first sample image block and a processed image block of the second sample image block, where the processed image block of the first sample image block is a partial area in the first sample image block, the processed image block of the second sample image block is a partial area in the second sample image block, and the first sample image block is any one sample image block in the multiple sample image blocks; and splicing the processed image blocks of the multiple sample image blocks to obtain a processed sample image of the sample image to be processed.

[0185] The above process of processing the sample image to obtain the specific process and Figure 4 The process of processing the image to be processed and obtaining the processed image is similar. For more details, please refer to Figure 4 and its related descriptions.

[0186] Step 1130: Adjust the parameters of the neural network according to the processed sample image, the standard image, and the loss function.

[0187] The loss function is related to the difference between the processed sample image and the standard image. In some embodiments, the loss function may include a mean square error (MSE) loss function related to the difference between the processed sample image and the standard image, a similarity loss function (SSIM) loss function related to the similarity between the processed sample image and the standard image, or a combination of one or more of the following.

[0188] The processor can adjust the model parameters according to the loss function through various feasible parameter adjustment methods (such as gradient descent method, grid search method, etc.) to achieve the parameter optimization goal. Among them, the parameter optimization goal may include adjusting the model parameters to minimize the value of the loss function, reduce the value of the loss function to a certain smaller value, or achieve convergence of the loss function.

[0189] according to Figure 11 Different training tasks can obtain different model parameters. Figure 4 For example, the training task corresponding to high-resolution rendering obtains model parameter P1, the training task corresponding to anti-aliasing obtains model parameter P2, the training task corresponding to denoising obtains model parameter P3, and the training task corresponding to the superposition of high-resolution rendering and anti-aliasing obtains model parameter P4, etc. When various model parameters are applied to a neural network, the neural network can implement the corresponding image processing.

[0190] This application also provides an image processing device, such as Figure 13 As shown, the image processing device 1300 includes an image segmentation module 1310, an image processing module 1320, and an image stitching module 1330. The image segmentation module 1310, the image processing module 1320, and the image stitching module 1330 can be used to perform Figure 4 The method steps in .

[0191] The image division module 1310 is used to divide the image to be processed into a plurality of image blocks, where adjacent image blocks in the plurality of image blocks have overlapping areas.

[0192] The image processing module 1320 is used to perform image processing on the first image block and the second image block adjacent to the first image block to obtain a processed image block of the first image block and a processed image block of the second image block, where the processed image block of the first image block is a partial area in the first image block, and the processed image block of the second image block is a partial area in the second image block. The image processing includes at least one of high-resolution rendering, anti-aliasing and denoising, and the first image block is any one of the multiple image blocks.

[0193] The image stitching module 1330 is used to stitch the processed image blocks of multiple image blocks to obtain a processed image of the image to be processed.

[0194] Optionally, the image processing module 1320 is further used to input the first image block and the second image block adjacent to the first image block into a neural network, and perform image processing on the first image block and the second image block adjacent to the first image block, where the image processing includes convolution processing; wherein the neural network includes one or more convolution layers, and the size of the input image of the convolution layer is larger than the size of the output image of the convolution layer.

[0195] Optionally, the data volume of the feature maps output by one or more convolutional layers is smaller than the size of the storage space used to store the output results of the convolutional layers.

[0196] Optionally, the data volume of the feature maps output by one or more convolutional layers is determined based on the number of convolution kernels of the convolutional layer, the size of the convolution kernel, and the size of the input image of the convolutional layer.

[0197] Optionally, the image processing module 1320 is further configured to perform image processing on the first image block and the second image block according to image processing parameters obtained from a plurality of parameters.

[0198] Optionally, the image processing module 1320 is also used to obtain a first rendered image and a second rendered image, where the first rendered image includes an image obtained by low-resolution rendering of the first graphic data, and the second rendered image includes an image obtained by low-resolution rendering of the second graphic data; and superimpose the first rendered image and the second rendered image to obtain an image to be processed.

[0199] Optionally, the neural network is deployed in a framework layer of a system of a computer device.

[0200] Optionally, the image processing device 1300 further includes a neural network training module (not shown in the figure), which is used to: divide the sample image into multiple sample image blocks, where adjacent sample image blocks in the multiple sample image blocks have overlapping areas; perform image processing on a first sample image block and a second sample image block adjacent to the first sample image block to obtain a processed image block of the first sample image block and a processed image block of the second sample image block, where the processed image block of the first sample image block is a partial area in the first sample image block, the processed image block of the second sample image block is a partial area in the second sample image block, and the first sample image block is any one of the multiple sample image blocks; splice the processed image blocks of the multiple sample image blocks to obtain a processed sample image of the sample image to be processed; adjust the parameters of the neural network according to the processed sample image, the standard image and the loss function, where the loss function is related to the difference between the processed sample image and the standard image.

[0201] More details about the method steps implemented by the image segmentation module 1310, the image processing module 1320, and the image stitching module 1330 can be found in Figure 4-11 and its related descriptions.

[0202] Optionally, the image segmentation module 1310 , the image processing module 1320 , and the image stitching module 1330 may each include multiple sub-modules, and the multiple sub-modules may be deployed separately to implement part of the functions of the corresponding modules.

[0203] The apparatus can be implemented by software or hardware.

[0204] As an example of a software functional unit, a module can include a code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the above-mentioned computing device can be one or more. For example, the model updating device can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same AZ or in different AZs, and each AZ includes a data center or multiple data centers with close geographical locations. Generally, a region can include multiple AZs.

[0205] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is located within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0206] As an example of a hardware functional unit, a module can include at least one computing device, such as a server. Alternatively, the image processing device 1300 can be implemented using an ASIC or a PLD. The PLD can be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0207] The multiple computing devices included in the image processing apparatus 1300 can be distributed in the same region or in different regions. The multiple computing devices included in the model updating apparatus can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the YY apparatus can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0208] This application also provides a computer device 1400. Figure 14 As shown, computer device 1400 includes: a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via bus 1402. Computer device 1400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computer device 1400. The processor 1204 of computer device 1400 can be connected to a display via communication interface 1408 (not shown in the figure, the display can include various display devices that can realize display functions, such as plasma displays and liquid crystal displays).

[0209] The bus 1402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 14 The bus 1402 may include a path for transmitting information between various components of the computer device 1400 (eg, the memory 1406, the processor 1404, and the communication interface 1408).

[0210] The processor 1404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0211] The memory 1406 may include volatile memory, such as random access memory (RAM). The processor 1404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0212] The memory 1406 stores executable program codes, and the processor 1404 executes the executable program codes to respectively implement the functions of the aforementioned image segmentation module 1310, the image processing module 1320, and the image stitching module 1330, thereby implementing the image processing method provided in the embodiment of the present application, for example Figure 4 The image processing method provided. That is, the memory 1406 stores instructions for executing the image processing method provided in the embodiment of the present application.

[0213] The communication interface 1408 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computer device 1400 and other devices or a communication network.

[0214] This application also provides a computer device cluster 1500. Figure 15 As shown, the computer device cluster includes at least one computer device 1400. The memory 1406 in one or more computer devices 1400 in the computer device cluster may store the same instructions for executing the image processing method provided in the embodiment of the present application, for example Figure 4 Instructions for the provided image processing methods.

[0215] In some possible implementations, the memory 1406 of one or more computer devices 1400 in the computer device cluster may also store partial instructions for executing the image processing method provided in the embodiment of the present application, such as for executing Figure 4 Instructions for some steps of the image processing method provided. In other words, a combination of one or more computer devices 1400 can jointly execute instructions for executing the image processing method provided in the embodiment of the present application.

[0216] It should be noted that the memory 1406 in different computer devices 1400 in the computer device cluster can store different instructions, each for executing a portion of the functions of the image processing apparatus. In other words, the instructions stored in the memory 1406 in different computer devices 1400 can implement the functions of one or more of the aforementioned image segmentation module 1310, image processing module 1320, and image stitching module 1330.

[0217] In some possible implementations, one or more computer devices in the computer device cluster may be connected via a network, which may be a wide area network or a local area network.

[0218] In some possible implementations, the memory 1406 of one or more computer devices 1400 in the computer device cluster may also store partial instructions for executing the image processing method provided in the embodiment of the present application, such as for executing Figure 4 Instructions for some steps of the image processing method provided. In other words, a combination of one or more computer devices 1400 can jointly execute instructions for executing the image processing method provided in the embodiment of the present application.

[0219] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computer device or stored in any available medium. When the computer program product is run on at least one computer device, the at least one computer device executes the image processing method provided in the present application, for example, for executing Figure 4 Instructions for the provided image processing methods.

[0220] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computer device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computer device to execute the image processing method provided in the present application, for example, to execute Figure 4 Instructions for the provided image processing methods.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

[0222] The terms "first", "second", "third" and "fourth" in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects rather than to limit a specific order.

[0223] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

Claims

1. An image processing method, characterized in that: The method comprises: Dividing the image to be processed into a plurality of image blocks, wherein adjacent image blocks in the plurality of image blocks have overlapping areas; performing image processing on a first image block and a second image block adjacent to the first image block to obtain a processed image block of the first image block and a processed image block of the second image block, wherein the processed image block of the first image block is a partial area in the first image block, and the processed image block of the second image block is a partial area in the second image block, wherein the image processing includes processing for changing image quality characteristics of an image, wherein the image quality characteristics include one or more of resolution, sharpness, noise, and jaggies, and the first image block is any one of the multiple image blocks; The processed image blocks of the multiple image blocks are spliced ​​together to obtain a processed image of the image to be processed.

2. The method according to claim 1, characterized in that The performing image processing on the first image block and the second image block adjacent to the first image block includes: Inputting the first image block and the second image block adjacent to the first image block into a neural network, and performing the image processing on the first image block and the second image block adjacent to the first image block, wherein the image processing includes convolution processing; The neural network includes one or more convolutional layers, and the size of the input image of the convolutional layer is larger than the size of the output image of the convolutional layer.

3. The method according to claim 2, characterized in that The method is performed by a computer device, the system of which includes hardware, a hardware interface layer, a framework layer and an application layer, wherein the hardware interface layer is used to provide a hardware and software interface of the hardware, the framework layer is connected to the hardware interface layer, and the framework layer is used to provide a calling interface for the application layer, and the neural network is deployed in the framework layer.

4. The method according to claim 2, characterized in that The data volume of the feature maps output by the one or more convolutional layers is smaller than the size of the storage space used to store the output results of the convolutional layers.

5. The method according to claim 4, characterized in that The data volume of the feature maps output by the one or more convolutional layers is determined according to the number of convolution kernels of the convolutional layer, the size of the convolution kernel and the size of the input image of the convolutional layer.

6. The method according to any one of claims 1 to 5, characterized in that The performing image processing on the first image block and the second image block adjacent to the first image block includes: The image processing is performed on the first image block and the second image block according to parameters of the image processing acquired from a plurality of parameters.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Acquire a first rendered image and a second rendered image, wherein the first rendered image comprises an image obtained by performing low-resolution rendering processing on the first graphic data, and the second rendered image comprises an image obtained by performing low-resolution rendering processing on the second graphic data; The first rendered image and the second rendered image are superimposed to obtain the image to be processed.

8. The method according to claim 2, characterized in that The training method of the neural network includes: Dividing the sample image into a plurality of sample image blocks, wherein adjacent sample image blocks in the plurality of sample image blocks have overlapping areas; performing the image processing on a first sample image block and a second sample image block adjacent to the first sample image block to obtain a processed image block of the first sample image block and a processed image block of the second sample image block, wherein the processed image block of the first sample image block is a partial area of ​​the first sample image block, the processed image block of the second sample image block is a partial area of ​​the second sample image block, and the first sample image block is any one of the multiple sample image blocks; splicing the processed image blocks of the plurality of sample image blocks to obtain a processed sample image of the sample image; Parameters of the neural network are adjusted based on the processed sample image, the standard image, and a loss function related to the difference between the processed sample image and the standard image.

9. An image processing method, characterized in that: The method is executed by a processing device, and includes: Acquire at least two rendered images, the at least two rendered images comprising a first rendered image obtained by rendering the first graphic data and a second rendered image obtained by rendering the second graphic data; Running a graphics synthesizer implements: superimposing the at least two rendered images to obtain a composite image; performing image processing on the composite image through a neural network to obtain a processed image of the composite image, wherein the image processing includes at least one of high-resolution rendering, anti-aliasing, and denoising; the graphics synthesizer is a program running on the processing device.

10. The method according to claim 9, characterized in that The neural network includes one or more convolutional layers, and the data amount of the feature maps output by the one or more convolutional layers is smaller than the size of a buffer used to store the output results of the convolutional layers.

11. An image processing device, characterized in that: The device comprises: An image division module, configured to divide an image to be processed into a plurality of image blocks, wherein adjacent image blocks in the plurality of image blocks have overlapping areas; an image processing module, configured to perform image processing on a first image block and a second image block adjacent to the first image block to obtain a processed image block of the first image block and a processed image block of the second image block, wherein the processed image block of the first image block is a partial area of ​​the first image block, and the processed image block of the second image block is a partial area of ​​the second image block, wherein the image processing includes processing for changing image quality characteristics of an image, wherein the image quality characteristics include one or more of resolution, sharpness, noise, and jaggies, and the first image block is any one of the multiple image blocks; The image stitching module is used to stitch the processed image blocks of the plurality of image blocks to obtain the processed image of the image to be processed.

12. An image processing device, characterized in that: The apparatus includes a graphic synthesizer, the graphic synthesizer including a synthesis module and a neural network; The synthesis module is used to superimpose at least two rendered images to obtain a synthesized image, wherein the at least two rendered images include a first rendered image obtained by rendering the first graphic data and a second rendered image obtained by rendering the second graphic data; The neural network is used to perform image processing on the composite image to obtain a processed image of the composite image, wherein the image processing includes at least one of high-resolution rendering, anti-aliasing, and denoising.

13. A processor, characterized in that: The processor is configured to execute the method according to any one of claims 1 to 10.

14. A computer device, characterized in that: The computer device includes a processor and a memory; The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory, so that the computer device performs the method according to any one of claims 1 to 10.

15. A computer program product comprising instructions, characterized in that When the instructions are executed by a computer device, the computer device is caused to perform the method according to any one of claims 1 to 10.

Images