Image processing system and method

Abstract

The embodiment of the present specification provides an image processing system, method and storage medium. The system comprises a server and a client; wherein the server comprises a cloud rendering system and a high dynamic range image encoder, the cloud rendering system is used for rendering a three-dimensional design scheme into a two-dimensional first image, and the high dynamic range image encoder is used for compressing and encoding the first image into a second image; the client comprises a high dynamic range image decoder, an image processor and an interaction module, the high dynamic range image decoder is used for decoding the second image into a third image, the image processor is used for obtaining an output fourth image based on the third image and adjustment information of the third image input through the interaction module, and the fourth image is displayed to a user.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of image processing, and in particular, to an image processing system and method. BACKGROUND

[0002] With the rapid development of network technology, image and video processing on the web page has become more and more common. In the existing web page three-dimensional (3D) design system, for the already rendered effect picture, it is usually only possible to adjust the static low-precision image (for example, an image of 8 bits per color channel) based on technologies such as canvas and Cascading Style Sheets (CSS), and to make relatively conventional effect adjustments (for example, brightness, contrast, etc.). When the adjustment range is large, image defects will also be generated. In desktop software systems (for example, desktop software in the film and television industry), high-precision images (for example, an image of 32 bits per color channel) can usually be processed, but the processing of high-precision image or video materials requires professional and expensive hardware devices and complex advanced software systems, and is mainly for professionals in the film industry, which is not suitable for ordinary designers.

[0003] Therefore, it is desirable to provide an image processing system and method to directly process high-precision images and videos on the web page. SUMMARY

[0004] One of the embodiments of the present specification provides an image processing system. The system includes a server and a client; the server includes a cloud rendering system and a high dynamic range image encoder, the cloud rendering system is used to render a three-dimensional design scheme into a two-dimensional first image, and the high dynamic range image encoder is used to compress and encode the first image into a second image; the client includes a high dynamic range image decoder, an image processor and an interaction module, the high dynamic range image decoder is used to decode the second image into a third image, the image processor is used to obtain an output fourth image based on the third image and adjustment information of the third image input through the interaction module, and the fourth image is displayed to a user.

[0005] In some embodiments, the image processor can include a processing operation on at least two layers; for each of the at least two layers, the processing operation includes an adjustment operation on an image, and each of the adjustment operations on the image can be implemented using a separate graphics processor call; the at least two layers can be assembled into a layer stack, and the fourth image can be obtained by inputting the third image into the layer stack for processing.

[0006] In some embodiments, the client can include three buffers for storing the third image, input and output of the current layer among the at least two layers, respectively.

[0007] In some embodiments, the color channel of the input and output of the at least two layers can be 16 bits.

[0008] In some embodiments, the client can be based on a web page implementation, including that the client decodes the image file compressed and encoded by the server into a data format that can be processed by a browser interface; the client includes a decompression algorithm and a decoding algorithm developed using a preset programming language, the decompression algorithm is used to decompress the image file compressed using a preset compression format, and the decoding algorithm is decoded based on a preset precision floating point number, and the client converts the decompression algorithm and the decoding algorithm into a program module that can be directly run by the browser using a preset compilation method.

[0009] In some embodiments, the client can further include a color lookup table decoder, which is used to decode a color lookup table file when the user selects the color lookup table file, and convert a three-dimensional color lookup table in the color lookup table file into a two-dimensional texture format supported by the browser interface; the color lookup table decoder is developed using the preset programming language and is converted into a program module that can be directly run by the browser through the preset compilation method.

[0010] In some embodiments, the adjustment information of the third image input through the interaction module includes at least one of depth of field, noise reduction, glare special effect, fog special effect, exposure, color grading and post-processing.

[0011] One of the embodiments of the present specification provides an image processing method for a server. The method includes: rendering a three-dimensional design scheme into a two-dimensional first image; compressing and encoding the first image into a second image; transmitting the second image to a client, so that the client: decodes the second image into a third image; based on the third image and adjustment information of the third image, obtains an output fourth image and displays it to a user.

[0012] One of the embodiments of the present specification provides an image processing method for a client. The method includes: receiving a second image from a server, the second image being obtained by compressing and encoding a two-dimensional first image, the first image being obtained by rendering a three-dimensional design scheme into two dimensions; decoding the second image into a third image; based on the third image and adjustment information of the third image, obtaining an output fourth image and displaying it to a user.

[0013] One of the embodiments of the present specification provides an image processing system. The system is applied to a client; the client comprises a high dynamic range image decoder, an image processor and an interaction module, the high dynamic range image decoder is used to decode a second image into a third image, the image processor is used to obtain an output fourth image based on the third image and adjustment information of the third image input through the interaction module and show the fourth image to a user; wherein the second image is obtained by compressing and encoding a first image by a high dynamic range image encoder, and the first image is generated by rendering a three-dimensional design scheme by a cloud rendering system.

[0014] One of the embodiments of the present specification provides a computer readable storage medium, the storage medium stores computer instructions, when the computer reads the computer instructions in the storage medium, the computer executes the image processing method. BRIEF DESCRIPTION OF DRAWINGS

[0015] The present specification will be further illustrated in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same numbers represent the same structures, wherein:

[0016] Figure 1 is a schematic diagram of an application scenario of an image processing system according to some embodiments of the present specification;

[0017] Figure 2 is a schematic diagram of an image processing system according to some embodiments of the present specification;

[0018] Figure 3 is an exemplary flowchart of an image processing method according to some embodiments of the present specification;

[0019] Figure 4 is a schematic diagram of an image processing method according to some embodiments of the present specification;

[0020] Figure 5 is a schematic diagram of image rendering based on layers according to some embodiments of the present specification. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some examples or embodiments of the present specification, and for those skilled in the art, the present specification can also be applied to other similar scenarios without creative labor. Unless it is obvious from the language environment or otherwise stated, the same reference numbers in the drawings represent the same structure or operation.

[0022] It should be understood that the terms "system," "device," "unit," and / or "module" as used herein are a method of distinguishing different components, elements, parts, sections, or assemblies from one another. However, the terms can be replaced by other expressions as long as the same meaning is conveyed.

[0023] As indicated in the specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "the," and / or "this" are not limited in scope to the singular, but include the plural as well. In general, the terms "include," "including," and / or "comprising" are intended to be inclusive, but not exclusive, to the steps and elements explicitly mentioned, and that these steps and elements can include other steps or elements not expressly mentioned.

[0024] Flowcharts are used in the specification to illustrate the operations performed by the system according to embodiments of the specification. It should be understood that the preceding or following operations are not necessarily performed in the order shown. Instead, various steps can be processed in reverse order or simultaneously. In addition, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0025] Figure 1 is a schematic diagram of an application scenario of an image processing system according to some embodiments of the specification.

[0026] In some embodiments, the image processing system 100 can be referred to as system 100. As shown in Figure 1 In some embodiments, the system 100 can include a server 110, a storage device 120, a user terminal 130, and a network 140.

[0027] The server 110 can process data and / or information obtained from external devices of the system and / or other components of the system, and perform the image processing method shown in some embodiments of the specification based on the data and / or information to complete one or more functions described in some embodiments of the specification. In some embodiments, the server 110 can perform rendering calculation on a three-dimensional design scheme of a user (e.g., a designer, etc.) obtained from the user terminal 130, and process it into a two-dimensional high dynamic range image (e.g., an image, a video, etc.). In some embodiments, the server 110 can obtain pre-stored data and / or information, such as a three-dimensional design scheme, a pre-set compression algorithm, etc., from the storage device 120, to perform the image processing method shown in some embodiments of the specification, such as rendering calculation, image compression, etc.

[0028] In some embodiments, the server 110 can include rendering components, encoding components, and the like, which can be cloud-based or local. In some embodiments, the server 110 can include one or more sub-processing devices (e.g., single-core processing devices or multi-core multi-threaded processing devices). By way of example only, the server 110 can include a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), a microcontroller unit (MCU), a reduced instruction set computer (RISC), and the like, or any combination thereof.

[0029] The storage device 120 can store data or information generated by other devices. In some embodiments, the storage device 120 can store data and / or information involved in the processing procedures of the server 110 and / or the user terminal 130, such as a user’s three-dimensional design scheme, images before and after encoding and decoding, and the like. In some embodiments, the storage device 120 can include one or more storage components, each of which can be a separate device or a part of other devices. The storage device can be local or implemented through the cloud.

[0030] The user terminal 130 can interact with a user, who can complete specified operations, such as inputting a three-dimensional design scheme, processing images, and the like, through the user terminal 130. In some embodiments, the user terminal 130 can send a three-dimensional design scheme to the server 110 to enable the server 110 to generate a two-dimensional image that the user terminal 130 can process. In some embodiments, the user terminal 130 can include a Web-based image decoder, an image processor, and the like, through which the user terminal 130 can process a two-dimensional image, such as decoding, generating texture resources, and the like. In some embodiments, the user terminal 130 can be one or any combination of a mobile device 130-1, a tablet computer 130-2, a laptop computer 130-3, a desktop computer 130-4, and the like, which have input and / or output functions.

[0031] The network 140 can connect components of the system and / or connect the system with external resource parts. The network 140 enables communication between components and / or between the system and other parts outside the system, facilitating exchange of data and / or information. In some embodiments, one or more components (e.g., the server 110, the storage device 120, the user terminal 130) in the system 100 can send data and / or information to other components through the network 140. In some embodiments, the network 140 can be any one or more of a wired network or a wireless network.

[0032] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of the present specification. Various changes and modifications can be made by those of ordinary skill in the art under the guidance of the present specification. The features, structures, methods and other characteristics of the exemplary embodiments described in the present specification can be combined in various ways to obtain additional and / or alternative exemplary embodiments. For example, the server 110 and the user terminal 130 can have respective storage modules, or can share a cloud storage. However, these changes and modifications will not depart from the scope of the present specification.

[0033] Figure 2 is a schematic diagram of an image processing system according to some embodiments of the present specification.

[0034] As shown in Figure 2 some embodiments, the image processing system 200 can include a server 210 and a client 220. The server 210 can include a cloud rendering system 211 and a high dynamic range image encoder 212, and the client 220 can include a high dynamic range image decoder 221, an image processor 222 and an interaction module 223. In some embodiments, the server 210 can be implemented by the server 110, and the client 220 can be implemented by the user terminal 130.

[0035] In some embodiments, the cloud rendering system 211 can be used to render a three-dimensional design scheme into a two-dimensional first image.

[0036] The three-dimensional design scheme refers to a scheme used by a user (e.g., an advertisement designer, an animation designer, a game designer, etc.) to implement a three-dimensional design, which can be represented by various files, such as project files of three-dimensional design software such as 3DS MAX, MAYA, Blender, etc. In some embodiments, the cloud rendering system 211 can obtain the three-dimensional design scheme in various ways, such as from the client 220.

[0037] The first image is a raw two-dimensional high dynamic range image and / or video generated by rendering a three-dimensional design, wherein the image can include a regular still image, an interactive virtual reality (VR) panorama, etc. High dynamic range (HDR) refers to a greater dynamic range of luminosity than standard digital imaging or photography techniques can reproduce. A channel is a two-dimensional table used to store values between 0 and 255, usually represented by a black and white image, wherein black represents 0 and white represents 255. Most imaging devices use 8-bit imaging per channel, so the image is 256 levels (non-high dynamic range). HDR imaging is suitable for images using more than 8 bits per channel (usually 32-bit floating point values), allowing a greater dynamic range. In this application, high dynamic range refers to 32-bit or 16-bit floating point numbers per channel. High dynamic range image (HDR image) uses a format with more than 256 color levels to store the image, usually 32+32+32 bits are allocated to each pixel to store color information, i.e. a 32-bit floating point number is used to store each primary color. In some embodiments, the first image can include multiple channel information, such as RGB color channels, normal channels, Z-depth channels, velocity field channels, etc. In some embodiments, the first image can be output using a preset high-precision floating point number (e.g. 32-bit floating point number), which can preserve the original color range and improve the quality of the image and / or video. In some embodiments, the image can be represented as a file, so the image can be equivalent to an image file. For example, the first image can include files in.BMP,.PNG,.JPG,.AVI,.MPG,.MP4,.WMV,.MOV, etc.

[0038] In some embodiments, the cloud rendering system 211 can be implemented by a cloud computing platform. By using a cloud computing platform, the hardware and software costs can be reduced while meeting the high processing power requirements, and the flexibility of the system is improved. In some embodiments, the cloud rendering system 211 can also be implemented in other ways, such as by a local distributed processing system or a centralized processing system, etc.

[0039] In some embodiments, the high dynamic range image encoder 212 can be used to compress and encode the first image into a second image.

[0040] The second image is an image obtained by performing a preset processing on the first image. In some embodiments, the preset processing can include at least one of encoding using a preset high-precision floating point number, compressing using a preset compression format, etc. In some embodiments, the high dynamic range image encoder 212 can encode the first image of the preset high-precision floating point number into the image of the preset precision floating point number. In which, the precision of the preset precision floating point number can be lower than the preset high-precision floating point number, for example, the preset high-precision floating point number is a full precision of 32 bits (i.e. IEEE single precision, IEEE refers to Institute of Electrical and Electronics Engineers), and the preset precision floating point number is a half precision of 16 bits (i.e. IEEE half precision). In some embodiments, the high dynamic range image encoder 212 can compress the encoded first image using a preset compression format. In which, the preset compression format can include one or any combination of.zip,.rar,.7z,.tar,.gz, etc. In some embodiments, the second image can include various preset compression format files described above.

[0041] In some embodiments, the high dynamic range image encoder 212 can be implemented in various ways, such as a cloud computing platform, a local distributed processing system, or a centralized processing system, etc.

[0042] In some embodiments of the present specification, by pre-encoding the image of the high-precision floating point number into the image of the lower-precision floating point number on the server side, and further compressing the image, the file size of the image is reduced, the time for the user to obtain the file is reduced, the network resource occupation is reduced, and the burden of local file processing is also reduced, which is convenient for the user to perform real-time image processing and other operations.

[0043] In some embodiments, one or more components in the client 220 can be implemented based on a web page and run in a browser, for example, at least one of the high dynamic range image decoder 221, the image processor 222, and the interaction module 223 can be a web application. In some embodiments, one or more components in the client 220 can be a local application program, for example, a client application program, etc.

[0044] In some embodiments, the high dynamic range image decoder 221 can be used to decode the second image into a third image.

[0045] The third image is an image obtained by performing a preset processing on the second image. In some embodiments, the preset processing performed on the second image can include at least one of decoding, decompression, etc. In some embodiments, the high dynamic range image decoder 221 can decode the image file (e.g., the second image) compressed and encoded by the server 210 into a data format (e.g., the third image) that can be processed by the browser interface. The browser interface can include any browser-based GPU rendering interface and programming language, such as WebGL, WebGPU, etc. The data format can include various media file formats, such as.glTF,.OBJ,.STL,.babylon,.x3d,.JPG,.BMP,.PNG,.MP4, etc. In some embodiments, the high dynamic range image decoder 221 can include decompression algorithms and decoding algorithms developed using a preset programming language. The preset programming language can include C, C++, Java, Rust, etc. In some embodiments, the decompression algorithms can be used to decompress the image file (e.g., the second image) compressed using a preset compression format, and the decoding algorithms can decode the image file using a preset precision floating point number. In some embodiments, the client 220 can convert these decompression algorithms and decoding algorithms into program modules that can be directly run by the browser using a preset compilation method. The preset compilation method can include WebAssembly, JavaScript, etc.

[0046] In some embodiments, the interaction module 223 can be used to interact with the user. In some embodiments, the interaction module 223 can display the image and the operation interface, and receive the operation instruction of the user. In some embodiments, the image processor 222 can be used to obtain the output fourth image based on the third image and the adjustment information of the third image input through the interaction module 223, and display the fourth image to the user.

[0047] The fourth image is an image obtained by adjusting the third image. In some embodiments, the adjustments can include at least one of adjustments to color, contrast, brightness, noise, special effects, etc. In some embodiments, the interaction module 223 can be implemented based on a preset manner, such as based on HTML5 canvas.

[0048] In some embodiments, the user can input adjustment information of the third image through the interaction module 223, which can include at least one of depth of field, noise reduction, lens flare effect, fog effect, exposure, color grading, and post-processing. Among them, the depth of field adjustment can include focus plane and focus range adjustment, etc., and the size and shape (polygon or circle) of the circle of confusion can be set; the noise reduction adjustment can include setting the intensity of noise reduction, etc.; the lens flare effect adjustment can include setting the transparency, size, intensity, shape (e.g., circular or star-shaped), color, etc. of the lens flare; the fog effect adjustment includes setting the color, density, depth range, height range, etc. of the fog; the exposure adjustment includes exposure control type (e.g., Reinhard mode, Exponential mode, etc.), gamma correction value, brightness, highlight range, contrast, dark corner, etc.; the color grading adjustment includes color grading using a color lookup table file (e.g., by selecting a.cube file); the post-processing adjustment includes setting brightness, color scale, color phase, saturation, color temperature, sharpening, etc. In some embodiments, the user can interactively adjust the above parameter information in the interaction module 223, so that the image processor 222 renders the third image based on the adjustment information, and displays the fourth image obtained after processing to the user in real time.

[0049] In some embodiments, the client 220 can further include a color lookup table decoder 224. The color lookup table decoder 224 can be used to decode the color lookup table file when the user selects the color lookup table file, and convert the three-dimensional color lookup table in the color lookup table file into a two-dimensional texture format supported by the browser interface. Among them, the color lookup table file can include.cube,.3dl,.csp,.cms,.lut, etc., and the two-dimensional texture format supported by the browser includes.BMP,.PNG,.JPG, etc. The color lookup table file in this specification is a kind of three-dimensional lookup table (LUT) file, which can map the color of an image to another color. In some embodiments, the color lookup table file can include any one or more of the picture processing templates pre-configured by the designer, the system built-in picture processing templates, etc. The picture processing template is used to adjust the picture or video to the corresponding effect based on the parameters in the template. In some embodiments, the color lookup table decoder 224 can be developed using a preset programming language, and converted into a program module that can be directly run by the browser through a preset compilation method. In some embodiments, after converting the three-dimensional color lookup table into a two-dimensional texture format supported by the browser interface, the image processor 222 can use these two-dimensional texture formats for real-time color grading of the image.

[0050] In some embodiments, the interaction module 223 can also be used for real-time interaction of the user with the panorama and video. Among them, for the panorama, the user can use a mouse, trackball, touch screen, etc. input device to control the rotation, switching of the perspective, stretching, etc. operation of the panorama; for the video, the user can use the input device to control the play, pause, fast forward, fast backward, move to a specific video frame, etc. operation.

[0051] In some embodiments of the present specification, by interactively adjusting the image in multiple types on the webpage side and presenting the adjustment effect in real time, the user (e.g. designer) can master the image adjustment effect in real time, so as to make corrections and adjustments according to the adjustment effect, so as to better achieve the design purpose, and the adjustment effect of the image is better.

[0052] In some embodiments, the image processor 222 can include one or more image processing operations, and each specific operation can be composed of one or more rendering drawing operations, wherein each drawing operation can correspond to a layer (Layer). In some embodiments, the image processor 222 can include processing operations on at least two layers. Among them, for each of these layers, the processing operation can include an image adjustment operation. In some embodiments, each image adjustment operation can be implemented using a separate graphics processor (GPU) call (e.g. draw call). In some embodiments, the layers processed by the image processor 222 can be assembled into a layer stack (Layer Stack), and the fourth image can be obtained by inputting the third image into the layer stack for processing. In the present specification, the entire architecture based on layer rendering drawing operation is referred to as rendering pipeline.

[0053] Figure 5 is a schematic diagram of image rendering based on layers according to some embodiments of the present specification, i.e. a schematic diagram of the rendering pipeline. The following will be described with Figure 5 as an example. As shown in Figure 5 , the layer stack 520 is composed of layers 520-1, 520-2, …, 520-N, wherein N≥2; the input image 510 is the original rendering image, which can be a half-precision floating-point number to avoid precision loss; the output image 530 is the image obtained by processing the input image 510 through the layer stack 520.

[0054] In some embodiments, for each layer, the image adjustment operation can include processing the input image with different image processing algorithms (specific algorithm referred to as layer operation) to obtain the output image. As Figure 5As shown, in some embodiments, the initial output image of each layer can be mixed with the input image again according to a blend mode as the final output of this layer. The blend mode can be configured by the user. In some embodiments, the blend mode can include alpha blending, transparency blending, subtractive blending, additive blending, multiplicative blending, etc.

[0055] In some embodiments, each layer operation can be implemented using a single GPU draw call, and the assembly of the layer stack can be controlled using the CPU code on the web side. This makes it easier to manage and customize the pipeline, ensuring the flexibility of the framework.

[0056] In some embodiments, the output of each layer can be used as the input of the next layer until the entire layer stack is processed and the final output image is obtained. For example, the output of layer 520-1 can be used as the input of layer 520-2, and the final output image 530 is the output of the last layer 520-N of the layer stack 520.

[0057] In some embodiments of the present specification, the processing efficiency and flexibility of the processing framework are improved by including different image adjustment algorithms in the graphics processor.

[0058] Since the processing devices used by users, such as computers, often do not have high hardware configurations, in order to reduce the video memory overhead caused by high-resolution and high-precision image processing pipelines, the precision of texture resources can be reduced. However, traditional low dynamic range (e.g., 8 bits per color channel) image precision is insufficient, and when the adjustment range is large, banding artifacts (e.g., jagged, striped, etc.) can occur. In some embodiments, for the input and output of layers in the image processor 222 and the intermediate texture resources, the client 220 can encode them into a format of pre-set precision floating point numbers (e.g., IEEE half-precision floating point numbers, i.e., 16 bits per color channel). In this way, even after multiple iterations, there will be no significant loss of precision, and no visible banding artifacts, solving the problems of precision loss and banding artifacts in traditional low dynamic range (e.g., 8 bits per color channel) image processing.

[0059] In some embodiments, since a layer-based processing method is used, a Ping-Pong buffer can be used to manage the buffers uniformly in the image processor 222. In some embodiments, the image processor 222 can include three buffers for storing the third image, the input and output of the current layer in the at least two layers. The buffers can include texture buffers of a graphics processor (GPU), buffers in memory, etc. In this way, the video memory overhead of the graphics processor can be effectively reduced.

[0060] For each layer, since there is only one input image and one output image, only two buffers are actually needed. For example, Figure 5 In some embodiments, assuming that the buffers for the layer stack 520 are buffer A and B, for the layer 520-1, buffer A can be used as the input buffer and buffer B as the output buffer; for the layer 520-2, buffer B can be used as the input buffer and buffer A as the output buffer, and the rest of the buffers follow the same pattern. In some embodiments, in order to directly display the input image (e.g., the third image) of a layer in the system, the image processor 222 can include a buffer for displaying the input image of the layer. In some embodiments, when the image size of the input image and / or the output image changes, the image processor 222 can adjust the size of the buffer. For example, the image size is 4K at the maximum and 1K at the minimum, and the buffer can be set to 4K. In some embodiments, when the image size exceeds the preset buffer size (e.g., 4K), the image processor 222 can increase the buffer size according to the image size, wherein the image processor 222 can split the original image into smaller tiles (e.g., 4K, 6K, etc.).

[0061] In some embodiments, in the image processor 222, each layer can include a separate output buffer. On this basis, if the user only adjusts the parameters of a certain layer, the image processor 222 can reuse the old intermediate result, recalculate the layer operations of this layer and the layers after it, and the layer operations of the layers before this layer are not re-executed, wherein the intermediate result can be saved in the separate output buffer of the previous layer corresponding to this layer. In this way, the time and resource overhead of reprocessing are saved, the processing speed of adjustment is accelerated, and the image rendering processing efficiency is improved.

[0062] In some embodiments, for each user, the image processing system 200 can generate one or more preset image adjustment schemes of the user based on the historical operations of the user, and the adjustment of the third image can be performed based on the preset image adjustment schemes. In some embodiments, the preset image adjustment scheme can be obtained by learning the historical operation data of the user through a machine learning model.

[0063] Figure 3 is an exemplary flowchart of an image processing method according to some embodiments of the present specification.

[0064] As shown in Figure 3 , the flow 300 includes the following steps. In some embodiments, the flow 300 can be performed by the image processing system 200, wherein steps 310-330 can be performed by the server 210 and steps 340-350 can be performed by the client 220.

[0065] Step 310, render the three-dimensional design scheme into a two-dimensional first image.

[0066] In some embodiments, the server 210 can render the three-dimensional design scheme of a user (e.g., a designer, etc.) into a two-dimensional first image through the cloud rendering system 211. The first image can include a static image, a video, an interactive VR panorama, etc. For more information on how to render a three-dimensional design scheme into a two-dimensional first image, see the description of the cloud rendering system 211 in Figure 2 , which will not be repeated here.

[0067] Step 320, compress and encode the first image into a second image.

[0068] In some embodiments, the server 210 can compress and encode the first image through the high dynamic range image encoder 212 to obtain the second image. The second image can include various compressed file formats. In some embodiments, the high dynamic range image encoder 212 can encode the first image using a preset precision floating point number, and then compress the encoded first image using a preset compression format to obtain the second image. For more information on how to compress and encode the first image into the second image, see the description of the high dynamic range image encoder 212 in Figure 2 , which will not be repeated here.

[0069] Step 330, transmit the second image to the client.

[0070] In some embodiments, the server 210 can transmit the second image to the client 220 for image processing. After generating the second image, the server 210 can actively send the second image to the client 220, or the client 220 can send a request to the server 210 to obtain the second image.

[0071] Step 340, decode the second image into a third image.

[0072] In some embodiments, after receiving the second image sent by the server 210, the client 220 can use the decompression algorithm and decoding algorithm developed by the preset programming language to decode the second image through the high dynamic range image decoder 221, thereby obtaining the third image. The third image can include various data formats that can be processed by a browser interface. For more information on how to decode the second image into the third image, see the description of the high dynamic range image decoder 221 in Figure 2 , which will not be repeated here.

[0073] At step 350, the fourth image is obtained based on the third image and the adjustment information of the third image, and is displayed to the user.

[0074] In some embodiments, the client 220 can display the third image to the user through the interaction module 223, and receive the adjustment information of the third image from the user. In some embodiments, the client 220 can render the third image according to the adjustment information of the third image through the image processor 222, and obtain the output image as the fourth image and display it to the user. For more information about how to obtain the output fourth image based on the third image and the adjustment information of the third image, and display it to the user, please refer to the description of the image processor 222, the interaction module 223 and the color grading lookup table decoder 224 in Figure 2

[0075] In some embodiments of the present specification, the three-dimensional design scheme is rendered into a two-dimensional high dynamic image resource on the server, and the high-precision images and videos are directly adjusted and processed on the client, so that the user can directly perform advanced post-processing (such as advanced color grading, adding various special effects, etc.) on the images such as static images, dynamic videos, interactive panoramic maps (VR) and the like on the webpage. This facilitates user operation, reduces the hardware and software costs of user operation, and at the same time, improves the adjustment range of the image and the quality of the processed image.

[0076] Figure 4 is a schematic diagram of an image processing method according to some embodiments of the present specification.

[0077] In some embodiments, Figure 4 The flow 400 shown in the figure can be executed by the image processing system 200.

[0078] As shown in the figure, Figure 4 The server 210 can render the three-dimensional design scheme 410 through the cloud rendering system 415 to obtain a two-dimensional image 420, compress and encode the two-dimensional image 420 through the server high dynamic range image encoder 425 to obtain a compressed and encoded image 430, and then send the compressed and encoded image 430 to the client 220. The cloud rendering system 211 can include the cloud rendering system 415, the first image can include the two-dimensional image 420, the high dynamic range image encoder 212 can include the server high dynamic range image encoder 425, and the second image can include the compressed and encoded image 430.

[0079] ​The client 220 can decode the compressed image 430 through the web-based high dynamic range image decoder 435 to obtain a decoded image 440, and render and adjust the decoded image 440 through the real-time image processor 445 to obtain an adjusted image 470. The high dynamic range image decoder 221 can include the web-based high dynamic range image decoder 435, the third image can include the decoded image 440, the image processor 222 can include the real-time image processor 445, and the fourth image can include the adjusted image 470.

[0080] The real-time image processor 445 can perform rendering on the decoded image 440 based on the adjustment information 450. The user (e.g., a designer, etc.) can input the adjustment information 450 through the interactive color adjustment terminal 460, the real-time image processor 445 can perform real-time re-rendering on the decoded image 440 based on the adjustment information 450, and the real-time image processor 445 can display the re-rendered image to the user through the interactive color adjustment terminal 460. If the user is not satisfied with the re-rendered image, the user can input new adjustment information 450, the real-time image processor 445 can perform re-rendering on the decoded image 440 based on the new adjustment information 450, and the real-time image processor 445 can output the re-rendered image as the adjusted image 470. In this process, the color lookup table decoder 455 can decode the color lookup table file (e.g., a.cube file) selected by the user and convert the three-dimensional color lookup table in the file into a two-dimensional texture format (e.g.,.BMP,.PNG,.JPG, etc.) supported by the browser interface (e.g., Web GL). The interactive module 223 can include the interactive color adjustment terminal 460, and the adjustment information of the third image can include the adjustment information 450. The color lookup table decoder 224 can include the color lookup table decoder 455.

[0081] It should be noted that the above description of the processes 300 and 400 is merely for example and illustration, and does not limit the scope of the present specification. Those skilled in the art can make various modifications and changes to the processes 300 and 400 under the guidance of the present specification. However, these modifications and changes are still within the scope of the present specification. For example, the compression operation in step 320 can be omitted, and the decompression operation in step 340 can be omitted to save processing time.

[0082] The beneficial effects that the embodiments of the present specification can bring include but are not limited to: (1) by rendering a three-dimensional design scheme into a two-dimensional high dynamic image resource on the server side, and directly adjusting and processing these high-precision images and videos on the client side, so that users (such as designers, etc.) can directly adjust the static images, dynamic videos, interactive panoramic maps (VR) and other images on the webpage, similar to the advanced post-processing in the film industry (such as advanced color grading, adding various special effects, etc.), which facilitates user operation, reduces the hardware and software costs of user operation, and at the same time improves the adjustment range of the image and the quality of the processed image; (2) by using a cloud computing platform for three-dimensional to two-dimensional operation, the hardware and software costs can be reduced while meeting the high processing capacity requirements, and the flexibility of the system is improved; (3) by pre-encoding and compressing high-precision floating-point images on the server side, the file size of the image is reduced, the time for users to obtain the file is reduced, the network resource occupation is reduced, and the burden of local file processing is also reduced, which is convenient for users to perform real-time image processing and other operations; (4) by interactively adjusting the image on the webpage in multiple types and presenting the adjustment effect in real time, the user can master the image adjustment effect in real time, so as to correct and adjust according to the adjustment effect, so as to better achieve the design purpose and the image adjustment effect is better; (5) by including different image adjustment algorithms in the graphics processor, the processing efficiency and the flexibility of the processing framework are improved; (6) by encoding high-precision floating-point texture resources into a lower-precision floating-point format, the precision loss and banding defects in traditional low dynamic range image processing are solved. It should be noted that different embodiments can have different beneficial effects, and in different embodiments, the beneficial effects that can be produced can be any one or a combination of several of the above, or any other beneficial effects that can be obtained.

[0083] The above has described the basic concepts, and it is obvious that the above detailed disclosure is only as an example and does not constitute a limitation on the present specification. Although it is not explicitly stated here, those skilled in the art can make various modifications, improvements and corrections to the present specification. Such modifications, improvements and corrections are suggested in the present specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present specification.

[0084] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0085] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0086] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0087] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0088] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0089] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. An image processing system, comprising: a server and a client; the server comprises a cloud rendering system and a high dynamic range image encoder, the cloud rendering system is configured to render a three-dimensional design into a two-dimensional first image, and the high dynamic range image encoder is configured to compress and encode the first image into a second image; the client comprises a high dynamic range image decoder, an image processor and an interaction module, the high dynamic range image decoder is configured to decode the second image into a third image, the image processor is configured to perform rendering processing based on the third image and adjustment information of the third image input through the interaction module, to obtain an output fourth image and display the output fourth image to a user; the image processor comprises a plurality of image adjustment operations, each of the adjustment operations corresponds to a layer, and an output of each of the layers is used as an input of a next layer; the rendering processing is performed based on a layer stack composed of the layers to obtain the output fourth image.

2. The system of claim 1, wherein: each of the adjustment operations is implemented using a single graphics processor call.

3. The system of claim 2, wherein, the image processor comprises processing operations performed on at least two layers, and the client comprises three buffers for storing the third image, an input of a current layer of the at least two layers and an output of the current layer, respectively.

4. The system of claim 3, wherein, color channels of the input and the output of the at least two layers are 16 bits.

5. The system of claim 1, wherein, the client is implemented based on a webpage, and comprises: the client decodes the image file compressed and encoded by the server into a data format that can be processed by a browser interface; the client comprises a decompression algorithm and a decoding algorithm developed using a preset programming language, the decompression algorithm is configured to decompress the image file compressed using a preset compression format, the decoding algorithm is configured to decode based on a preset precision floating point number, and the client converts the decompression algorithm and the decoding algorithm into program modules that can be directly run by the browser using a preset compilation method.

6. The system of claim 5, wherein, the client further comprises a color lookup table decoder, the color lookup table decoder is configured to decode a color lookup table file when the color lookup table file is selected by a user, and convert a three-dimensional color lookup table in the color lookup table file into a two-dimensional texture format supported by the browser interface; the color lookup table decoder is developed using the preset programming language and converted into program modules that can be directly run by the browser using the preset compilation method.

7. The system of claim 1, wherein, the adjustment information of the third image input through the interaction module comprises at least one of a depth of field, noise reduction, glare special effect, fog special effect, exposure, color grading and post-processing.

8. An image processing method for a server, comprising: the method comprises: rendering a three-dimensional design into a two-dimensional first image; compressing and encoding the first image into a second image; transmitting the second image to a client, so that the client: decodes the second image into a third image; The third image is rendered based on the third image and adjustment information of the third image, to obtain an output fourth image and display the fourth image to a user; the adjustment information includes multiple image adjustment operations, each adjustment operation corresponding to a layer, and an output of each layer serving as an input of a next layer; the rendering is performed based on the layers to form a layer stack, to obtain the output fourth image.

9. An image processing method for a client, comprising: The method includes: receiving a second image from a server, the second image being obtained by compressing and encoding a two-dimensional first image, the first image being obtained by rendering a three-dimensional design scheme into two dimensions; decoding the second image into a third image; The third image is rendered based on the third image and adjustment information of the third image, to obtain an output fourth image and display the fourth image to a user; the adjustment information includes multiple image adjustment operations, each adjustment operation corresponding to a layer, and an output of each layer serving as an input of a next layer; the rendering is performed based on the layers to form a layer stack, to obtain the output fourth image.

10. An image processing system, characterized by: The system is applied to a client; The client includes a high dynamic range image decoder, an image processor, and an interaction module; the high dynamic range image decoder is configured to decode a second image into a third image; the image processor is configured to render the third image based on the third image and adjustment information of the third image input through the interaction module, to obtain an output fourth image and display the fourth image to a user; wherein the second image is obtained by compressing and encoding a first image by a high dynamic range image encoder, and the first image is generated by rendering a three-dimensional design scheme by a cloud rendering system; the adjustment information includes multiple image adjustment operations, each adjustment operation corresponding to a layer, and an output of each layer serving as an input of a next layer; the rendering is performed based on the layers to form a layer stack, to obtain the output fourth image.

Images