Image processing method, apparatus and system and storage medium

By dividing the image into multiple pixel blocks and determining the sub-code rate based on its area and brightness variance values, the problem of low image compression quality in cloud image rendering is solved, and high-quality image transmission under limited channel bandwidth is achieved and Optimization of computing power.

WO2024245004A1PCT designated stage expired Publication Date: 2024-12-05SHENZHEN HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/093850
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-17
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In cloud image rendering, since each area of ​​the image is compressed at the same or similar code rate, the compressed image quality is low, which wastes server computing power and is difficult to meet user needs.

Method used

By dividing the image into multiple pixel blocks, the sub-code rate is determined according to the area and number of pixel points of each block, and the picture quality level is divided according to the brightness variance value, and each area is based on the target bit rate and picture quality level. Blocks allocate different bit counts for compression.

Benefits of technology

In the case of limited channel bandwidth, the compressed image quality is improved, the server's computing power waste during rendering images is reduced, and the user's usage needs are met.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024093850_05122024_PF_FP_ABST
    Figure CN2024093850_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of video coding. Provided are an image processing method, apparatus and system and a storage medium. In the present application, a server acquires a first image, then divides the first image into a plurality of pixel blocks, determines a sub-code rate of each pixel block, and finally compresses the first image according to the plurality of sub-code rates of the plurality of pixel blocks, each pixel block comprising a plurality of pixels in the first image, and the sub-code rate of each pixel block being determined according to the area of each pixel block and the number of pixels comprised in each pixel block. Under the condition of a limited channel bandwidth between a server and a terminal device, that is a fixed target code rate of the first image, the present application can determine sub-code rates of different pixel blocks of the first image according to pixels comprised in each pixel block, thereby improving the image quality of the compressed first image, and reducing the waste of computation capability of the server during rendering of the first image.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing method, device, system and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 26, 2023, with application number 202310611801.0 and application name “Image processing method, device, system and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of video coding technology, and in particular to an image processing method, device, system and storage medium. Background Art

[0003] Cloud 3D rendering applications (cloud 3D) refers to the process of rendering an image using a rendering application deployed on a server, which then distributes the bitrate and compresses the image before sending it to the user's terminal device. Bitrate allocation refers to allocating different numbers of bits to different areas of the image. In order to adapt to the limited channel bandwidth between the server and the terminal device, the server usually compresses the rendered image using lossy compression. However, since each area of ​​the image is compressed at the same or similar bitrate, the quality of each area in the compressed image is lower than the quality of the rendered image, resulting in a waste of computing power consumed by the server in the process of rendering the image, and the image quality after lossy compression is poor, which is difficult to meet the user's usage needs.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, device, system, and storage medium, which can reduce the waste of computing power consumed in rendering images and improve the quality of compressed images.

[0006] To achieve the above-mentioned purpose, in a first aspect, an embodiment of the present application provides an image processing method, comprising: acquiring a first image; then dividing the first image into a plurality of pixel blocks, and determining the sub-code rate of each pixel block. Finally, compressing the first image according to the multiple sub-code rates of the plurality of pixel blocks. Each pixel block includes a plurality of pixel points in the first image, and the sub-code rate of each pixel block is determined according to the area of ​​each pixel block and the number of pixel points included in each pixel block. The method provided by the present application can determine the sub-code rate of different pixel blocks of the first image according to the area and the number of pixel points included in each pixel block when the channel bandwidth between the server and the terminal device is limited, that is, when the target code rate of the first image is fixed, thereby improving the image quality of the compressed first image and reducing the waste of computing power of the server in the process of rendering the first image.

[0007] In a possible implementation of the first aspect, determining the sub-code rate for each pixel block includes: determining the image quality level of each pixel block based on the area of ​​each pixel block and the number of pixels included in each pixel block, the image quality level being used to characterize the degree of discreteness between the brightness values ​​corresponding to multiple pixels in each pixel block; and determining the sub-code rate for each pixel block based on the target bit rate of the first image and the image quality level of each pixel block. The method provided in the present application determines the image quality level of each pixel block by the degree of discreteness between the brightness values ​​corresponding to multiple pixels in each pixel block, and allocates a different number of bits to each pixel block based on the image quality level and target bit rate of the first image, thereby improving the image quality of the compressed first image and reducing the waste of computing power of the server in rendering the first image when the channel bandwidth between the server and the terminal device is limited.

[0008] In one possible implementation of the first aspect, determining the image quality level of each pixel block includes: determining a luminance value for each pixel in a first image; determining a luminance variance value for a plurality of pixels included in each pixel block; and determining the image quality level of each pixel block based on the luminance variance value. The method provided herein determines the luminance variance value for each pixel block in the first image, thereby determining the image quality level of each pixel block based on the luminance variance value, thereby determining the sub-rate of each pixel block in the first image based on the image quality level of each pixel block.

[0009] In one possible implementation of the first aspect, the image quality level of each pixel block includes a first level, a second level, or a third level; the first level indicates that the brightness variance value of each pixel block is less than or equal to a first threshold; the second level indicates that the brightness variance value of each pixel block is greater than the first threshold and less than a second threshold; the second threshold is greater than the first threshold; and the third level indicates that the brightness variance value of each pixel block is greater than or equal to the second threshold. The method provided in this application sets a first threshold and a second threshold, and determines multiple image quality levels based on the first and second thresholds, thereby achieving a more fine-grained division of image quality levels for pixel blocks, thereby allocating a bit rate to each pixel block based on different image quality levels.

[0010] In a possible implementation of the first aspect, different image quality levels are associated with different bit allocation weights, the image quality levels are positively correlated with the bit allocation weights, and a sub-rate for each pixel block is determined based on a target bit rate of the first image and the image quality level of each pixel block, including: determining a bit allocation weight corresponding to each pixel block in the first image based on the image quality level of each pixel block; and determining the sub-rate for each pixel block based on the target bit rate of the first image and the bit allocation weight corresponding to each pixel block. The method provided in this application configures different bit allocation weights for different image quality levels, thereby determining the sub-rate for each pixel block based on the image quality level of each pixel block in each frame of the image.

[0011] In one possible implementation of the first aspect, obtaining a first image to be processed includes: obtaining multiple source image frames to be processed; rendering the multiple source image frames to obtain the rendered multiple image frames; the rendering method includes executing at least one of a rendering process of a rendering application or artificial intelligence generated content (AI generated content, AIGC), and the rendered multiple image frames include the first image. The method provided in this application can render the first image in a variety of ways, meeting user needs in different usage scenarios.

[0012] In a possible implementation of the first aspect, a multi-frame source image includes a first source image and a second source image, the first source image corresponds to the first image, the second source image is adjacent to the first source image, and the timestamp of the second source image is after the timestamp of the first source image; the multi-frame source image is rendered to obtain the rendered multi-frame image, including: determining the rendering method of each pixel block in the second source image according to the image quality level of each pixel block in the first image; different image quality levels are configured with corresponding rendering methods, and the image quality level is positively correlated with the rendering accuracy of the rendering method corresponding to the image quality level; rendering the second source image according to the rendering method of each pixel block in the second source image to obtain the second image. The method provided in the present application determines the rendering method of each pixel block on the second image based on the image quality level corresponding to each pixel block of the first image during the image rendering process, and finally renders the second image. The coordination of the image rendering process and the image compression process is achieved, that is, based on the image quality level corresponding to each pixel block of the first image, on the one hand, different areas of the second image are rendered with different degrees of fineness. On the other hand, different numbers of bits are allocated to different areas of the first image. In this way, the image quality of the compressed image can be improved when the channel bandwidth between the server and the terminal device is limited, and the waste of computing power consumed by the server during the image rendering process can be reduced.

[0013] In one possible implementation of the first aspect, the method further includes: after rendering and compression of each frame in the multi-frame image, generating a bitstream based on the compressed multi-frame image, and transmitting the bitstream. The method provided in this application can improve the image quality of the multi-frame image included in the bitstream in cloud-based image rendering scenarios with limited channel bandwidth, thereby enhancing the user experience. It can also reduce the waste of server computing power during the image rendering process.

[0014] In a second aspect, the present application provides an image processing device, which includes: an acquisition unit for acquiring a first image to be processed; a determination unit for dividing the first image into multiple pixel blocks and determining a sub-code rate of each pixel block; each pixel block includes multiple pixel points in the first image, and the sub-code rate of each pixel block is determined based on the area of ​​each pixel block and the number of pixel points included in each pixel block; a compression unit for compressing the first image according to multiple sub-code rates of the multiple pixel blocks.

[0015] In a possible implementation of the second aspect, the determination unit is specifically configured to determine an image quality level of each pixel block based on an area of ​​each pixel block and the number of pixels included in each pixel block, where the image quality level is used to characterize the degree of discreteness between brightness values ​​corresponding to multiple pixels in each pixel block; and determine a sub-code rate for each pixel block based on a target bit rate of the first image and the image quality level of each pixel block.

[0016] In a possible implementation of the second aspect, the determination unit is specifically used to: determine the brightness value of each pixel in the first image; determine the brightness variance value of multiple pixels included in each pixel block; and determine the image quality level of each pixel block based on the brightness variance value.

[0017] In a possible implementation of the second aspect, the image quality level of each pixel block includes a first level, a second level or a third level; the first level indicates: the brightness variance value of each pixel block is less than or equal to a first threshold; the second level indicates: the brightness variance value of each pixel block is greater than the first threshold and less than the second threshold; the second threshold is greater than the first threshold; the second level indicates: the brightness variance value of each pixel block is greater than or equal to the second threshold.

[0018] In a possible implementation of the second aspect, different image quality levels are associated with different bit allocation weights, and the image quality level is positively correlated with the bit allocation weight. The determination unit is specifically used to determine the bit allocation weight corresponding to each pixel block in the first image based on the image quality level of each pixel block; and determine the sub-code rate of each pixel block according to the target code rate of the first image and the bit allocation weight corresponding to each pixel block.

[0019] In a possible implementation of the second aspect, the acquisition unit is specifically used to acquire multiple source frames to be processed; render the multiple source frames to acquire the rendered multiple frames; the rendering method includes executing at least one of a rendering process of a rendering application or artificial intelligence generated content AIGC, and the rendered multiple frames include a first image.

[0020] In a possible implementation of the second aspect, the multi-frame source image includes a first source image and a second source image, the first source image corresponds to the first image, the second source image is adjacent to the first source image, and the timestamp of the second source image is after the timestamp of the first source image; the acquisition unit is specifically used to determine the rendering method of each pixel block in the second source image according to the image quality level of each pixel block in the first image; different image quality levels are configured with corresponding rendering methods, and the image quality level is positively correlated with the rendering accuracy of the rendering method corresponding to the image quality level; the second source image is rendered according to the rendering method of each pixel block in the second source image to obtain the second image.

[0021] In a possible implementation manner of the second aspect, the apparatus further includes a sending unit configured to generate and send a bit stream based on the compressed multiple frames of images when each frame of the multiple frames of images is rendered and compressed.

[0022] In a third aspect, the present application provides an image processing system, comprising a terminal device and a server; the terminal device is configured to: send a request to the server for obtaining a bit stream, the request including a continuous multi-frame image; the server is configured to: render the multi-frame image in response to the request; divide each frame image in the multi-frame image into a plurality of pixel blocks, and determine a sub-coding rate for each pixel block; each pixel block includes a plurality of pixel points in each frame image, and the sub-coding rate of each pixel block is determined based on the area of ​​each pixel block and the number of pixel points included in each pixel block; compress each frame image in the multi-frame image according to the multiple sub-coding rates of the plurality of pixel blocks; generate a bit stream based on the compressed multi-frame image; and send the bit stream to the terminal device.

[0023] In a fourth aspect, a server is provided, comprising: a processor and a memory, wherein the processor is connected to the memory, the memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions stored in the memory, thereby implementing any one of the methods provided in the first aspect.

[0024] In a fifth aspect, a chip is provided, which includes: a processor and an interface circuit; the interface circuit is used to receive code instructions and transmit them to the processor; the processor is used to run the code instructions to execute any one of the methods provided in the first aspect above.

[0025] In a sixth aspect, a computer-readable storage medium is provided, which stores computer execution instructions. When the computer execution instructions are run on a computer, the computer executes any one of the methods provided in the first aspect above.

[0026] In a seventh aspect, a computer program product is provided, comprising computer execution instructions, which, when executed on a computer, cause the computer to execute any one of the methods provided in the first aspect.

[0027] Among them, the technical effects brought about by any design method in the second to seventh aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of a cloud-based image rendering scenario;

[0029] Figure 2 is a schematic diagram of another cloud-based image rendering scenario;

[0030] FIG3 is a schematic diagram of another cloud-based image rendering scenario;

[0031] FIG4a is a schematic diagram of the architecture of an image processing system according to an embodiment of the present application;

[0032] FIG4 b is a schematic diagram of an image processing method according to an embodiment of the present application;

[0033] FIG4c is a schematic diagram of another image processing method according to an embodiment of the present application;

[0034] FIG5 is a structural block diagram of a server according to an embodiment of the present application;

[0035] FIG6 is a schematic flow chart of an image processing method according to an embodiment of the present application;

[0036] FIG7 is a schematic diagram of an image rendering process shown in an embodiment of the present application;

[0037] FIG8 is a schematic diagram of a pixel block according to an embodiment of the present application;

[0038] FIG9 is a schematic flow chart of another image processing method according to an embodiment of the present application;

[0039] FIG10 is a schematic flow chart of another image processing method according to an embodiment of the present application;

[0040] FIG11 is a schematic diagram of generating a rendering density tree according to an embodiment of the present application;

[0041] FIG12 is a schematic diagram of rendering of a pixel block according to an embodiment of the present application;

[0042] FIG13 is a schematic diagram of rendering of another pixel block according to an embodiment of the present application;

[0043] FIG14 is a structural block diagram of an image processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0045] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. At the same time, in some embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0046] In addition, the network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0047] Cloud 3D rendering applications (cloud 3D) involve rendering an image using a rendering application deployed on a server. The server then allocates and compresses the image before sending it to the user's terminal device. Bitrate allocation refers to allocating different numbers of bits to different regions of the image. Referring to Figure 1, a schematic diagram of a cloud image rendering scenario shows that a user sends input operation information to a server 110 via a terminal device 100. The server 110 then renders the image using a rendering application based on the user's input operation information. The server then compresses the rendered image using an encoder and sends the compressed image to the user's terminal device 100. To accommodate the limited channel bandwidth between the server 110 and the terminal device 100, the server 110 typically compresses the rendered image using lossy compression. However, because each region of the image is compressed at the same or similar bitrate, the quality of each region in the compressed image is lower than that of the rendered image. This results in a waste of computing power consumed by the server 110 during the image rendering process, and the image quality after lossy compression is poor, making it difficult to meet user needs.

[0048] In related art 1, referring to FIG2 , server 110 typically feeds back the quality of the compressed image to a rendering application. The rendering application then renders the image based on the compressed image quality, ensuring that the rendered image quality matches the compressed image quality. This prevents the rendered image quality from being higher than the compressed image quality, thereby saving the computing power consumed by server 110 during the image rendering process. However, while this approach saves computing power on server 110 during the image rendering process, it suffers from the problem of poor rendered image quality, making it difficult to meet user needs.

[0049] In related technology 2, see Figure 3, the server 110 performs image recognition through the objects in the image, and then determines the importance of each object based on the position information of each object in the image. For example, objects that are closer to the camera position are determined to have high importance, and objects that are farther away from the camera position are determined to have low importance. Finally, the encoder determines the bit rate of different areas of each frame of the image based on the importance of each object and completes the compression. Although this method can improve the image quality of some areas on the image to a certain extent, there is a problem of poor accuracy in determining the importance of objects based on position information. For example, in some scenarios, object A that is farther away from the camera position is the content that the user needs to focus on, but because object A is farther away from the camera position, the server 110 determines that the importance of object A is low, and thus the bit rate allocated to object A is low. In this case, the picture quality of object A in the image is blurred, which is difficult to meet the user's usage needs.

[0050] Based on the above content, an embodiment of the present application provides an image processing method that can be applied to the above-mentioned cloud image rendering scenario. The server determines the image quality level of each pixel block of the first image, and then determines the sub-bit rate of each pixel block on the first image based on the target bit rate of the first image and the image quality level of each pixel block. Then, the first image is compressed based on the sub-bit rate of each pixel block to obtain a compressed first image. The method provided in the embodiment of the present application is based on the image quality level of each pixel block of the first image and the image quality level of each pixel block. Different bit rates are assigned to each pixel block. For example, a higher number of bits is assigned to areas with higher image quality levels on the first image, and a lower number of bits is assigned to areas with lower image quality levels on the first image. When the channel bandwidth between the server and the terminal device is limited, that is, when the target bit rate of the first image is fixed, the image quality of some areas in the compressed image can be improved, and the waste of computing power of the server in the process of rendering the first image can be reduced.

[0051] The image processing method provided in the embodiment of the present application can be applied to an image processing system. See Figure 4a, which is a schematic diagram of the architecture of an image processing system provided in the embodiment of the present application, including a terminal device 410 and an image processing apparatus 420.

[0052] In one example, referring to FIG4b, FIG4b is a process diagram of an image processing method shown in an embodiment of the present application. The image processing device 420 includes an acquisition unit 421, a determination unit 422, and a compression unit 423. The acquisition unit 421 is used to render according to the operation information input by the user to obtain a first image, and then send the first image to the determination unit 422. The determination unit 422 is used to divide the first image into multiple pixel blocks and determine the image quality level of each pixel block of the first image. Each pixel block includes multiple pixels in the first image. The image quality level is used to characterize the degree of discreteness between the brightness values ​​corresponding to the multiple pixels in each pixel block. The determination unit 422 is also used to send the image quality level of each pixel block of the first image to the compression unit 423. The compression unit 423 is used to determine the sub-code rate of each pixel block of the first image based on the target code rate of the first image and the image quality level of each pixel block, and to compress the first image according to the multiple sub-code rates of the multiple pixel blocks.

[0053] Optionally, referring to Figure 4c, when the first image is a frame in a continuous multi-frame image, the determination unit 422 is also used to send the image quality level of each pixel block of the first image to the acquisition unit 421, so that the acquisition unit 421 renders the second image based on the image quality level of each pixel block of the first image to obtain a second image, wherein the second image is an image adjacent to the first image in the continuous multi-frame image, and the timestamp of the second image is after the timestamp of the first image.

[0054] It should be understood that after obtaining the second image, the image processing device 420 provided in the embodiment of the present application determines the image quality level and sub-bit rate of each pixel block of the second image, and compresses the second image in the same manner as the above-mentioned processing method for the first image, which will not be repeated here.

[0055] In some embodiments, a communication connection is established between the terminal device 410 and the image processing device 420. The communication connection may include a wired communication connection or a wireless communication connection. For example, the wireless communication connection may include a short-range wireless communication connection such as a Bluetooth communication connection, a near-field communication connection, and a wireless local area network communication connection. Among them, the wireless communication technology for establishing a wireless communication connection between the terminal device 410 and the image processing device 420 includes but is not limited to at least one of the following: wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT) (for example, traditional Bluetooth or Bluetooth low energy (BLE)), near field communication (NFC), Zigbee, frequency modulation (FM) and infrared (IR), etc.

[0056] It can be understood that in the embodiment of the present application, the terminal device 410 can be, for example, a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a smart watch, a netbook, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, a vehicle-mounted device, a smart screen, a smart car, a smart speaker, a robot, etc. The present application does not impose any special restrictions on the specific form of the terminal device 410.

[0057] In the embodiments of the present application, the image processing device 420 may be a computing device with any computing processing capabilities, or a functional module within such a computing device. For example, the computing device may be a general-purpose computer, a laptop computer, or a server. The present application does not impose any particular limitation on the specific form of the image processing device 420.

[0058] Furthermore, the operating systems installed in the terminal device 410 and the image processing apparatus 420 include but are not limited to This application does not limit the specific types of the terminal device 410 and the image processing apparatus 420, or the type of the operating system if an operating system is installed.

[0059] The following embodiments are illustratively described by taking the image processing device 420 as a server as an example.

[0060] 5 shows a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 500 includes a processor 510, a memory 520, and at least one communication interface 530.

[0061] The processor 510 may include one or more processing cores. The processor 510 uses various interfaces and lines to connect to various parts within the server 500, and executes various functions and processes data of the server 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 510 can be implemented in the form of at least one hardware of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 510 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. It is understandable that the above-mentioned modem may not be integrated into the processor 510, but may be implemented separately through a communication chip.

[0062] The memory 520 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 520 may include a program storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as an image rendering function, an image compression function, etc.), instructions for implementing various method embodiments described below, and the like.

[0063] The communication interface 530 is used to communicate with a communication network or a device in a communication network, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), or an AP in a WLAN network. For example, the communication interface 530 of the server 500 communicates with the terminal device 410.

[0064] In a specific implementation, in terms of physical implementation, the above-mentioned devices (such as the processor 510, the memory 520 and the communication interface 530) can be devices in the same computing device; or, at least two of the devices can be set in the same computing device, that is, as different devices in a computing device, such as a deployment method similar to devices or devices in a distributed system.

[0065] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the server 500. In some embodiments, the server 500 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0066] The image processing method provided in the embodiments of the present application is described below with reference to the accompanying drawings.

[0067] FIG6 is a flowchart of an image processing method provided in an embodiment of the present application. Optionally, the method can be executed by the server 500 having the hardware structure shown in FIG5 . The method can include the following steps:

[0068] S601: Acquire a first image to be processed.

[0069] Specifically, the first image may be an image rendered by the server. For example, the server renders a first source image corresponding to the first image to obtain the first image.

[0070] In some embodiments, the server renders the first image by performing at least one of a rendering process of a rendering application or AI-generated content. For example, referring to FIG7 , FIG7 (a) is a schematic diagram of the process by which the server generates the first image based on the first source image using a rendering application 701. FIG7 (b) is a schematic diagram of the process by which the server generates the first image based on the first source image using AI-generated content 702.

[0071] It should be understood that the server may also render the first image based on other methods, and the application embodiment does not impose any particular limitation on the specific implementation method of the server obtaining the first image.

[0072] In a possible implementation, the above S601 specifically includes the following steps:

[0073] Obtain multiple source image frames to be processed, render the multiple source image frames, and obtain the rendered multiple image frames.

[0074] Specifically, the multiple frames of source images include a first source image corresponding to the first image, and the rendered multiple frames of images include the first image.

[0075] In some embodiments, the multiple source images are carried in a request sent by the terminal device to the server. The request is used to request the server to render the multiple source images carried in the request to obtain multiple images and return a bitstream including the multiple images.

[0076] Specifically, the request can be generated based on an operation input by a user on a screen displayed on a terminal device, and the bitstream is used to display the multiple frames of the screen that the terminal device should display after the user input operation. For example, in a scenario where a terminal device is running a game, when a user clicks on the game screen displayed on the terminal device, the terminal device generates a corresponding request in response to the user's click operation on the game interface and sends the request carrying multiple frames of source images to the server, so that the server returns a bitstream including multiple frames of images, which is used to display the multiple frames of the game screen that the terminal device should display after the user click operation.

[0077] S602: Divide the first image into a plurality of pixel blocks.

[0078] Specifically, each pixel block corresponds to a different area on the first image, and each pixel block includes multiple pixels in the first image. It should be understood that the shape of each pixel block and the number of pixels included in the pixel block can be customized based on user needs. For example, different pixel blocks on the same frame of image can include the same or different numbers of pixels, and different pixel blocks on the same frame of image can have the same or different shapes. The embodiments of the present application do not particularly limit the specific implementation of the pixel blocks.

[0079] For example, referring to FIG8 , FIG8 is a schematic diagram of a pixel block according to an embodiment of the present application. The server divides image A into four square pixel blocks, namely pixel block A, pixel block B, pixel block C, and pixel block D. Each pixel block includes 4*4 pixels.

[0080] S603: Determine the sub-coding rate of each pixel block.

[0081] Specifically, the sub-coding rate of each pixel block of the first image is the number of bits allocated by the server to each pixel block.

[0082] In a possible implementation, referring to FIG. 9 , the above S603 specifically includes the following steps:

[0083] S6031. Determine the image quality level of each pixel block according to the area of ​​each pixel block and the number of pixels included in each pixel block.

[0084] Specifically, the server can pre-gradiently set multiple image quality levels corresponding to pixel blocks. The multiple image quality levels are used to represent the different degrees of discreteness between the brightness values ​​corresponding to multiple pixel points in different pixel blocks, thereby representing the different picture qualities of each pixel block. The picture quality can also be understood as the degree of fineness of the pixel block display.

[0085] Optionally, the image quality level includes a first level, a second level, and a third level. Among them, the first level is the lowest image quality level and the third level is the highest image quality level. When the content on a pixel block requires low-precision display, the image quality level corresponding to the pixel block is the first level; when the content on a pixel block requires medium-precision display, the image quality level corresponding to the pixel block is the second level; when the content on a pixel block requires high-precision display, the image quality level corresponding to the pixel block is the third level.

[0086] It should be understood that the classification of the image quality levels of pixel blocks into the first, second, and third levels in the embodiments of the present application is merely exemplary, and the method provided in the embodiments of the present application provides a more fine-grained classification of the image quality levels of pixel blocks. For example, the image quality levels of pixel blocks may be divided into five levels based on the different display accuracy of the content on the pixel blocks. The embodiments of the present application do not impose any particular restrictions on the specific method of dividing the image quality levels of pixel blocks.

[0087] In some embodiments, referring to FIG. 10 , the above-mentioned S6031 specifically includes the following steps:

[0088] S60311. Determine the brightness value of each pixel in the first image.

[0089] Specifically, the color mode of the first image obtained by the server through the rendering application is the RGB color mode. Among them, the RGB color mode is a color standard, which obtains the color of each pixel on the first image by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. The server first switches the color mode of the first image from the RGB color mode to the YUV color mode. Among them, the YUV color mode represents the color of each pixel on the first image by a brightness value (Y) and two color difference values ​​(U and V). In other words, the server generates a YUV brightness map of the first image based on the rendered first image, and then obtains the brightness value corresponding to each pixel on the first image from the YUV brightness map of the first image.

[0090] S60312. Determine the brightness variance value of multiple pixel points included in each pixel block.

[0091] Specifically, the server calculates the brightness variance value of each pixel block based on the brightness value corresponding to each pixel point. The brightness variance value of the pixel block is equal to the average of the sum of squared deviations between the brightness value of each pixel point in the pixel block and the average brightness value of all pixels.

[0092] S60313. Determine the image quality level of each pixel block based on the brightness variance value.

[0093] In a possible implementation, the above S60313 specifically includes the following steps:

[0094] When the brightness variance value of the pixel block is less than or equal to the first threshold, the image quality level is determined to be the first level; when the brightness variance value of the pixel block is greater than the first threshold and less than the second threshold, the image quality level is determined to be the second level; when the brightness variance value of the pixel block is greater than or equal to the second threshold, the image quality level is determined to be the third level.

[0095] For example, in combination with Figure 8, when the first threshold is 2 and the second threshold is 10, assuming that the brightness variance value of pixel block A is 1, the brightness variance value of pixel block B is 4, the brightness variance value of pixel block C is 9, and the brightness variance value of pixel block D is 15, then the server determines that the image quality level of pixel block A is the first level, the image quality levels of pixel blocks B and pixel blocks C are the second level, and the image quality level of pixel block D is the third level.

[0096] It should be understood that the embodiments of the present application pre-set the first threshold and the second threshold, and divide the image quality level into the first level, the second level and the third level based on the first threshold and the second threshold for exemplary purposes only. The method provided in the embodiments of the present application can also set a greater number of thresholds, and then divide a greater number of threshold intervals based on a greater number of thresholds, so as to divide the image quality level of pixel blocks with finer granularity through a greater number of threshold intervals. The embodiments of the present application do not impose any special restrictions on the specific method of dividing the image quality level of pixel blocks.

[0097] It can be seen from the above S60311-S60313 that the method provided in the embodiment of the present application can determine the brightness variance value of each pixel block of the first image by generating a YUV brightness map corresponding to the first image, and then can determine the image quality level of each pixel block of the first image based on the brightness variance value.

[0098] In one possible implementation, the method provided in an embodiment of the present application also includes the following steps: marking different pixel blocks on the first image based on the image quality level corresponding to each pixel block, and then merging adjacent pixel blocks with the same image quality level to obtain a rendering density tree of the first image, and the rendering density tree of the first image is used to identify the image quality levels corresponding to different areas on the first image.

[0099] For example, referring to FIG11 in conjunction with FIG8 , FIG11 is a schematic diagram of generating a rendering density tree according to an embodiment of the present application. The server first marks pixel blocks in image A with a first quality level as 1, pixel blocks with a second quality level as 2, and pixel blocks with a third quality level as 3. Adjacent pixel blocks with the same quality level are then merged to ultimately obtain a rendering density tree corresponding to image A.

[0100] Optionally, the process of the server determining the image quality level of each pixel block of the first image and the process of the server generating the rendering density tree of the first image can be implemented based on a compute shader (CS) tool in a graphics processing unit (GPU) configured on the server.

[0101] S6032: Determine a sub-rate for each pixel block according to a target bit rate of the first image and an image quality level of each pixel block.

[0102] Specifically, the server determines the target bit rate of the first image based on the channel bandwidth of the server and the terminal device, that is, the total number of bits allocated by the server to the first image, and the sub-bit rate of each pixel block of the first image is the number of bits allocated by the server to each pixel block.

[0103] In some embodiments, different image quality levels are associated with different bit allocation weights, and the image quality levels are positively correlated with the bit allocation weights. The above S6032 specifically includes the following steps:

[0104] A bit allocation weight corresponding to each pixel block in the first image is determined based on the image quality level of each pixel block; and a sub-code rate of each pixel block is determined according to a target code rate of the first image and the bit allocation weight corresponding to each pixel block.

[0105] Exemplarily, the specific heat allocation weight associated with a first quality level is x, the bit allocation weight associated with a second quality level is y, and the bit allocation weight associated with a third quality level is z. The server determines a target bitrate for the first image based on the channel bandwidth between the server and the terminal device. The server then determines a bit allocation weight for each pixel block of the first image based on the image quality level of each pixel block. Finally, the server determines a bitrate for each pixel block of the first image based on the target bitrate for the first image and the bit allocation weight for each pixel block.

[0106] In a possible implementation, when the server performs coding tree unit (CTU) level bit rate allocation through an encoder, the above S603 specifically includes the following steps:

[0107] The server determines a bit allocation weight for each coding unit (CU) according to the bit allocation weight corresponding to each pixel block, and determines a sub-rate corresponding to each pixel block according to the bit allocation weight for each coding unit and a target bit rate of the first image.

[0108] Specifically, when a pixel block of the first image coincides with a coding unit block, the bit allocation weight of the pixel block is determined as the bit allocation weight of the coding unit block. When a pixel block of the first image includes multiple coding unit blocks, an average value of the bit allocation weights of the pixel block is determined based on the number of coding unit blocks, and the average value of the bit allocation weights of the pixel block is determined as the bit allocation weight of each coding unit block in the multiple coding unit blocks. When a coding unit block includes multiple pixel blocks, the sum of the bit allocation weights of the multiple pixel blocks is determined as the bit allocation weight of the coding unit block. Finally, the sub-code rate corresponding to each pixel block is determined based on the bit allocation weight of each coding unit block and the target code rate of the first image.

[0109] S604: Compress the first image according to multiple sub-rates of the multiple pixel blocks.

[0110] Optionally, the method provided in the embodiment of the present application further includes the following steps:

[0111] When each frame of the plurality of frames of images is rendered and compressed, a bit stream is generated according to the compressed plurality of frames of images;

[0112] A bit stream including multiple frames of images is sent to a terminal device.

[0113] As can be seen from the above S601-S604, the method provided in the embodiment of the present application is based on the image quality level of each pixel block of the first image, and allocates different bit rates to each pixel block based on the image quality level of each pixel block. For example, a higher number of bits is allocated to an area with a higher image quality level on the first image, and a lower number of bits is allocated to an area with a lower image quality level on the first image. When the channel bandwidth between the server and the terminal device is limited, that is, when the target bit rate of the first image is fixed, the image quality of some areas in the compressed image can be improved, and the waste of computing power of the server in the process of rendering the first image can be reduced.

[0114] In some embodiments, when a server sequentially renders multiple frames including a first image, the server may determine the image quality level corresponding to each pixel block of the first image after the first image is obtained by rendering the first source image, and then render the second source image to obtain the second image. The second source image is adjacent to the first source image, and the timestamp of the second source image is later than the timestamp of the first source image. Alternatively, after each frame of the multiple frames is rendered, the image quality level corresponding to each pixel block of each frame may be determined sequentially. The embodiments of the present application do not impose any particular restrictions on the order in which images are rendered.

[0115] Optionally, when the server completes rendering based on the first source image to obtain the first image, it determines the image quality level corresponding to each pixel block of the first image, and then renders the second source image to obtain the second image. In this case, the method provided in the embodiment of the present application further includes the following steps:

[0116] When rendering the second image, each pixel block of the second image is rendered based on the image quality levels of the plurality of pixel blocks of the first page image to obtain a rendered second image.

[0117] Specifically, when the server renders an image through a rendering application or AIGC, for each pixel point of each frame of the image, the server can use fine rendering or interpolation calculation to obtain it. Among them, the pixel points obtained by fine rendering have a higher degree of fineness, and the pixel points obtained by interpolation calculation have a lower degree of fineness.

[0118] Different image quality levels are associated with different rendering methods. Since the second image is the next frame after the first image, the second image corresponds to the pixel blocks on the first image in a one-to-one manner. Each pixel block on the first image corresponds to a different rendering method for each image quality level. Therefore, based on the image quality level corresponding to each pixel block in the first image, the rendering method corresponding to each pixel block on the first image is used to determine the rendering method for each pixel block corresponding to each position on the second source image. The server then renders each pixel block on the second source image using the different rendering methods to generate the second image.

[0119] For example, see FIG12 , which is a schematic diagram of rendering a pixel block shown in an embodiment of the present application. Assuming that the first image includes three pixel blocks, the image quality level of pixel block D1 is the first level, the image quality level of pixel block E1 is the second level, and the image quality level of pixel block F1 is the third level, and the rendering density tree i is the rendering density tree of the first image. When the server renders based on the second source image, since the positions of pixel block D2 and pixel block D1 of the second image correspond, the positions of pixel block E2 and pixel block E1 of the second image correspond, and the positions of pixel block F2 and pixel block F1 of the second image correspond, the rendering methods of different pixel blocks of the second source image are determined based on the rendering density tree i. Specifically, the rendering mode of pixel block D2 is determined based on the image quality level of pixel block D1, the rendering mode of pixel block E2 is determined based on the image quality level of pixel block E1, and the rendering mode of pixel block F2 is determined based on the image quality level of pixel block F1. The rendering mode of pixel block D2 is the first rendering mode corresponding to the first level, the rendering mode of pixel block E2 is the second rendering mode corresponding to the second level, and the rendering mode of pixel block F2 is the third rendering mode corresponding to the third level. The server then renders the second source image according to the rendering modes corresponding to pixel blocks D2, E2, and F2, respectively, to obtain a second image.

[0120] In one example, referring to FIG13 , FIG13 is a schematic diagram of a pixel block rendering shown in an embodiment of the present application, including the rendering process of three blocks with different image quality levels, wherein the pixel points filled with shading are pixel points obtained by fine rendering, and the pixel points not filled with shading are pixel points obtained by interpolation calculation. Assuming that the image quality level of pixel block A is the first level, the first rendering method corresponding to the first level is shown in (a) of FIG13 , and the pixel points of the first ratio in the pixel block are generated by fine rendering, and the remaining pixel points are generated by interpolation calculation. Among them, the first ratio is 25%. The image quality level of pixel block B is the second level, and the second rendering method corresponding to the second level is shown in (b) of FIG13 , and the pixel points of the second ratio in the pixel block are generated by fine rendering, and the remaining pixel points are generated by interpolation calculation. Among them, the second ratio is 50%. The third rendering method corresponding to the third level is shown in FIG13(c). Pixel block C has the third quality level. The rendering method for pixel block C is to generate pixels in the third ratio within the pixel block using fine rendering, while the remaining pixels are generated using interpolation. The third ratio is 100%.

[0121] It should be noted that the above different rendering methods corresponding to different image quality levels are only exemplary descriptions. The embodiment of the present application can divide the image quality levels into finer granularity and configure different rendering methods for each image quality level. The embodiment of the present application does not impose any special restrictions on this.

[0122] The method provided in the embodiment of the present application determines the rendering method of each pixel block on the second image based on the image quality level corresponding to each pixel block of the first frame image during the rendering of the second image, and finally renders the second source image to obtain the second image. The method realizes the coordination of the image rendering process and the image compression process of the server, that is, based on the image quality level corresponding to each pixel block of the first image, on the one hand, it realizes the rendering of different picture areas of the second image with different degrees of fineness. On the other hand, it realizes the compression of different picture areas of the first image with different degrees of fineness. In this way, when the channel bandwidth between the server and the terminal device is limited, the image quality of the compressed image can be improved, and the waste of computing power consumed by the server in the image rendering process can be reduced.

[0123] Optionally, the method provided in the embodiment of the present application further includes the following steps:

[0124] When each frame of the plurality of frames of images is rendered and compressed, a bit stream is generated according to the compressed plurality of frames of images;

[0125] A bit stream including multiple frames of images is sent to a terminal device.

[0126] In one possible implementation, the program for implementing each step of the image processing method provided in the embodiment of the present application may be configured inside the server and implemented in the form of multiple application programming interfaces (APIs) called by the server.

[0127] In one example, the process of generating the rendering density tree is implemented by the server calling an application programming interface (API). The server calls the API to generate the rendering density tree for the first image. For example, the name of the API for generating the rendering density tree for the first image may be:

[0128] glGenRenderDensityTree(int*pixels,double Thd,double k);

[0129] vkGenRenderDensityTree(int*pixels,double Thd,double k);

[0130] Wherein, pixels is each frame image in the multiple frames of images, Thd is the above-mentioned first threshold, and k*Thd is the above-mentioned second threshold.

[0131] This API is used to generate a rendering density tree for each frame of image in response to a call from the server.

[0132] In another example, the name of the API that returns the rendering density tree for each frame of an image could be:

[0133] xxGetRenderDensityTree();

[0134] glGetRenderDensityTree();

[0135] glGetRenderDensityTree();

[0136] The API is used to respond to a call from a server after generating a rendering density tree, so that the server obtains the rendering density tree of the first image.

[0137] In yet another example, the process of the server rendering the second source image based on the rendering density tree of the first image may also be implemented based on an API, and the name of the API may be:

[0138] glSetRenderDensityTree(Tree);

[0139] vkSetRenderDensityTree(Tree);

[0140] The API is used to render the second source image based on the rendering density tree of the first image to obtain the second image after the server receives the rendering density tree of the first image, that is, after receiving the image quality levels of different pixel blocks on the first image.

[0141] It should be noted that the above API is only an exemplary description. The embodiments of the present application can also implement the various steps of the image processing method provided in the present application through other methods, and the embodiments of the present application do not impose any special restrictions on this.

[0142] The above mainly introduces the solution of the embodiment of the present application from the perspective of method. It is understandable that, in order to realize the above functions, the above server includes at least one of the hardware structure and software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0143] The embodiment of the present application can divide the image processing device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0144] For example, FIG14 shows a block diagram of an image processing device provided in an embodiment of the present application. The device includes an acquisition unit 1410, a determination unit 1420, and a compression unit 1430. The acquisition unit 1410 is used to acquire a first image to be processed; the determination unit 1420 is used to divide the first image into multiple pixel blocks and determine the sub-code rate of each pixel block; each pixel block includes multiple pixel points in the first image, and the sub-code rate of each pixel block is determined based on the area of ​​each pixel block and the number of pixels included in each pixel block; the compression unit 1430 is used to compress the first image according to the multiple sub-code rates of the multiple pixel blocks.

[0145] Optionally, the determination unit 1420 is specifically used to determine the image quality level of each pixel block based on the area of ​​each pixel block and the number of pixels included in each pixel block, where the image quality level is used to characterize the degree of discreteness between the brightness values ​​corresponding to multiple pixels in each pixel block; and determine the sub-coding rate of each pixel block based on the target bit rate of the first image and the image quality level of each pixel block.

[0146] Optionally, the determination unit 1420 is specifically used to: determine the brightness value of each pixel in the first image; determine the brightness variance value of multiple pixels included in each pixel block; and determine the image quality level of each pixel block based on the brightness variance value.

[0147] Optionally, the image quality level of each pixel block includes a first level, a second level or a third level; the first level indicates: the brightness variance value of each pixel block is less than or equal to a first threshold; the second level indicates: the brightness variance value of each pixel block is greater than the first threshold and less than the second threshold; the second threshold is greater than the first threshold; the second level indicates: the brightness variance value of each pixel block is greater than or equal to the second threshold.

[0148] Optionally, different image quality levels are associated with different bit allocation weights, and the image quality level is positively correlated with the bit allocation weight. The determination unit 1420 is specifically used to determine the bit allocation weight corresponding to each pixel block in the first image based on the image quality level of each pixel block; and determine the sub-code rate of each pixel block according to the target code rate of the first image and the bit allocation weight corresponding to each pixel block.

[0149] Optionally, the acquisition unit 1410 is specifically used to obtain multiple frame source images to be processed; render the multiple frame source images to obtain the rendered multiple frame images; the rendering method includes executing at least one of the rendering process of the rendering application or artificial intelligence generated content AIGC, and the rendered multiple frame images include the first image.

[0150] Optionally, the multi-frame source image includes a first source image and a second source image, the first source image corresponds to the first image, the second source image is adjacent to the first source image, and the timestamp of the second source image is after the timestamp of the first source image; the acquisition unit 1410 is specifically used to determine the rendering method of each pixel block in the second source image according to the image quality level of each pixel block in the first image; different image quality levels are configured with corresponding rendering methods, and the image quality level is positively correlated with the rendering accuracy of the rendering method corresponding to the image quality level; the second source image is rendered according to the rendering method of each pixel block in the second source image to obtain a second image.

[0151] Optionally, the apparatus further includes a sending unit 1440 configured to generate and send a bit stream based on the compressed multiple frames of images when each frame of the multiple frames of images is rendered and compressed.

[0152] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any of the above image processing devices and the description of their beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0153] The present application also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the image processing methods described in each of the above embodiments. For explanations of the relevant contents and descriptions of the beneficial effects of any of the above-mentioned computer-readable storage media, please refer to the corresponding embodiments described above and will not be repeated here.

[0154] The embodiment of the present application also provides a chip. The chip integrates a control circuit and one or more ports for implementing the functions of the above-mentioned image processing device. Optionally, the functions supported by the chip can be referred to above and will not be repeated here. A person of ordinary skill in the art will understand that all or part of the steps of the above-mentioned embodiment can be implemented by a program to instruct the relevant hardware to complete the program, which can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The above-mentioned unit or processor can be a central processing unit, a general-purpose processor, a specific circuit structure (application specific integrated circuit, ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0155] The present application also provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to perform any of the methods in the above embodiments. The computer program product includes a plurality of computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes multiple available media integrated therein. The available media may be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., SSD).

[0156] It should be noted that the above-mentioned device for storing computer instructions or computer programs provided in the embodiments of the present application, such as but not limited to, the above-mentioned memory, computer-readable storage medium and communication chip, etc., all have non-volatile (non-transitory). Those skilled in the art should be aware that in the above-mentioned multiple examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as multiple instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0157] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image to be processed; Dividing the first image into a plurality of pixel blocks, and determining a sub-coding rate of each pixel block; Each pixel block includes a plurality of pixel points in the first image, and a sub-coding rate of each pixel block is determined according to an area of ​​each pixel block and a number of pixel points included in each pixel block; The first image is compressed according to a plurality of sub-rates of the plurality of pixel blocks.

2. The method according to claim 1, characterized in that The determining of the sub-code rate of each pixel block includes: The image quality level of each pixel block is determined according to the area of ​​each pixel block and the number of pixels included in each pixel block, and the image quality level is used to characterize the discrete degree between the brightness values ​​corresponding to multiple pixels in each pixel block; the sub-code rate of each pixel block is determined according to the target bit rate of the first image and the image quality level of each pixel block.

3. The method according to claim 2, characterized in that Determining the image quality level of each pixel block includes: Determine a brightness value of each pixel in the first image; Determine the brightness variance value of the plurality of pixels included in each pixel block; The image quality level of each pixel block is determined based on the brightness variance value.

4. The method according to claim 3, characterized in that The image quality level of each pixel block includes a first level, a second level or a third level; The first level indicates that: the brightness variance value of each pixel block is less than or equal to a first threshold; The second level indicates that: the brightness variance value of each pixel block is greater than the first threshold and less than the second threshold; The second threshold is greater than the first threshold; The second level indicates that the brightness variance value of each pixel block is greater than or equal to the second threshold.

5. The method according to any one of claims 2 to 4, characterized in that: Different image quality levels are respectively associated with different bit allocation weights, the image quality level is positively correlated with the bit allocation weight, and determining the sub-bit rate of each pixel block according to the target bit rate of the first image and the image quality level of each pixel block includes: Determining a bit allocation weight corresponding to each pixel block in the first image based on the image quality level of each pixel block; The sub-coding rate of each pixel block is determined according to the target coding rate of the first image and the bit allocation weight corresponding to each pixel block.

6. The method according to any one of claims 1 to 5, characterized in that The step of acquiring a first image to be processed comprises: Obtain multiple frames of source images to be processed; Render the multiple source image frames to obtain the rendered multiple image frames; the rendering method includes executing at least one of a rendering process of a rendering application or artificial intelligence generated content AIGC, and the rendered multiple image frames include the first image.

7. The method according to claim 6, characterized in that The multiple frames of source images include a first source image and a second source image, the first source image corresponds to the first image, the second source image is adjacent to the first source image, and a timestamp of the second source image is later than a timestamp of the first source image; The step of rendering the multiple source image frames to obtain the rendered multiple image frames includes: Determining a rendering method for each pixel block in the second source image according to the image quality level of each pixel block in the first image; different image quality levels are configured with corresponding rendering methods, and the image quality level is positively correlated with the rendering accuracy of the rendering method corresponding to the image quality level; The second source image is rendered according to the rendering method of each pixel block in the second source image to obtain the second image.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: When each frame of the multiple frames of images is rendered and compressed, a bit stream is generated according to the compressed multiple frames of images; The bit stream is transmitted.

9. An image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a first image to be processed; a determining unit, configured to divide the first image into a plurality of pixel blocks, and determine a sub-code rate of each pixel block; each pixel block includes a plurality of pixel points in the first image, and the sub-code rate of each pixel block is determined according to the pixel blocks of each pixel block; The area of ​​each pixel block and the number of pixels included in each pixel block are determined; A compression unit is used to compress the first image according to multiple sub-rates of the multiple pixel blocks.

10. An image processing system, characterized in that: Including terminal equipment and servers; The terminal device is used for: Sending a request for obtaining a bit stream to the server, wherein the request includes a plurality of continuous frames of images; The server is used to: In response to the request, rendering to obtain the multiple frames of images; Dividing each frame of the multiple frames into a plurality of pixel blocks, and determining a sub-coding rate of each pixel block; Each pixel block includes a plurality of pixel points in each frame of the image, and the sub-code rate of each pixel block is determined according to the area of ​​each pixel block and the number of pixel points included in each pixel block; compressing each frame of the multiple frames of images according to multiple sub-rates of the multiple pixel blocks; Generate a bit stream according to the compressed multiple frames of images; The bit stream is sent to the terminal device.

11. A server, characterized in that: The server includes a memory and multiple processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the server executes the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer can execute the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Code rate allocation method and device

    CN110324615A

  • Image coding method and device, computer equipment and storage medium

    CN110493600A

  • Image storage method and device, electronic equipment and storage medium

    CN112911186A

  • Image processing method, device and system and storage medium

    CN116847094A

  • Image processing method and apparatus

    WO2022067656A1