Image processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a storage medium, and relates to the technical field of image processing.The method comprises the steps that a to-be-processed image is obtained in response to a trigger operation; wherein any pixel point in the to-be-processed image has a corresponding probability, and the probability is used for indicating the possibility that any pixel point belongs to the region of interest; and based on the probability, adjusting a pixel value of a pixel point in the to-be-processed image to obtain a first target image after image quality optimization. Therefore, in consideration of different sensitivities of human eyes to different areas in the image, targeted processing is performed on each pixel point in the to-be-processed image based on the areas of interest sensitive to the human eyes, so that the processed image looks more striking, the image quality is improved, and better visual experience is brought to a user.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, electronic device, chip, and storage medium. Background Art

[0002] With the rapid development of digital image processing technologies, users' requirements for image quality are increasing day by day. Whether professional photographers, designers, or ordinary consumers, they all expect to obtain clearer, more detailed, and more vivid images. To meet this demand, image enhancement technologies have been widely studied and developed. Among them, image enhancement technologies can optimize the local contrast of images and improve the overall quality and visual effects of images. Summary of the Invention

[0003] The present application aims to solve at least one of the technical problems in the related technologies to some extent.

[0004] To this end, the present application provides an image processing method, apparatus, electronic device, chip, and storage medium to perform targeted processing on each pixel point in the image to be processed based on the region of interest sensitive to the human eye, so that the processed image looks more prominent, thereby improving the image quality and bringing a better visual experience to users.

[0005] An embodiment of one aspect of the present application provides an image processing method, including:

[0006] In response to a trigger operation, obtaining an image to be processed; wherein, any pixel point in the image to be processed has a corresponding probability, and the probability is used to indicate the possibility that the any pixel point belongs to the region of interest;

[0007] Based on the probability, adjusting the pixel value of the pixel point in the image to be processed to obtain a first target image with optimized image quality.

[0008] An embodiment of another aspect of the present application provides an image processing apparatus, including:

[0009] An obtaining module, configured to obtain an image to be processed in response to a trigger operation; wherein, any pixel point in the image to be processed has a corresponding probability, and the probability is used to indicate the possibility that the any pixel point belongs to the region of interest;

[0010] An adjusting module, configured to adjust the pixel value of the pixel point in the image to be processed based on the probability to obtain a first target image with optimized image quality.

[0011] In another embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the image processing method described in the foregoing one aspect is implemented.

[0012] In another embodiment of the present application, a chip is provided. The chip includes an interface circuit and a processing circuit that are coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is configured to execute the image processing method described in the foregoing one aspect.

[0013] In another embodiment of the present application, a non-transitory computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the image processing method described in the foregoing one aspect is implemented.

[0014] In another embodiment of the present application, a computer program product is provided, on which a computer program is stored. When the program is executed by a processor, the image processing method described in any of the foregoing aspects is implemented.

[0015] For the image processing method, device, electronic device, chip, and storage medium provided in the present application, considering that the human eye has different sensitivities to different regions in an image, and based on the regions of interest that the human eye is sensitive to, targeted processing is performed on each pixel point in the image to be processed, so that the processed image looks more prominent, thereby improving the image quality and bringing a better visual experience to the user.

[0016] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings

[0017] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0018] Figure 1 is a schematic flowchart of the first image processing method provided by an embodiment of the present application;

[0019] Figure 2 is a schematic flowchart of the second image processing method provided by an embodiment of the present application;

[0020] Figure 3 is a schematic flowchart of the third image processing method provided by an embodiment of the present application;

[0021] Figure 4 is a schematic flowchart of the fourth image processing method provided by an embodiment of the present application;

[0022] Figure 5 Schematic diagram of the implementation principle of the image processing method provided by any embodiment of this application;

[0023] Figure 6 Schematic diagram of the structure of an image processing apparatus provided by an embodiment of this application;

[0024] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of this application;

[0025] Figure 8 Schematic diagram of the structure of a chip proposed by an embodiment of this application. Detailed implementation manners

[0026] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as limiting the present application.

[0027] Local Tone Mapping (LTM) technology is an important method in image enhancement technology. By optimizing the local contrast of an image, the overall quality and visual effect of the image can be improved. In the related art, the LTM technology uses the local statistical information of the image, such as the local histogram, etc., to optimize the quality of the local image, and is widely used in fields such as image display, image rendering, and video transcoding.

[0028] However, since the human eye has different sensitivities to different regions in an image, the above method does not distinguish whether different regions in the image are regions of interest to the human eye, and cannot perform targeted processing on the image content, resulting in the optimized image quality not meeting the user's expectations.

[0029] Therefore, in view of at least one of the problems existing in the above related art, the present application proposes an image processing method, apparatus, electronic device, chip, and storage medium.

[0030] The image processing method, apparatus, electronic device, chip, and storage medium of the embodiments of the present application will be described below with reference to the accompanying drawings. Before specifically describing the embodiments of the present application, for the convenience of understanding, common technical terms are first introduced:

[0031] RGB color space (or color space): Based on three primary colors, namely red (abbreviated as R), green (abbreviated as G), and blue (abbreviated as B), different degrees of superposition are performed to produce rich and extensive colors, which is commonly known as the three-primary color model.

[0032] YUV color space (or color space): A color model that separates luminance information from chrominance information and is commonly used in video compression and transmission. Among them, Y represents luminance, that is, the luminance information of the image, without color information; U and V represent blue difference (Cb) and red difference (Cr) respectively, that is, chrominance information, which describes the specific hue of the color.

[0033] Figure 1 It is a schematic flowchart of the first image processing method provided by the embodiments of this application.

[0034] It should be noted that the image processing method of the embodiments of this application can be applied to an image processing device. In some possible embodiments, the image processing device can be configured in an electronic device or a chip so that the electronic device or the chip can perform image processing functions. Additionally, in some possible embodiments, the image processing device can also be software in an electronic device, etc.

[0035] In any one of the embodiments of this application, the chip can be integrated into an electronic device. The chip includes a central processing unit (CPU for short), an image signal processing (ISP for short), an application-specific integrated circuit (ASIC for short), a digital signal processor (DSP for short), a field-programmable gate array (FPGA for short), a system on chip (SOC for short), a reduced instruction set computer (RISC for short), etc., and will not be listed one by one here.

[0036] Among them, the electronic device includes but is not limited to: terminals, servers, etc. Among them, a terminal is an entity on the user side for receiving or transmitting signals, such as a mobile phone. A terminal can also be referred to as a terminal device (terminal), user equipment (user equipment, abbreviated as UE), mobile station (mobile station, abbreviated as MS), mobile terminal device (mobile terminal, abbreviated as MT), etc. The terminal can be a car with communication function, intelligent car, mobile phone (mobile phone), wearable device, tablet computer (Pad), computer with wireless transceiver function, virtual reality (virtual reality, abbreviated as VR) terminal, augmented reality (augmented reality, abbreviated as AR) terminal, wireless terminal in industrial control (industrial control), wireless terminal in self-driving (self-driving), wireless terminal in remote medical surgery (remote medical surgery), wireless terminal in smart grid (smart grid), wireless terminal in transportation safety (transportation safety), wireless terminal in smart city (smart city), wireless terminal in smart home (smart home), and so on. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the terminal.

[0037] As Figure 1 shown, the image processing method may include the following steps S101 to S102:

[0038] Step S101, in response to a trigger operation, obtain an image to be processed; wherein, any pixel point in the image to be processed has a corresponding probability, and the probability is used to indicate the possibility that any pixel point belongs to the region of interest.

[0039] Among them, the trigger operation includes but is not limited to: shooting operation, display operation, upload operation, etc.

[0040] Among them, the image to be processed includes but is not limited to: the captured shooting image, the new image generated according to the shooting image, a certain video frame in the captured video stream, the new image generated according to the video frame, etc.

[0041] Among them, each pixel point in the image to be processed has a corresponding probability, and the probability is used to indicate the possibility that the corresponding pixel point belongs to the region of interest. Exemplarily, machine learning technology or deep learning technology can be used to predict the region of interest of the image to be processed to obtain a mask image (Mask), where each pixel point in the mask image is used to indicate the probability that the corresponding pixel point in the image to be processed is the region of interest.

[0042] As a possible implementation, when the trigger operation includes a shooting operation, the captured image collected by an electronic device (such as a terminal) can be used as the image to be processed. Alternatively, an image to be processed can be generated based on the captured image. Exemplarily, taking the captured image as an image in the RGB color space as an example, the captured image can be format-converted to obtain an image to be processed in another color space (such as the YUV color space).

[0043] As another possible implementation, when the trigger operation includes a display operation, the image to be displayed in the electronic device (such as a terminal) can be used as the image to be processed.

[0044] Among them, there is no restriction on the acquisition method of the image to be displayed. Exemplarily, the image to be displayed can be an image captured by a camera, or the image to be displayed can be a synthetic image, or the image to be displayed can be an online-captured image, or the image to be displayed can be an image obtained from a training set or a test set, etc. The embodiments of the present application do not limit this.

[0045] As yet another possible implementation, when the trigger operation includes an upload operation, the image sent from the client to the server can be used as the image to be processed. Alternatively, an image to be processed can be generated based on the image sent from the client to the server.

[0046] As still another possible implementation, when the trigger operation includes an upload operation, a video frame in the video stream sent from the client to the server can be used as the image to be processed. Alternatively, an image to be processed can be generated based on the video frame.

[0047] For example, the video stream sent from the client to the server can be an encoded video stream. The server can decode the video stream to obtain the source video stream before encoding, and use each video frame or key video frame in the source video stream as the image to be processed.

[0048] It should be noted that the above trigger operation and the acquisition method of the image to be processed are only exemplary descriptions, but the present application is not limited thereto. In actual applications, other trigger operations can also be used to obtain the image to be processed, and the embodiments of the present application do not limit this.

[0049] Step S102: Adjust the pixel values of the pixel points in the image to be processed based on probability to obtain a first target image with optimized image quality.

[0050] In the embodiments of the present application, the pixel values of the pixel points in the image to be processed can be adjusted based on the probabilities corresponding to the respective pixel points in the image to be processed to obtain a first target image with optimized image quality.

[0051] In any one embodiment of the present application, when the trigger operation includes a shooting operation, preview can also be performed based on the first target image. Exemplarily, taking the execution entity of the present application as a terminal and the image format of the image to be processed as the YUV format as an example, the first target image can be converted into an RGB format image for preview.

[0052] In any one embodiment of the present application, when the trigger operation includes a display operation, image rendering can also be performed based on the first target image for display. Exemplarily, taking the execution entity of the present application as a terminal and the image format of the image to be processed as the YUV format as an example, the first target image can be converted into the RGB format for image rendering for display.

[0053] In any one embodiment of the present application, when the trigger operation includes an upload operation, transcoding processing can also be performed based on the first target image. Exemplarily, taking the execution entity of the present application as a server as an example, the server can perform transcoding processing on the first target image to adapt to various types of terminals.

[0054] The image processing method of the embodiments of the present application takes into account that the human eye has different sensitivities to different regions in an image. Based on the regions of interest sensitive to the human eye, targeted processing is performed on each pixel point in the image to be processed, which can make the processed image look more prominent, thereby improving the image quality and bringing a better visual experience to the user.

[0055] The embodiments of the present application provide another image processing method. Figure 2 It is a schematic flowchart of the second image processing method provided by the embodiments of the present application.

[0056] It should be noted that this image processing method can be executed alone, or can be executed in combination with any one embodiment or possible implementation manner in the present application, or can also be executed in combination with any one technical solution in the related art. The embodiments of the present application do not limit this.

[0057] As Figure 2 shown, this image processing method may include the following steps S201 to S204:

[0058] Step S201, in response to a trigger operation, obtain an image to be processed.

[0059] It should be noted that the explanation of step S201 can refer to the relevant description in any one embodiment of the present application and will not be elaborated here.

[0060] Step S202, obtain a mask image and local histograms of multiple image blocks in the image to be processed.

[0061] In the embodiments of the present application, deep learning technology or machine learning technology can be adopted to predict the region of interest for the image to be processed, and a mask image is obtained. Each pixel point in the mask image is used to indicate the probability that the corresponding pixel point in the image to be processed is the region of interest, where the probability is used to indicate the possibility that the corresponding pixel point belongs to the region of interest.

[0062] As an example, the image to be processed can be an image in a target color space. The target component can be extracted from multiple color components corresponding to the target color space in the image to be processed, and the region of interest is predicted based on the target component to obtain the mask image.

[0063] Exemplarily, taking the YUV color space as the target color space as an example, the multiple color components include the Y component, the U component, and the V component. The Y component can be used as the target component, and the Y component of the image to be processed is input into a neural network (or other processing unit) to obtain the output mask image Mask. Each pixel point in Mask is used to indicate the probability that the corresponding pixel point in the image to be processed is the region of interest, and the value range of the probability is (0, 1). Thus, only the Y component of the image needs to be processed, which can improve the image processing speed.

[0064] In the embodiments of the present application, histogram statistics can also be performed on multiple image blocks in the image to be processed to obtain the local histogram of each image block, where the local histogram is used to indicate the number of pixel points with the same pixel value in the image block.

[0065] As an example, the image to be processed can be divided into blocks to obtain multiple image blocks, and histogram statistics are performed on the target components (such as the Y component) of the multiple image blocks to obtain the local histograms of the multiple image blocks; where the local histogram is used to indicate the number of pixel points with the same pixel value in the image block. Exemplarily, the s-th image block is marked as I s , I s 's local histogram can be Hist s .

[0066] As another example, normalization processing can be performed on each pixel point in the image to be processed, and the image to be processed after normalization processing is divided into blocks to obtain multiple image blocks. Thus, in the present application, histogram statistics can be performed on the target components of the multiple image blocks to obtain the local histograms of the multiple image blocks; where the local histogram is used to indicate the number of pixel points with the same pixel value in the image block.

[0067] Exemplarily, taking the Y component as the target component as an example, marking the image to be processed as I, the pixel value of the pixel point in I as i, and the maximum pixel value as M, then the pixel value t of the pixel point in the image to be processed after normalization processing is: For the normalized image, it can be divided into S h ×S w image blocks I s (i = 1, 2, …, S h ×S w ), that is, the normalized I is horizontally divided into S w image blocks and vertically divided into S h image blocks. For the s-th image block I s , perform histogram statistics on the Y component to obtain the local histogram Hist s of the s-th image block I s .

[0068] For example, for an image block I s with a length of W and a height of H, the total number of its pixel points is N = W × H. Let be the number of pixel points with pixel value i in the s-th image block I s , then there is:

[0069]

[0070] where L is the maximum pixel value corresponding to the pixel points in the normalized image I to be processed.

[0071] At this time, the local histogram Hist s of the s-th image block I s can be expressed as:

[0072]

[0073] Step S203: Determine the target mapping curves of multiple image blocks according to the mask image and the local histograms of multiple image blocks.

[0074] Among them, the target mapping curve is used to adjust the local tone (such as contrast and brightness) of the corresponding image block.

[0075] In the embodiments of the present application, the target mapping curves (such as LTM curves) of multiple image blocks can be calculated according to the mask image Mask and the local histograms of multiple image blocks. Exemplarily, based on the mask image, the local histograms of multiple image blocks can be reshaped to obtain the target mapping curves of multiple image blocks.

[0076] In any embodiment of the present application, the probability distribution of each image block can be calculated according to the mask image. Among them, the probability distribution of each image block is used to indicate the probability mean of each pixel point with the same pixel value in the image block. In the present application, according to the probability distribution of each image block, the local histogram of each image block can be reshaped specifically to obtain the target mapping curve of each image block.

[0077] Exemplarily, the mask image Mask can be divided into S h ×S w image patches M s , where M s corresponds to I s one by one. In this application, for the s-th image patch M s , first, various pixel points with the same pixel value can be determined from I s (such as marked as pixel point A). After that, the pixel values of the pixel points corresponding to each pixel point A in M s (i.e., the probability of pixel point A) can be accumulated and then averaged to obtain the probability mean value of pixel point A. For example, if the probability distribution of M s (or I s ) is Per s , then Per s can be used to reshape the local histogram Hist s of I s to obtain the target mapping curve of I s .

[0078] Step S204: Based on the target mapping curves of multiple image patches, perform mapping processing on each pixel point in the image to be processed to obtain a first target image.

[0079] In the embodiments of this application, according to the target mapping curves of multiple image patches, targeted mapping processing can be performed on each pixel point in the image to be processed to obtain a first target image with optimized image quality.

[0080] The image processing method of the embodiments of this application can calculate the target mapping curves (such as the LTM curve) of each image patch based on the local histograms and image masks of the image patches in the image to be processed, and perform targeted processing on each pixel point in the image to be processed based on the target mapping curves of each image patch, which can make the processed image look more prominent, thereby improving the image quality and bringing a better visual experience to the user.

[0081] The embodiments of this application provide another image processing method, Figure 3 which is a schematic flowchart of the third image processing method provided by the embodiments of this application.

[0082] It should be noted that this image processing method can be executed alone, or can be executed together with any one of the embodiments or possible implementation manners in this application, or can also be executed together with any one of the technical solutions in the related art. The embodiments of this application do not limit this.

[0083] Such asFigure 3 As shown in Figure 3 , the image processing method may include the following steps S301 to S305:

[0084] Step S301: In response to a trigger operation, obtain an image to be processed.

[0085] Step S302: Obtain a mask image and local histograms of multiple image blocks in the image to be processed.

[0086] Among them, each pixel point in the mask image is used to indicate the probability that the corresponding pixel point in the image to be processed is an area of interest.

[0087] Step S303: Determine the probability distribution of multiple image blocks according to the mask image; where the probability distribution is used to indicate the probability mean of each pixel point with the same pixel value in the image block.

[0088] It should be noted that the explanations of steps S301 to S303 can be referred to the relevant descriptions in any embodiment of this application, and will not be elaborated here.

[0089] As an example, the mask image Mask can be divided into S h ×S w image blocks M s , where M s corresponds to I s one by one. In this application, for the s-th image block M s , first, each pixel point with the same pixel value (such as marked as pixel point A) can be determined from I s . Then, the pixel values (i.e., the probability of pixel point A) of the pixel points in M s corresponding to each pixel point A can be accumulated and averaged to obtain the probability mean of pixel point A.

[0090] For example, for an image block M s with a length of W and a height of H, the total number of its pixel points is N = W × H. Let per i be the mean of the pixel values (i.e., the probability of pixel point A) of the pixel points (such as marked as pixel point B) in the s-th image block M s corresponding to each pixel point A with a pixel value of i. Then the probability distribution Per s of M s (or I s ) can be expressed as:

[0091] Per s = per i , i ∈ [0, L]; (3)

[0092] Step S304: Reshape the local histograms of multiple image patches according to the probability distributions of the multiple image patches to obtain the target mapping curves of the multiple image patches.

[0093] Among them, the target mapping curve is used to adjust the local tone (such as contrast and brightness) of the corresponding image patch.

[0094] In the embodiment of the present application, the local histogram of each image patch can be specifically reshaped according to the probability distribution of each image patch to obtain the target mapping curve of each image patch. That is, for the s-th image patch I s , Per s can be used to s reshape the local histogram Hist s of I s to obtain the target mapping curve of I

[0095] In any embodiment of the present application, the target mapping curves of multiple image patches can be reshaped by the following steps A to C:

[0096] Step A: For any one of the multiple image patches, reshape the local histogram of the image patch according to the probability distribution of the image patch to obtain the target histogram of the image patch.

[0097] As an example, the local histogram corresponding to the image patch can be multiplied by the probability distribution to obtain the weighted histogram of the image patch, and the mean value of the weighted histogram of the image patch can be calculated to obtain the intermediate histogram of the image patch. Thus, in the present application, the target histogram of the image patch can be determined according to the difference between the local histogram corresponding to the image patch and the intermediate histogram.

[0098] Exemplarily, the following formula (4) can be used to calculate the weighted histogram of the s-th image patch I s , the following formula (5) can be used to calculate the intermediate histogram of the s-th image patch I s , and the following formulas (6) and (7) can be used to calculate the target histogram of the s-th image patch I s

[0099]

[0100] Among them, represents the number of pixel points with pixel value i in the weighted histogram; represents the number of pixel points with pixel value i in the intermediate histogram.

[0101] Step B: Generate the histogram accumulation function of the image patch according to the target histogram of the image patch.

[0102] As an example, the following formula (8) can be adopted to calculate the histogram accumulation function of the s-th image block I s :

[0103]

[0104] Step C: Normalize the histogram accumulation functions of multiple image blocks to obtain the target mapping curves of multiple image blocks.

[0105] As an example, taking the target mapping curve as the LTM curve for illustration, mark the target mapping curve of the s-th image block I s as LTMCurve s , then there is:

[0106]

[0107] Step S305: Based on the target mapping curves of multiple image blocks, perform mapping processing on each pixel point in the image to be processed to obtain a first target image.

[0108] It should be noted that the explanation of step S305 can refer to the relevant description in any embodiment of this application, and will not be elaborated here.

[0109] In the image processing method of the embodiment of this application, according to the probability that each pixel point indicated by the mask image is an area of interest, the probability distribution of each image block is determined, and based on the probability distribution of each image block, the local histogram of each image block is reshaped specifically, which can improve the accuracy and reliability of the target mapping curve of each reshaped image block, and further improve the processing quality of the image.

[0110] The embodiment of this application provides another image processing method, Figure 4 which is a schematic flowchart of the fourth image processing method provided by the embodiment of this application.

[0111] It should be noted that this image processing method can be executed alone, or can also be executed in combination with any one embodiment or possible implementation manner in this application, or can also be executed in combination with any one technical solution in related technologies. The embodiment of this application does not limit this.

[0112] As Figure 4 shown, this image processing method may include the following steps S401 to S406:

[0113] Step S401: In response to a trigger operation, obtain an image to be processed.

[0114] Step S402: Obtain the masked image and the local histograms of multiple image patches in the image to be processed.

[0115] Among them, each pixel in the masked image is used to indicate the probability that the corresponding pixel in the image to be processed is an area of interest.

[0116] Step S403: Determine the target mapping curves of multiple image patches according to the masked image and the local histograms of multiple image patches.

[0117] Among them, the target mapping curve is used to adjust the local tone (such as contrast and brightness) of the corresponding image patch.

[0118] It should be noted that the explanations of steps S401 to S403 can be referred to the relevant descriptions in any embodiment of this application, and will not be elaborated here.

[0119] Step S404: For any pixel in the image to be processed, query whether there is an adjacent image patch to the pixel in multiple image patches. If so, execute step S405; if not, execute step S406.

[0120] In the embodiment of this application, for any pixel in the image to be processed, it can be queried whether there is an adjacent image patch to the pixel in multiple image patches in the image to be processed.

[0121] As an example, if the pixel is inside the image patch it belongs to, that is, the pixel is not on the boundary of the image patch it belongs to, then there is no adjacent image patch to the pixel.

[0122] As another example, if the pixel is on the boundary of the image to be processed and not at the vertex of the image patch it belongs to, then there is no adjacent image patch to the pixel.

[0123] As yet another example, if the pixel is at the vertex of the image patch it belongs to and not on the boundary of the image to be processed: when the pixel is at the upper left corner of the image patch it belongs to, the left image patch, the upper image patch, and the upper left image patch of the image patch the pixel belongs to can be selected as the adjacent image patches to the pixel; when the pixel is at the upper right corner of the image patch it belongs to, the right image patch, the upper image patch, and the upper right image patch of the image patch the pixel belongs to can be selected as the adjacent image patches to the pixel; when the pixel is at the lower left corner of the image patch it belongs to, the left image patch, the lower image patch, and the lower left image patch of the image patch the pixel belongs to can be selected as the adjacent image patches to the pixel; when the pixel is at the lower right corner of the image patch it belongs to, the right image patch, the lower image patch, and the lower right image patch of the image patch the pixel belongs to can be selected as the adjacent image patches to the pixel.

[0124] As yet another example, if the pixel is located at the boundary of the image block it belongs to, and not at the boundary of the image to be processed, and not at the vertex angle of the image block it belongs to: When the pixel is located at the left boundary of the image block it belongs to, the left image block of the image block where the pixel is located can be selected as the image block adjacent to the pixel; when the pixel is located at the right boundary of the image block it belongs to, the right image block of the image block where the pixel is located can be selected as the image block adjacent to the pixel; when the pixel is located at the upper boundary of the image block it belongs to, the upper image block of the image block where the pixel is located can be selected as the image block adjacent to the pixel; when the pixel is located at the lower boundary of the image block it belongs to, the lower image block of the image block where the pixel is located can be selected as the image block adjacent to the pixel.

[0125] As yet another example, if the pixel is located at the boundary of the image to be processed and at the vertex angle of the image block it belongs to, then the pixel may or may not have an adjacent image block: When the block where the pixel is located is at the vertex angle of the image to be processed, such as the upper left corner, if the pixel is located at the upper left corner of the image block it belongs to, then the pixel has no adjacent image block; if the pixel is located at the upper right corner of the image block it belongs to, then the right image block of the image block where the pixel is located can be used as the image block adjacent to the pixel; if the pixel is located at the lower left corner of the image block it belongs to, then the lower image block of the image block where the pixel is located can be used as the image block adjacent to the pixel; if the pixel is located at the lower right corner of the image block it belongs to, then the right image block, lower image block, and lower right image block of the image block where the pixel is located can be used as the image blocks adjacent to the pixel.

[0126] It should be noted that step S405 and step S406 are two parallel implementation methods. In actual application, only one of them needs to be executed.

[0127] Step S405: Map the pixel value of any pixel according to the target mapping curve of the adjacent image block and the image block where the any pixel is located, so as to obtain the first pixel value of the first pixel corresponding to the any pixel in the first target image.

[0128] In an embodiment of the present application, for any pixel point in the image to be processed, when there are image blocks adjacent to this pixel point among multiple image blocks, the pixel value of this pixel point can be mapped according to the target mapping curve of the adjacent image block and the image block where this pixel point is located at the same time, so as to obtain the pixel value of the pixel point (denoted as the first pixel point in the present application) corresponding to this pixel point in the first target image, which is denoted as the first pixel value in the present application.

[0129] As a possible implementation manner, for any pixel point in the image to be processed, the first pixel value of the first pixel point corresponding to this pixel point in the first target image can be calculated by the following steps D to F:

[0130] Step D: For any pixel point p in the image to be processed, use the target mapping curve of the image block where this pixel point p is located to map the pixel value of this pixel point p, and obtain the first mapping value of this pixel point p.

[0131] Exemplarily, mark the first mapping value of this pixel point p as

[0132] Step E: Use the target mapping curve of the image block adjacent to this pixel point p to map the pixel value of this pixel point p, and obtain the second mapping value of this pixel point p.

[0133] Exemplarily, mark the second mapping value of this pixel point p as where l refers to the number or serial number of the image block adjacent to this pixel point p, and its value range is: l = 1, 2, 3.

[0134] Step F: Determine the first pixel value of the first pixel point corresponding to this pixel point p in the first target image according to the first mapping value and the second mapping value of this pixel point p.

[0135] In any embodiment of the present application, the first mapping value and the second mapping value of this pixel point p can be weighted and summed to obtain the first pixel value of the first pixel point corresponding to this pixel point p in the first target image.

[0136] As an example, the distance between the pixel point p and its adjacent image block (denoted as the first distance in this application) can be calculated, and the distance between the pixel point p and the image block where it is located (denoted as the second distance in this application) can be calculated. Then, according to the first distance of the image block adjacent to the pixel point p, the weighted weight of the image block adjacent to the pixel point p (denoted as the first weight in this application) can be calculated, and according to the second distance of the image block where the pixel point p is located, the weighted weight of the image block where the pixel point p is located (denoted as the second weight in this application) can be calculated. Thus, in this application, according to the first weight of the image block adjacent to the pixel point p and the second weight of the image block where the pixel point p is located, the first mapping value and the second mapping value of the pixel point p can be weighted and summed to obtain the first pixel value of the first pixel point corresponding to the pixel point p in the first target image.

[0137] Among them, the first distance and the first weight are negatively correlated, that is, the smaller the first distance, the larger the first weight, and vice versa, the larger the first distance, the smaller the first weight.

[0138] Among them, the second distance and the second weight are also negatively correlated.

[0139] Exemplarily, mark the second distance between the pixel point p and the image block where it is located as distance0, and the first distance between the pixel point p and the l-th adjacent image block as distance l , and its value range is: l = 1, 2, 3, and the first pixel value of the first pixel point is p o , then the following formulas (10) and (11) can be used to calculate p o :

[0140] w j = 1.0 - distance j ; (10)

[0141]

[0142] Among them, R = 0, 1, 2, 3. When j = 0, w0 is the second weight of the image block where the pixel point is located. When j is not 0, w j is the first weight of the j-th adjacent image block.

[0143] Step S406, according to the target mapping curve of the image block where any pixel point is located, perform mapping processing on the pixel value of any pixel point to obtain the first pixel value of the first pixel point corresponding to any pixel point in the first target image.

[0144] In an embodiment of the present application, for any pixel point p in the image to be processed, in the case where there is no image block adjacent to the pixel point p among multiple image blocks, the pixel value of the pixel point p can be directly mapped based on the target mapping curve of the image block where the pixel point p is located, to obtain the first mapped value of the pixel point p, and the first mapped value of the pixel point p is used as the first pixel value p of the first pixel point corresponding to the pixel point p in the first target image. o .

[0145] In any embodiment of the present application, in order to avoid inter-frame jitter, the first pixel value of the first pixel point can be further updated based on the second pixel value of the second pixel point corresponding to the first pixel point in the previous frame image, so as to obtain the final pixel value of the first pixel point in the first target image, which is denoted as the target pixel value in the present application.

[0146] That is, in the present application, the second pixel value of the second pixel point corresponding to the first pixel point in the second target image can be obtained, where the second target image is obtained by performing mapping processing on the previous frame image of the image to be processed. Then, based on the second pixel value of the second pixel point, the first pixel value of the first pixel point can be updated to obtain the target pixel value of the first pixel point in the first target image. Exemplarily, the first pixel value of the first pixel point and the second pixel value of the second pixel point can be fused to obtain the target pixel value of the first pixel point.

[0147] Among them, the method of performing mapping processing on the previous frame image is similar to the method of performing mapping processing on the image to be processed, and will not be elaborated here.

[0148] Exemplarily, based on formula (11), formula (12) can be adopted to calculate the final mapped value of the first pixel point, which is denoted as the target pixel value p in the present application. out :

[0149] p out =α×p o +(1 - α)×p o_prev ,α∈[0,1]; (12)

[0150] Among them, p o_prev refers to the second pixel value (or the final mapped value) of the second pixel point, and α refers to the fusion coefficient.

[0151] The image processing method of the embodiment of the present application simultaneously performs mapping processing on the pixel value of each pixel point based on the target mapping curve of the image block where the pixel point is located and the target mapping curve of its adjacent image block, which can improve the image quality and visual effect after mapping processing.

[0152] In any embodiment of the present application, the solution provided by the present application is mainly applied to the display field and the image rendering field, and can be extended to image processing on the server, such as being applied to video transcoding on the server, or can also be extended to the ISP side and applied to the photographing process. According to the region of interest of the human eye and the local statistical information of the image, the tone (such as contrast, brightness) of the image is enhanced to bring a better visual experience to the user. Taking the image format to be enhanced as the YUV format as an example, the implementation principle of image enhancement can be as follows Figure 5 as shown, mainly including the following steps:

[0153] Step a: If the high dynamic range (HDR) video stream to be enhanced is from the server side, the HDR video stream can be decoded by a decoder to obtain an image in YUV format; if the image to be enhanced is in RGB format from the ISP side, an RGB2YUV conversion matrix can be used to convert the RGB format image into a YUV format image.

[0154] Among them, the encoding standards of HDR video streams include but are not limited to: High Efficiency Video Coding (HEVC or H.265 for short), AV1 (AOMedia Video 1), Advanced Video Coding (AVC or H.264 for short), etc.

[0155] Optionally, the YUV format image I can be normalized. For example, if the pixel value in I is marked as i and the maximum pixel value is M, then the normalized pixel value of i is

[0156] Step b: Input the Y component of the nth frame image I into a neural network (or other processing unit) to obtain a mask image Mask, where each pixel point in Mask is used to indicate the probability that the corresponding pixel point in I is a region of interest.

[0157] Step c: Divide I and Mask into the same number of image blocks, where the image blocks in I and Mask correspond one by one. For example, divide them into 4×6 image blocks.

[0158] Step d: For each image block I in I s statistical histogram of the Y component is obtained to get the local histogram of each image block I s of.

[0159] For example, for the image I, it is divided into S h ×S w image blocks I s, and perform histogram statistics on the luminance component of each image block I s to obtain the local histogram of I s : For an image block I s with a length of W and a height of H, the total number of its pixels is N = W × H. Let be the number of pixels with pixel value i in the s-th image block I s , then: At this time, the local histogram of the s-th image block I s can be expressed as:

[0160] Step e: Calculate the probability distribution Per s of the image block M s in Mask. Based on the local histogram Hist s of I s and the probability distribution Per s (or I s ) of M s , calculate the LTM curve of I s .

[0161] First, for the mask image Mask, Mask can be divided into S h ×S w image blocks M s . In each M s , accumulate and average the probabilities corresponding to the pixels with the same pixel value in I s to obtain the probability distribution of M s or I s : For an image block M s with a length of W and a height of H, the total number of its pixels is N = W × H. Let per i be the average probability corresponding to the pixels with pixel value i in the s-th image block M s . The probability distribution Per s or I s can be expressed as: Per s = per s , i ∈ [0, L]. i

[0162] After that, the probability distribution Per s of I s can be used to reshape the local histogram Hist s of I s to obtain the adjusted histogram, which is denoted as the target histogram in this application where the reshape formula includes but is not limited to the following formula:

[0163] ​

[0164] After that, the target histogram of I s can be accumulated to obtain the histogram accumulation function HistSum : s :

[0165]

[0166] Finally, HistSum s can be normalized to obtain the LTM curve LTMCurve of I s : s :

[0167]

[0168] Step f: For each pixel point p in each image block, the pixel value of the pixel point p is mapped by using the LTM curves of the image block where the pixel point p is located and the image blocks adjacent to the pixel point p, to obtain a set of mapped values

[0169] First, for any pixel point p in I, the pixel value of the pixel point p is mapped by using the LTM curve of the image block where the pixel point p is located, to obtain:

[0170] After that, it can be determined whether there are image blocks adjacent to the pixel point p among multiple image blocks. If not, is used as the finally mapped pixel value p o corresponding to the pixel point p. If so, the pixel value of the pixel point p is mapped by using the LTM curve of the adjacent image block, to obtain:

[0171] Step g: The output set of mapped values is weighted and averaged to obtain the finally mapped pixel value p o (i.e., the initial mapping result of the nth frame):

[0172] w j = 1.0 - distance j ;

[0173]

[0174] where R = 0, 1, 2, 3, and distance j is the normalized distance between the pixel point p and the adjacent image block or the image block where it is located.

[0175] Optionally, after step g, subsequent step h may further be performed.

[0176] Step h: Temporal smoothing: Perform weighted averaging on the final pixel value p of the pixel point p and the pixel point at the same position in the previous frame o to obtain the final pixel value p of the pixel point p o_prev (i.e., the final mapping result of the (n-1)th frame). out (i.e., the final mapping result of the nth frame).

[0177] That is, to avoid jitter between frames, the mapping value p of the pixel point p in the current frame o is fused with the mapping value p of the pixel point at the same position in the previous frame o_prev to obtain the final mapping value p out :

[0178] p out = α × p o + (1 - α) × p o_prev , where α ∈ [0, 1];

[0179] where α is the fusion coefficient.

[0180] In summary, considering that the human eye has different sensitivities to different regions in an image, by adjusting the regions of interest in the image that are sensitive to the human eye, the adjusted image can look more prominent and a better image quality can be obtained.

[0181] To implement the above embodiments, an image processing apparatus is further proposed in an embodiment of the present application.

[0182] Figure 6 It is a schematic structural diagram of an image processing apparatus provided in an embodiment of the present application.

[0183] As Figure 6 shown, the image processing apparatus 600 may include: an acquisition module 610 and an adjustment module 620.

[0184] Among them, the acquisition module 610 is configured to obtain an image to be processed in response to a trigger operation; any pixel point in the image to be processed has a corresponding probability, and the probability is used to indicate the possibility that any pixel point belongs to a region of interest;

[0185] The adjustment module 620 is configured to adjust the pixel values of the pixel points in the image to be processed based on the probability to obtain a first target image with optimized image quality.

[0186] Further, in an implementation manner of an embodiment of the present application, the trigger operation includes a shooting operation, the image to be processed is a captured image obtained by acquisition, or the image to be processed is generated based on the captured image; the image processing apparatus 600 may further include:

[0187] A preview module for previewing based on a first target image.

[0188] In an implementation manner of the embodiment of the present application, the trigger operation includes a display operation, the image to be processed is an image to be displayed, and the image processing device 600 may further include:

[0189] A rendering module for rendering an image based on the first target image for display.

[0190] In an implementation manner of the embodiment of the present application, the trigger operation includes an upload operation in which the client sends the image to be processed to the server, or the trigger operation includes an upload operation in which the client sends a video stream to the server, and the image to be processed is a video frame in the video stream; the image processing device 600 may further include:

[0191] A transcoding module for transcoding with the first target image.

[0192] In an implementation manner of the embodiment of the present application, the adjustment module 620 is configured to: determine a target mapping curve of a plurality of image blocks according to local histograms of the plurality of image blocks in the mask image and the image to be processed; wherein the target mapping curve is used to adjust the local tone of the corresponding image block; each pixel point in the mask image is used to indicate the probability that the corresponding pixel point in the image to be processed is a region of interest; based on the target mapping curves of the plurality of image blocks, perform mapping processing on each pixel point in the image to be processed to obtain the first target image.

[0193] In an implementation manner of the embodiment of the present application, the adjustment module 620 is configured to: determine a probability distribution of a plurality of image blocks according to the mask image; wherein the probability distribution is used to indicate the probability mean of each pixel point with the same pixel value in the image block; according to the probability distribution of the plurality of image blocks, reshape the local histograms of the plurality of image blocks to obtain the target mapping curves of the plurality of image blocks.

[0194] In an implementation manner of the embodiment of the present application, the adjustment module 620 is configured to: reshape the local histograms of the plurality of image blocks according to the probability distribution of the plurality of image blocks to obtain the target histograms of the plurality of image blocks; generate a histogram accumulation function of the plurality of image blocks according to the target histograms of the plurality of image blocks; perform normalization processing on the histogram accumulation function of the plurality of image blocks to obtain the target mapping curves of the plurality of image blocks.

[0195] In an implementation manner of the embodiment of the present application, the adjustment module 620 is configured to: for any one of a plurality of image blocks, multiply the local histogram corresponding to the any one of the image blocks by a probability distribution to obtain a weighted histogram of the any one of the image blocks; calculate the mean value of the weighted histogram of the any one of the image blocks to obtain an intermediate histogram of the any one of the image blocks; and determine the target histogram of the any one of the image blocks according to the difference between the local histogram corresponding to the any one of the image blocks and the intermediate histogram.

[0196] In an implementation manner of the embodiment of the present application, the adjustment module 620 is configured to: for any pixel point in the image to be processed, query whether there is an image block adjacent to the any pixel point among a plurality of image blocks; in response to the existence of an adjacent image block, perform mapping processing on the pixel value of the any pixel point according to the target mapping curve of the adjacent image block and the image block where the any pixel point is located to obtain the first pixel value of the first pixel point corresponding to the any pixel point in the first target image; or, in response to the non-existence of an adjacent image block, perform mapping processing on the pixel value of the any pixel point according to the target mapping curve of the image block where the any pixel point is located to obtain the first pixel value of the first pixel point.

[0197] In an implementation manner of the embodiment of the present application, the adjustment module 620 is configured to: perform mapping processing on the pixel value of the any pixel point by using the target mapping curve of the image block where the any pixel point is located to obtain a first mapping value of the any pixel point; perform mapping processing on the pixel value of the any pixel point by using the target mapping curve of the adjacent image block to obtain a second mapping value of the any pixel point; and determine the first pixel value of the first pixel point according to the first mapping value and the second mapping value of the any pixel point.

[0198] In an implementation manner of the embodiment of the present application, the adjustment module 620 is configured to: obtain a first distance between the any pixel point and the adjacent image block, and a second distance between the any pixel point and the image block where the any pixel point is located; determine a first weight of the adjacent image block according to the first distance of the adjacent image block, and determine a second weight of the image block where the any pixel point is located according to the second distance; and perform weighted summation on the first mapping value and the second mapping value of the any pixel point according to the first weight of the adjacent image block and the second weight of the image block where the any pixel point is located to obtain the first pixel value of the first pixel point.

[0199] In an implementation manner of the embodiment of the present application, the image processing apparatus 600 may further include:

[0200] An update module, configured to obtain a second pixel value of a second pixel corresponding to a first pixel in a second target image; wherein, the second target image is obtained by performing a mapping process on a previous frame image of the image to be processed; based on the second pixel value of the second pixel, update the first pixel value of the first pixel to obtain a target pixel value of the first pixel in the first target image.

[0201] In an implementation manner of the embodiment of the present application, the image to be processed is an image in a target color space, and the mask image is obtained by using the following module:

[0202] An extraction module, configured to extract a target component from multiple color components corresponding to the target color space in the image to be processed;

[0203] A prediction module, configured to perform an interested region prediction according to the target component to obtain a mask image.

[0204] In an implementation manner of the embodiment of the present application, the local histograms of multiple image blocks are obtained by using the following module:

[0205] A processing module, configured to perform a normalization process on each pixel in the image to be processed;

[0206] A blocking module, configured to block the normalized image to be processed to obtain multiple image blocks;

[0207] A statistics module, configured to perform a histogram statistics on the target components of multiple image blocks to obtain local histograms of multiple image blocks; wherein, the local histogram is used to indicate the number of pixels with the same pixel value in the image block.

[0208] It should be noted that the foregoing explanations of the embodiments of the image processing method also apply to the image processing apparatus of this embodiment, and will not be elaborated here.

[0209] In the image processing apparatus of the embodiment of the present application, considering that the human eye has different sensitivities to different regions in the image, and based on the interested regions sensitive to the human eye, targeted processing is performed on each pixel in the image to be processed, which can make the processed image look more prominent, thereby improving the image quality and bringing a better visual experience to the user.

[0210] To implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the image processing method described in any of the foregoing embodiments.

[0211] Figure 7A schematic structural diagram of an electronic device provided by an embodiment of the present application. For example, the electronic device 700 may be a vehicle, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0212] Referring to Figure 7 , the electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0213] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0214] The memory 704 is configured to store various types of data to support the operation of the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0215] The power component 706 provides power to various components of the electronic device 700. The power component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 700.

[0216] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0217] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 further includes a speaker for outputting audio signals.

[0218] The I / O interface 712 provides an interface between the processing component 702 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0219] The sensor assembly 714 includes one or more sensors for providing an assessment of the status of various aspects of the electronic device 700. For example, the sensor assembly 714 can detect the on / off state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700. The sensor assembly 714 can also detect a change in the position of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and a change in the temperature of the electronic device 700. The sensor assembly 714 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 can also include a light sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0220] The communication component 716 is configured to facilitate communication between the electronic device 700 and other devices in a wired or wireless manner. The electronic device 700 can access a wireless network based on communication standards, such as WiFi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0221] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and is used to execute the above method.

[0222] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions. The above instructions can be executed by a processor 720 of the electronic device 700 to complete the above method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0223] To implement the above embodiments, the present application also proposes a chip. The chip includes an interface circuit and a processing circuit that are coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is configured to execute the image processing method provided in any of the foregoing embodiments.

[0224] Figure 8 It is a schematic structural diagram of a chip proposed in an embodiment of the present application. Reference may be made to Figure 8 the schematic structural diagram of the chip 800 shown, but not limited thereto.

[0225] The chip 800 includes a processing circuit 801, and the processing circuit 801 is configured to execute any of the above image processing methods.

[0226] In some embodiments, the chip 800 further includes one or more interface circuits 802. Optionally, the interface circuit 802 is connected to a memory 803. The interface circuit 802 can be used to receive signals from the memory 803 or other devices, and the interface circuit 802 can be used to send signals to the memory 803 or other devices. For example, the interface circuit 802 can read instructions stored in the memory 803 and send the instructions to the processing circuit 801.

[0227] In some embodiments, the interface circuit 802 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 801 performs other steps.

[0228] In some embodiments, terms such as interface circuit, interface, transceiver pin, transceiver, etc. may be used interchangeably.

[0229] In some embodiments, the chip 800 further includes one or more memories 803 for storing instructions. Optionally, all or part of the memories 803 may be outside the chip 800.

[0230] To implement the above embodiments, the present application also proposes a non-transitory computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the image processing method as described in any of the foregoing method embodiments.

[0231] To implement the above embodiments, the present application also proposes a computer program product, on which a computer program is stored. The computer program implements the image processing method as described in any of the foregoing method embodiments when executed by a processor.

[0232] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0233] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0234] Any process or method description represented in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0235] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing a logical function, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with such instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (Random Access Memory, abbreviated as RAM), a read-only memory (Read-Only Memory, abbreviated as ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (Compact Disc Read-Only Memory, abbreviated as CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then stored in a computer memory.

[0236] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following techniques well known in the art or a combination thereof can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0237] Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0238] In addition, in each embodiment of the present application, the functional units can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0239] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that, Including: Upon triggering an operation, obtain an image to be processed; wherein, any pixel point in the image to be processed has a corresponding probability, and the probability is used to indicate the possibility that the any pixel point belongs to a region of interest; Based on the probability, adjust the pixel values of the pixel points in the image to be processed to obtain a first target image with optimized image quality.

2. The method according to claim 1, wherein The triggering operation includes a shooting operation, the image to be processed is a captured shooting image, or the image to be processed is generated based on the shooting image; The method further includes: Perform a preview based on the first target image.

3. The method according to claim 1, wherein The triggering operation includes a display operation, the image to be processed is an image to be displayed, and the method further includes: Perform image rendering based on the first target image for display.

4. The method according to claim 1, wherein The triggering operation includes an upload operation in which the client sends the image to be processed to the server, or the triggering operation includes an upload operation in which the client sends a video stream to the server, and the image to be processed is a video frame in the video stream; The method further includes: Perform transcoding processing with the first target image.

5. The method according to any one of claims 1 to 4, characterized in that, The adjusting the pixel values of the pixel points in the image to be processed based on the probability to obtain a first target image with optimized image quality includes: Determine target mapping curves of a plurality of image blocks in the mask image and the image to be processed according to local histograms of the plurality of image blocks; wherein, the target mapping curves are used to adjust local hues of corresponding image blocks; each pixel point in the mask image is used to indicate the probability that the corresponding pixel point in the image to be processed is a region of interest; Based on the target mapping curves of the plurality of image blocks, perform mapping processing on each pixel point in the image to be processed to obtain the first target image.

6. The method according to claim 5, characterized in that The determining the target mapping curves of the plurality of image blocks according to the mask image and the local histograms of the plurality of image blocks in the image to be processed includes: Determine a probability distribution of the plurality of image blocks according to the mask image; wherein, the probability distribution is used to indicate the probability mean of pixel points with the same pixel value in the image block; Reshape the local histograms of the plurality of image blocks according to the probability distribution of the plurality of image blocks to obtain the target mapping curves of the plurality of image blocks.

7. The method according to claim 6, wherein The reshaping the local histograms of the plurality of image blocks according to the probability distribution of the plurality of image blocks to obtain the target mapping curves of the plurality of image blocks includes: Reshape the local histograms of the plurality of image blocks according to the probability distribution of the plurality of image blocks to obtain target histograms of the plurality of image blocks; Generate a histogram accumulation function of the plurality of image blocks according to the target histograms of the plurality of image blocks; Perform normalization processing on the histogram accumulation function of the plurality of image blocks to obtain the target mapping curves of the plurality of image blocks.

8. The method according to claim 7, wherein The reshaping the local histograms of the plurality of image blocks according to the probability distribution of the plurality of image blocks to obtain the target histograms of the plurality of image blocks includes: For any one of the multiple image blocks, multiply the local histogram corresponding to the any one of the image blocks by the probability distribution to obtain the weighted histogram of the any one of the image blocks; Calculate the mean of the weighted histogram of the any one of the image blocks to obtain the intermediate histogram of the any one of the image blocks; Determine the target histogram of the any one of the image blocks according to the difference between the local histogram and the intermediate histogram corresponding to the any one of the image blocks.

9. The method according to claim 5, wherein, The mapping process of each pixel point in the image to be processed based on the target mapping curve of the multiple image blocks to obtain the first target image includes: For any one pixel point in the image to be processed, query whether there is an image block adjacent to the any one pixel point among the multiple image blocks; In response to the existence of the adjacent image block, perform a mapping process on the pixel value of the any one pixel point according to the target mapping curves of the adjacent image block and the image block where the any one pixel point is located to obtain the first pixel value of the first pixel point corresponding to the any one pixel point in the first target image; or, In response to the non - existence of the adjacent image block, perform a mapping process on the pixel value of the any one pixel point according to the target mapping curve of the image block where the any one pixel point is located to obtain the first pixel value of the first pixel point.

10. The method according to claim 9, characterized in that, The step of, in response to the existence of the adjacent image block, performing a mapping process on the pixel value of the any one pixel point according to the target mapping curves of the adjacent image block and the image block where the any one pixel point is located to obtain the first pixel value of the first pixel point corresponding to the any one pixel point in the first target image includes: Use the target mapping curve of the image block where the any one pixel point is located to perform a mapping process on the pixel value of the any one pixel point to obtain the first mapping value of the any one pixel point; Use the target mapping curve of the adjacent image block to perform a mapping process on the pixel value of the any one pixel point to obtain the second mapping value of the any one pixel point; Determine the first pixel value of the first pixel point according to the first mapping value and the second mapping value of the any one pixel point.

11. The method according to claim 10, wherein, The step of determining the first pixel value of the first pixel point according to the first mapping value and the second mapping value of the any one pixel point includes: Obtain the first distance between the any one pixel point and the adjacent image block, and the second distance between the any one pixel point and the image block where the any one pixel point is located; Determine the first weight of the adjacent image block according to the first distance of the adjacent image block, and determine the second weight of the image block where the any one pixel point is located according to the second distance; Perform a weighted sum on the first mapping value and the second mapping value of the any one pixel point according to the first weight of the adjacent image block and the second weight of the image block where the any one pixel point is located to obtain the first pixel value of the first pixel point in the first target image.

12. The method according to claim 9, wherein The method further includes: Obtain the second pixel value of the second pixel point corresponding to the first pixel point in the second target image; wherein, the second target image is obtained by performing a mapping process on the previous frame image of the image to be processed. Update the first pixel value of the first pixel point based on the second pixel value of the second pixel point to obtain the target pixel value of the first pixel point in the first target image.

13. The method according to claim 5, wherein The image to be processed is an image in a target color space. The mask image is obtained by the following steps: Extract a target component from multiple color components corresponding to the target color space in the image to be processed; Perform region of interest prediction based on the target component to obtain the mask image.

14. The method according to claim 13, wherein The local histograms of the multiple image blocks are obtained by the following steps: Normalize each pixel point in the image to be processed; Divide the normalized image to be processed into blocks to obtain the multiple image blocks; Perform histogram statistics on the target components of the multiple image blocks to obtain the local histograms of the multiple image blocks; wherein, the local histogram is used to indicate the number of pixel points with the same pixel value in the image block.

15. An image processing apparatus, characterized in that, Comprising: An acquisition module, configured to acquire an image to be processed in response to a trigger operation; wherein, any pixel point in the image to be processed has a corresponding probability, and the probability is used to indicate the possibility that the any pixel point belongs to a region of interest; An adjustment module, configured to adjust the pixel value of the pixel points in the image to be processed based on the probability to obtain a first target image with optimized image quality.

16. An electronic device, characterized in that, Comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the method according to any one of claims 1 to 14 are implemented.

17. A chip, characterized in that, The chip includes an interface circuit and a processing circuit coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is used to implement the method according to any one of claims 1 to 14.

18. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instruction is executed by the processor, the steps of the method according to any one of claims 1 to 14 are implemented.

19. A computer program product, characterized in that, Comprising a computer program, when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images