Image processing method, device, terminal device, storage medium and program product

Abstract

The application relates to an image processing method and device, a terminal device, a storage medium and a program product. The method is applied to a first image processing model on a terminal device and comprises the following steps: splitting a to-be-processed image into multiple partitions based on partition parameters; performing histogram statistics on each partition to obtain a histogram corresponding to each partition; determining a mapping parameter corresponding to each partition according to a mapping parameter lookup table and the histogram corresponding to each partition; and adjusting pixel values of the corresponding partition in the to-be-processed image by using the mapping parameter to obtain a processed image. According to the image processing method, accurate image details can be retained, the image processing process can be simplified, the calculation amount can be reduced, and the execution efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method, apparatus, terminal device, storage medium, and program product. Background Technology

[0002] Existing image signal processors (ISPs) in terminal devices are typically designed to output images that are visually more pleasing to the human eye. However, these visually pleasing images may not be suitable for object detection, such as vehicle or pedestrian detection. Therefore, based on existing image signal processors, it is necessary to design an image processing method to obtain images suitable for object detection.

[0003] Existing image processing methods either fail to preserve accurate image details or are complex to implement, computationally intensive, and inefficient. Therefore, how to simplify the image processing flow, reduce computational load, and improve execution efficiency while preserving accurate image details has become a research hotspot in this field. Summary of the Invention

[0004] In view of this, an image processing method, apparatus, terminal device, storage medium and program product are proposed. The image processing method according to the embodiments of this application can simplify the image processing process, reduce the amount of computation and improve the execution efficiency while preserving accurate image details.

[0005] In a first aspect, embodiments of this application provide an image processing method applied to a first image processing model on a terminal device. The method includes: splitting an image to be processed into multiple partitions based on partitioning parameters, wherein the partitioning parameters indicate the size of each partition; performing histogram statistics on each partition to obtain a histogram corresponding to each partition; determining a mapping parameter corresponding to each partition based on a mapping parameter lookup table and the histogram corresponding to each partition; and adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain a processed image.

[0006] According to the image processing method of this application embodiment, the image to be processed is divided into multiple partitions based on partitioning parameters. The partitioning parameters indicate the size of each partition, effectively increasing the overall receptive field of the first image processing model while controlling subsequent computational load and improving image processing efficiency. Histogram statistics are performed on each partition to obtain a histogram corresponding to each partition. Based on the mapping parameter lookup table and the histogram corresponding to each partition, the mapping parameters corresponding to each partition are determined, significantly reducing the time required to determine the mapping parameters and further improving image processing efficiency by reducing latency. Adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters yields the processed image. Since the mapping parameters correspond one-to-one with the partitions, the image processing effect is improved, making the processed image more adaptable to the scene. In this case, the image processing method according to the embodiment of this application simplifies the image processing flow, reduces computational load, and improves execution efficiency while preserving accurate image details.

[0007] According to the first aspect, in a first possible implementation of the image processing method, the horizontal axis of the histogram is a preset range of pixel values, and the vertical axis of the histogram is the number of pixels; determining the mapping parameters corresponding to each partition based on the mapping parameter lookup table and the histogram corresponding to each partition includes: for any partition, obtaining the feature vector corresponding to the partition based on the number of pixels in each pixel value range included in the histogram corresponding to the partition; finding a set of mapping parameters matching the feature vector in the mapping parameter lookup table, and using it as the mapping parameters corresponding to the partition.

[0008] In this way, the mapping parameters for each partition can be determined. With an existing mapping parameter lookup table, the mapping parameters can be determined without model calculation, thus reducing data processing costs.

[0009] According to a first possible implementation of the first aspect, in a second possible implementation of the image processing method, the set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to be processed to the processed image, and a second mapping parameter corresponding to the intercept of the tangent line of the mapping curve.

[0010] By replacing the complete mapping curve with a set of mapping parameters corresponding to the slope and intercept of the tangent, the amount of computation can be greatly reduced and the execution efficiency improved.

[0011] According to the second possible implementation of the first aspect, in the third possible implementation of the image processing method, adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain the processed image includes: for any pixel in any partition, determining the product of the first mapping parameter corresponding to the partition and the pixel value of the pixel; and obtaining the pixel value of the pixel in the processed image based on the sum of the product and the second mapping parameter corresponding to the partition.

[0012] In this way, adjusting the pixel values ​​of an image based on the mapping parameters can actually be a simple linear function mapping, requiring only a minimal amount of computation to achieve fast and effective pixel value adjustment.

[0013] According to the first aspect, or any possible implementation of the first aspect above, in a fourth possible implementation of the image processing method, the method further includes: constructing multiple sample feature vectors based on partitioning parameters, wherein each sample feature vector corresponds to all combinations of the number of pixels in each pixel value interval of the sample histogram under the partitioning parameters; inputting the multiple sample feature vectors into the mapping parameter prediction model respectively, wherein the mapping parameter prediction model outputs the mapping parameters corresponding to each sample feature vector; generating the mapping parameter lookup table according to the multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector, wherein the mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters.

[0014] After obtaining the mapping parameter lookup table in this way, when the terminal device acquires the image to be processed, it is no longer necessary to perform calculations through the mapping parameter prediction model, thereby effectively reducing the amount of computation.

[0015] According to the fourth possible implementation of the first aspect, in the fifth possible implementation of the image processing method, the value of the receptive field of the mapping parameter prediction model is less than the first threshold.

[0016] This method can improve the efficiency of generating the mapping parameter lookup table and ensure its accuracy.

[0017] According to the fifth possible implementation of the first aspect, in the sixth possible implementation of the image processing method, the terminal device further includes a second image processing model, the second image processing model includes at least one layer, each layer includes at least one first image processing model, when the second image processing model includes multiple layers, the layers are connected in series; when each layer includes multiple first image processing models, the first image processing models are connected in parallel, and the partitioning parameters of the first image processing models in the same layer are different.

[0018] By using parallel pathways to increase the receptive field and using serial pathways to enhance the interaction between different regions, image processing performance can be improved.

[0019] According to the sixth possible implementation of the first aspect, in the seventh possible implementation of the image processing method, for any layer of the second image processing model, when the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the image to be processed by the first image processing model of the next layer; when the layer includes multiple first image processing models, the average of the processed images obtained by the multiple first image processing models is used as the image to be processed by the first image processing model of the next layer.

[0020] In this way, the image processing effect of the second image processing model can be improved step by step.

[0021] Secondly, embodiments of this application provide an image processing apparatus applied to a first image processing model on a terminal device. The apparatus includes: a splitting module for splitting an image to be processed into multiple partitions based on partitioning parameters, wherein the partitioning parameters indicate the size of each partition; a statistics module for performing histogram statistics on each partition to obtain a histogram corresponding to each partition; a prediction module for determining a mapping parameter corresponding to each partition based on a mapping parameter lookup table and the histogram corresponding to each partition; and a mapping module for adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain a processed image.

[0022] According to the second aspect, in a first possible implementation of the image processing device, the horizontal axis of the histogram is a preset plurality of pixel value intervals, and the vertical axis of the histogram is the number of pixels; determining the mapping parameters corresponding to each partition according to the mapping parameter lookup table and the histogram corresponding to each partition includes: for any partition, obtaining the feature vector corresponding to the partition according to the number of pixels in each pixel value interval included in the histogram corresponding to the partition; finding a set of mapping parameters that match the feature vector in the mapping parameter lookup table, and using it as the mapping parameters corresponding to the partition.

[0023] According to the first possible implementation of the second aspect, in the second possible implementation of the image processing apparatus, the set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to be processed to the processed image, and a second mapping parameter corresponding to the intercept of the tangent line of the mapping curve.

[0024] According to the second possible implementation of the second aspect, in the third possible implementation of the image processing device, adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain the processed image includes: for any pixel in any partition, determining the product of the first mapping parameter corresponding to the partition and the pixel value of the pixel; and obtaining the pixel value of the pixel in the processed image based on the sum of the product and the second mapping parameter corresponding to the partition.

[0025] According to the second aspect, or any possible implementation of the second aspect above, in a fourth possible implementation of the image processing apparatus, the apparatus further includes: a construction module, configured to construct multiple sample feature vectors based on partitioning parameters, wherein each sample feature vector corresponds to all combinations of the number of pixels in each pixel value interval of the sample histogram under the partitioning parameters; an input module, configured to input the multiple sample feature vectors into the mapping parameter prediction model respectively, wherein the mapping parameter prediction model outputs the mapping parameters corresponding to each sample feature vector; and a generation module, configured to generate the mapping parameter lookup table according to the multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector, wherein the mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters.

[0026] According to the fourth possible implementation of the second aspect, in the fifth possible implementation of the image processing device, the value of the receptive field of the mapping parameter prediction model is less than the first threshold.

[0027] According to the fifth possible implementation of the second aspect, in the sixth possible implementation of the image processing device, the terminal device further includes a second image processing model, the second image processing model includes at least one layer, each layer includes at least one first image processing model, when the second image processing model includes multiple layers, the layers are connected in series; when each layer includes multiple first image processing models, the first image processing models are connected in parallel, and the partitioning parameters of the first image processing models in the same layer are different.

[0028] According to the sixth possible implementation of the second aspect, in the seventh possible implementation of the image processing device, for any layer of the second image processing model, when the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the image to be processed by the first image processing model of the next layer; when the layer includes multiple first image processing models, the average of the processed images obtained by the multiple first image processing models is used as the image to be processed by the first image processing model of the next layer.

[0029] Thirdly, embodiments of this application provide a terminal device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement one or more of the image processing methods described in the first aspect or various possible implementations of the first aspect when executing the instructions.

[0030] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the image processing methods described in the first aspect or various possible implementations of the first aspect.

[0031] Fifthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes one or more of the image processing methods described in the first aspect or various possible implementations of the first aspect.

[0032] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0033] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0034] Figure 1 The image shows the target detection performance of the original image and the enhanced image obtained by processing it with an image signal processor of the prior art when used for target detection.

[0035] Figure 2 Examples are shown of the original image and the image obtained by processing it using an image signal processor with existing technology.

[0036] Figure 3 This illustrates an exemplary application scenario of the image processing method according to embodiments of this application.

[0037] Figure 4 A flowchart illustrating an image processing method according to an embodiment of this application is shown.

[0038] Figure 5 An exemplary structure of a first image processing model according to an embodiment of this application is shown.

[0039] Figure 6 This illustrates an exemplary operation of the mapping parameter prediction module 300 according to an embodiment of this application.

[0040] Figure 7 An exemplary structure of a second image processing model according to an embodiment of this application is shown.

[0041] Figure 8 This paper presents a comparison between the target detection results of the processed image obtained according to the embodiments of this application and the target detection results of the image obtained by the prior art.

[0042] Figure 9 The target detection results of the processed image obtained according to the embodiments of this application are compared with the target detection results of the image obtained by the prior art.

[0043] Figure 10 This paper presents a comparison between the target detection results of the processed image obtained according to the embodiments of this application and the target detection results of the image obtained by the prior art.

[0044] Figure 11 This paper presents a comparison between the target detection results of the processed image obtained according to the embodiments of this application and the target detection results of the image obtained by the prior art.

[0045] Figure 12 This paper presents a comparison between the target detection results of the processed image obtained according to the embodiments of this application and the target detection results of the image obtained by the prior art.

[0046] Figure 13 An exemplary structural diagram of an image processing apparatus according to an embodiment of this application is shown.

[0047] Figure 14 An exemplary structural diagram of a terminal device according to an embodiment of this application is shown. Detailed Implementation

[0048] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0049] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0050] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0051] The following is a description of the terminology that may appear in this article.

[0052] Neural networks (NNs) are computational models that mimic the structure and function of biological neural networks, and are widely used in the field of artificial intelligence. Through different network structures and the use of different training data, neural networks can achieve a variety of functions.

[0053] Image signal processor (ISP): A processing unit installed in a camera or other terminal device equipped with an image sensor, used to convert the photoelectric signals captured by the image sensor into a three-channel RGB image. An image signal processor contains multiple sub-modules, each typically employing traditional algorithms (non-neural network algorithms), a combination of traditional and neural network algorithms, or a pure neural network algorithm.

[0054] Mean average accuracy (mAP): A metric representing the detection accuracy of a model, typically the average of the accuracy rates for each class. A higher mAP indicates higher detection accuracy.

[0055] Peak signal-to-noise ratio (PSNR): In this field, it is one of the metrics used to measure the similarity between a test image and a ground truth image. This metric focuses only on the pixel values ​​of the image, ignoring structural and texture information. Generally speaking, the higher the PSNR, the closer the pixel values ​​of the test image are to the ground truth image.

[0056] Structural similarity (SSIM) is one of the metrics used in this field to measure the similarity between a test image and a ground truth image. This metric considers both the brightness, contrast, and structural information of the images. Generally, the higher the structural similarity, the closer the brightness, contrast, and structural information of the test image are to the ground truth image.

[0057] A histogram is a representation of the pixel value distribution of an image. Given an image with pixel values ​​ranging from 0 to 1, multiple equal-length pixel value intervals are defined based on this range, and the number of pixels in each interval is counted. Figure 1 It is usually presented in the form of a bar chart.

[0058] Receptive field: A term in deep neural networks used to represent the size of the area of ​​the original image that neurons at different locations within the neural network can perceive. A larger receptive field value indicates a wider range of the original image that the neuron can access, meaning that the neuron may contain more global and semantically higher-level features; conversely, a smaller receptive field value indicates that the features it contains are more local and detailed. The value of the receptive field can be used to roughly determine the level of abstraction of each layer in a neural network.

[0059] Image (quality) enhancement refers to techniques that process an image to improve its brightness, color, contrast, saturation, dynamic range, etc., thereby enhancing its visual appeal or other performance metrics. In this article, image enhancement and image quality enhancement refer to the same concept.

[0060] With the iterative updates of neural network technology, image signal processing methods based on deep learning neural networks have made significant progress in the field of image signal processing. Existing terminal devices, such as mobile phones, tablets, and surveillance cameras, mostly employ image signal processing methods based on deep learning neural networks to implement all or part of the image signal processor. After multiple rounds of optimization by developers, these neural network-based image signal processors can output very high-quality enhanced images, with color and contrast enhancements that are very close to the visual effects of the human eye.

[0061] However, on some terminal devices, the enhanced image output by the image signal processor may not be used to display to the user, but rather for object detection. For example, in vehicles with autonomous driving capabilities, the image output by the image signal processor may primarily be used for vehicle or pedestrian detection. An enhanced image that appears better to the human eye does not necessarily mean higher object detection accuracy. Figure 1 The image shows the target detection performance of the original image and the enhanced image obtained by processing it with an image signal processor of the prior art when used for target detection.

[0062] like Figure 1 As shown, the left image is an enhanced image processed by an image signal processor to improve its visual appearance to the human eye, while the right image is the original image. The left image has higher brightness and contrast. When both images are fed into the same object detection model (not shown), it can be seen that the model fails to detect pedestrians in the brighter, more contrasting image. This illustrates that images that appear visually appealing to the human eye may not be suitable for object detection.

[0063] Furthermore, the performance of deep learning neural networks is usually positively correlated with their complexity. That is, when you want to obtain higher quality images, the neural network model usually needs to be designed to be more complex, which also increases the complexity of the image signal processor. Figure 2 Examples are shown, including the original image and the image obtained by processing it using an image signal processor based on existing technology. For example... Figure 2 As shown, the top left corner is the original image, and the bottom right corner is the ground truth image. The top right corner shows enhanced image 1, output by an image signal processor using a simple neural network model (computational complexity = 27.8 * 10^9 floating-point operations per second (GFLOPS)), with a peak signal-to-noise ratio (PSNR) of 26.22 and a structural similarity (SSIM) of 0.9773. The bottom left corner shows enhanced image 2, output by an image signal processor using a complex neural network model (computational complexity = 4956 * 10^9 floating-point operations per second), with a PSNR of 34.95 and a SSIM of 0.9976. It can be seen that the image output by the complex neural network model is closer to the ground truth image because it has a higher PSNR and SSIM, and its brightness and contrast are also closer to the ground truth image.

[0064] However, on some terminal devices, many subtasks may run simultaneously on the chip, and the computing power that a single subtask can be allocated is limited. Therefore, models that can be deployed on chips generally cannot be too complex, and some computationally intensive models are difficult to deploy on chips.

[0065] To reduce computational load and improve execution efficiency, existing technology proposes an image processing scheme. This scheme mainly includes the following steps:

[0066] (1) The input image is downsampled to obtain a low-resolution image, and a set of fusion parameters w1, w2, w3 are predicted based on the downsampled image;

[0067] (2) Based on the fusion parameters w1, w2, and w3, the three basic color lookup tables are fused to obtain the fused color lookup table;

[0068] (3) Based on the fused color lookup table, interpolate the input image to obtain the enhanced image.

[0069] The drawback of this approach is that when the original image size is large, such as a 12-megapixel 4K resolution image with a total pixel count of 3072x4096, the downsampled low-resolution image struggles to capture accurate local detail information in different regions of the original image. Furthermore, the interpolation operation using the fused color lookup table relies on highly customized operators, which cannot be implemented with existing operators, making it very difficult to deploy the model on the chip of a terminal device.

[0070] In summary, existing image processing methods either struggle to preserve accurate image details or are complex to implement, computationally intensive, and inefficient. Therefore, this application proposes an image processing method, apparatus, terminal device, storage medium, and program product. The image processing method according to the embodiments of this application simplifies the image processing flow, reduces computational load, and improves execution efficiency while preserving accurate image details.

[0071] Figure 3 This illustrates an exemplary application scenario of the image processing method according to embodiments of this application.

[0072] like Figure 3 As shown, the application scenario may include a terminal device, which may be equipped with a processor for running the second image processing model. The second image processing model may include at least one layer, each layer including one or more first image processing models for executing the image processing methods of the embodiments of this application. An example of the structure of the second image processing model can be found in [reference needed]. Figure 7 An example of the structure of the first image processing model can be found in [link to example]. Figure 5 .

[0073] The terminal device may further include an image sensor for acquiring image data and an image signal processor for processing the image data. The image output by the image signal processor may be an enhanced image optimized for human visual perception as described above, and may serve as the image to be processed in the first image processing model of the first layer. Alternatively, the terminal device may be connected to other terminal devices equipped with image sensors and image signal processors to obtain enhanced images output by the image signal processors of other terminal devices, which may serve as the image to be processed in the first image processing model of the first layer. Or, the terminal device may be equipped with an image signal processor and connected to other terminal devices equipped with image sensors to obtain image data acquired by the image sensors of other terminal devices, process it through the image signal processor to obtain an enhanced image, and serve as the image to be processed in the first image processing model of the first layer. This application does not limit the specific method of acquiring the image to be processed.

[0074] In the second image processing model, each layer of the first image processing model processes the image to be processed by executing the image processing method of the embodiments of this application to obtain a processed image, making the processed image more suitable for target detection. The processed image obtained by the first image processing model of the same layer can be used to obtain the image to be processed by the first image processing model of the next layer. The processed image obtained by the first image processing model of the last layer can be used to obtain an image for target detection.

[0075] The terminal device in this application can be a smartphone, wearable electronic device (such as a smart bracelet, smartwatch, etc.), virtual reality device, vehicle, aircraft, or other device with target detection requirements, or a computing device connected to the aforementioned device with target detection requirements. Based on this, the image processing method of this application embodiment can be applied to image / video quality enhancement tasks in various application scenarios such as mobile phone photography, autonomous driving, and smart city initiatives.

[0076] Figure 4 A flowchart illustrating an image processing method according to an embodiment of this application is shown.

[0077] like Figure 4 As shown, in one possible implementation, the method is applied to a first image processing model on a terminal device, and the method includes steps S41-S44:

[0078] Step S41: The image to be processed is split into multiple partitions based on partitioning parameters, where the partitioning parameters indicate the size of each partition.

[0079] Figure 5 An exemplary structure of a first image processing model according to an embodiment of this application is shown. For example... Figure 5 As shown, the first image processing model may include a partitioning module 100 for performing step S41. Assuming the image to be processed is 1000x1000 pixels and the partitioning parameter is 4x4, the image to be processed can be divided into 250x250 partitions based on the partitioning parameter. Each partition can be the same size and all are 4x4. In this way, a high-resolution image to be processed can be divided into several smaller blocks with lower resolution.

[0080] Step S42: Perform histogram statistics for each partition to obtain the histogram corresponding to each partition.

[0081] like Figure 5 As shown, the first image processing model may include a partitioned histogram calculation module 200, used to execute step S42. When performing histogram statistics, histogram statistics can be performed on each of the 250x250 partitions obtained in step S41 to obtain the corresponding histogram. The horizontal axis of the generated histogram can be multiple preset pixel value ranges, and the vertical axis can be the number of pixels. Assuming the pixel value range of the image is 0-1, four pixel value ranges can be preset, corresponding to pixel value ranges of 0-0.25, 0.25-0.5, 0.5-0.75, and 0.75-1 respectively. Since the size of each partition is 4x4, that is, each partition includes 16 pixels, the value range of the vertical axis corresponding to any pixel value range of each histogram can be between 0 and 16.

[0082] Step S43: Determine the mapping parameters for each partition based on the mapping parameter lookup table and the histogram corresponding to each partition.

[0083] like Figure 5 As shown, the first image processing model may include a mapping parameter prediction module 300 for performing step S43. The mapping parameter prediction module 300 may include a mapping parameter lookup table 302. The mapping parameter lookup table 302 may be obtained from the mapping parameter prediction model 301. The mapping parameter prediction model 301 may be part of the mapping parameter prediction module 300, or it may not be included in the first image processing model; this application does not impose any limitations on this. Exemplary methods for obtaining the mapping parameter lookup table 302 can be found in the relevant description below.

[0084] The mapping parameter lookup table 302 indicates the correspondence between partitions and mapping parameters. In this embodiment, mapping parameters can be grouped, and a group of mapping parameters can perform the same function as the mapping curve between the original image and the processed image in the prior art. For a certain partition, the histogram obtained in step S42 can be directly used as an index to look up the mapping parameter lookup table 302 to obtain the corresponding mapping parameter. Since there is no need to input the histogram into the mapping parameter prediction model 301 for calculation based on the existing mapping parameter lookup table 302, this greatly reduces the amount of computation and significantly improves efficiency. An exemplary implementation of step S43 can be found in the further description below.

[0085] Step S44: Adjust the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain the processed image.

[0086] like Figure 5 As shown, the first image processing model may include a mapping module 400 for executing step S44. The mapping parameters may be associated with at least one attribute such as partition contrast and brightness, allowing adjustment of the partition's contrast and brightness by adjusting pixel values. Since the mapping parameters are one-to-one with partitions, different partitions can use different mapping parameters. Therefore, adjusting the pixel values ​​of the corresponding partitions in the image to be processed can make the pixel values ​​of the processed image closer to the application scenario requirements, i.e., more suitable for object detection. An exemplary implementation of step S44 can be found in the further description below.

[0087] According to the image processing method of this application embodiment, the image to be processed is divided into multiple partitions based on partitioning parameters. The partitioning parameters indicate the size of each partition, effectively increasing the overall receptive field of the first image processing model while controlling subsequent computational load and improving image processing efficiency. Histogram statistics are performed on each partition to obtain a histogram corresponding to each partition. Based on the mapping parameter lookup table and the histogram corresponding to each partition, the mapping parameters corresponding to each partition are determined, significantly reducing the time required to determine the mapping parameters and further improving image processing efficiency by reducing latency. Adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters yields the processed image. Since the mapping parameters correspond one-to-one with the partitions, the image processing effect is improved, making the processed image more adaptable to the scene. In this case, the image processing method according to the embodiment of this application simplifies the image processing flow, reduces computational load, and improves execution efficiency while preserving accurate image details.

[0088] The following describes an exemplary method for determining the mapping parameters (step S43) corresponding to each partition according to an embodiment of this application.

[0089] In one possible implementation, the horizontal axis of the histogram represents a preset range of pixel values, and the vertical axis represents the number of pixels; step S43 includes:

[0090] For any given partition, the feature vector corresponding to that partition is obtained based on the number of pixels in each pixel value interval included in the histogram corresponding to that partition.

[0091] Find a set of mapping parameters that match the feature vector in the mapping parameter lookup table, and use them as the mapping parameters for that partition.

[0092] As described above, the mapping parameter lookup table 302 can indicate the correspondence between partitions and mapping parameters. For example, it can be used to record the correspondence between the feature vectors corresponding to partitions and the mapping parameters. Therefore, each histogram can be further simplified into a feature vector, and then the matching mapping parameter can be found in the mapping parameter lookup table 302. When there are 4 preset pixel value intervals, the way to simplify the histogram into a feature vector is to represent each histogram as a feature vector of length 4, where each value represents the ordinate corresponding to one pixel value interval in the histogram; when there are 250x250 partitions, a total of 250x250x4 feature vectors can be obtained.

[0093] In this way, the mapping parameters for each partition can be determined. With an existing mapping parameter lookup table, the mapping parameters can be determined without model calculation, thus reducing data processing costs.

[0094] In one possible implementation, a set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to the processed image, and a second mapping parameter corresponding to the intercept of the tangent line of the mapping curve.

[0095] For example, existing solutions for adjusting image pixel values ​​often employ a mapping curve-based adjustment strategy, meaning that the same mapping curve is used to adjust pixel values ​​at any given location. This results in some areas of pixel values ​​being stretched while others are compressed, indicating that the same mapping curve is not suitable for different regions. While predicting a mapping curve for each region could ensure the accuracy of pixel value adjustment, it would lead to significant computational waste.

[0096] To address this, this application proposes using the slope and intercept of the tangent line of the curve instead of the complete mapping curve as parameters for pixel value adjustment. Since the pixel values ​​of pixels within a partition are relatively close, a partition can be approximated as a point on the curve. Therefore, determining the complete mapping curve corresponding to a partition can be simplified to determining the slope and intercept of the tangent line at that point on the complete mapping curve. The slope and intercept can be represented by a first mapping parameter and a second mapping parameter, respectively. The first and second mapping parameters of the tangent line at the same point can be used as a set of mapping parameters. In this case, for any pixel within the same partition, although the pixel values ​​may differ, they can correspond to the same mapping parameters.

[0097] By replacing the complete mapping curve with a set of mapping parameters corresponding to the slope and intercept of the tangent, the amount of computation can be greatly reduced and the execution efficiency improved.

[0098] The following describes an exemplary method of adjusting the pixel values ​​of the corresponding partitions in the image to be processed using mapping parameters to obtain the processed image (step S44) according to an embodiment of this application.

[0099] In one possible implementation, step S44 includes:

[0100] For any pixel in any partition, determine the product of the first mapping parameter corresponding to the partition and the pixel value of the pixel;

[0101] The pixel value of the pixel in the processed image is obtained by summing the product and the second mapping parameter corresponding to the partition.

[0102] For example, assuming the partitioning parameter is 4x4 and there are 250x250 partitions, the pixel values ​​of the pixels in partition i (1≤i≤250x250) can be regarded as a 4x4 pixel value matrix. The first mapping parameter and the second mapping parameter obtained in step S43 are x1 and x2, respectively. Then the product of the first mapping parameter x1 corresponding to partition i and the pixel value matrix img_i of partition i can be x1*img_i; the pixel value matrix out_i of partition i in the processed image can be the sum of x1*img_i and the second mapping parameter x2 corresponding to partition i, that is, out_i=x1*img_i+x2.

[0103] In this way, adjusting the pixel values ​​of an image based on the mapping parameters can actually be a simple linear function mapping, requiring only a minimal amount of computation to achieve fast and effective pixel value adjustment.

[0104] The following example uses the mapping parameter prediction model 301 set in the mapping parameter prediction module 300 to introduce the training method of the mapping parameter prediction model 301 and an exemplary method for generating a mapping parameter lookup table using the trained mapping parameter prediction model 301.

[0105] The mapping parameter prediction model 301 can be a neural network model, which can be trained using existing techniques. Through multiple iterations, the mapping parameter prediction model can "predict the corresponding mapping parameters based on the feature vector corresponding to the histogram of a partition". For example, a sample image can be split into multiple sample partitions based on partition parameters, and histogram statistics can be performed on each sample partition to obtain the sample histogram corresponding to each sample partition. The exemplary implementation method is the same as steps S41 and S42 described above, and can also be completed by the partition module 100 and the partition histogram calculation module 200 described above, respectively, which will not be repeated here. The partition parameters used when splitting the sample image can be the same as or different from the partition parameters used in step S41, and this application does not impose any restrictions on this.

[0106] After obtaining the sample histograms corresponding to the partitions, the sample histograms can be input into the mapping parameter prediction module 300 described above. Figure 6 This illustrates an exemplary operation of the mapping parameter prediction module 300 according to an embodiment of this application. Figure 6As shown, when the terminal device is in the training phase, the mapping parameter prediction module 300 can first obtain the feature vector based on the histogram, which is then used as the input to the mapping parameter prediction model 301. The mapping parameter prediction model 301 is trained iteratively multiple times. This application does not limit the specific training method of the mapping parameter prediction model 301. When the terminal device is not in the training phase, the mapping parameter prediction module 300 can run the mapping parameter lookup table 302 to execute step S43 as described above.

[0107] The following describes an exemplary method for obtaining the mapping parameter lookup table 302.

[0108] In one possible implementation, the method further includes:

[0109] Multiple sample feature vectors are constructed based on partitioning parameters. Each sample feature vector corresponds to all combinations of the number of pixels in each pixel value range of the sample histogram under the partitioning parameters.

[0110] Multiple sample feature vectors are input into the mapping parameter prediction model, and the mapping parameter prediction model outputs the mapping parameters corresponding to each sample feature vector.

[0111] A mapping parameter lookup table is generated based on multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector. The mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters.

[0112] For example, after training the mapping parameter prediction model 301, multiple sample feature vectors can be constructed based on partitioning parameters. Each sample feature vector corresponds to all combinations of pixel counts within each pixel value interval of the sample histogram under that partitioning parameter. That is, by traversing all possible histogram cases under each partitioning parameter, the corresponding sample feature vectors are obtained. These multiple sample feature vectors are then input into the mapping parameter prediction model 301, which outputs the mapping parameters corresponding to each sample feature vector. This process can be completed offline. A mapping parameter lookup table is generated based on the multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector. This lookup table records each sample feature vector and its corresponding set of mapping parameters. For example, all <input feature vectors, output mapping parameters> can be saved to form the mapping parameter lookup table 302.

[0113] After obtaining the mapping parameter lookup table in this way, when the terminal device acquires the image to be processed, it is no longer necessary to perform calculations through the mapping parameter prediction model, thereby effectively reducing the amount of computation.

[0114] In one possible implementation, the receptive field value of the mapping parameter prediction model is less than a first threshold.

[0115] As mentioned above, the receptive field represents the size of the range of the original image perceived by neurons at different locations within a neural network. For the mapping parameter prediction model, the receptive field size can be the kernel size of the convolution operation in that model. It can be seen that a larger receptive field results in a more complex convolution operation, slower generation of the mapping parameter lookup table 302, and a greater impact of complex convolution operations on the accuracy of the mapping parameters.

[0116] To address this, this application proposes setting the receptive field of the mapping parameter prediction model to be less than a first threshold. This ensures the speed of generating the mapping parameter lookup table and reduces the impact of the receptive field size on the accuracy of the mapping parameters. The specific value of the first threshold can be set as needed, and this application does not impose any limitations on it. In one example, the receptive field of the mapping parameter prediction model can be set to 1x1. In this case, the mapping parameters can be related only to the partition histogram and are not affected by other factors. Furthermore, since the receptive field of the first image processing model in this application embodiment can be increased by adjusting the partition parameters, even if the receptive field of the mapping parameter prediction model is reduced, the lost accuracy can be compensated for by increasing the partition parameters in subsequent operations, thus not reducing the overall performance of the network.

[0117] This method can improve the efficiency of generating the mapping parameter lookup table and ensure its accuracy.

[0118] In one possible implementation, the terminal device further includes a second image processing model, which comprises at least one layer, each layer comprising at least one first image processing model.

[0119] When the second image processing model includes multiple layers, the layers are connected in series.

[0120] When each layer includes multiple first image processing models, the first image processing models are connected in parallel, and the partitioning parameters of the first image processing models in the same layer are different.

[0121] For example, the partitioning parameters of the first image processing model are usually unchanged. Therefore, in order to further improve the image processing effect, this application embodiment also proposes to connect multiple first image processing models in series and / or in parallel to obtain a second image processing model, and use the second image processing model to perform image processing. Figure 7 An exemplary structure of a second image processing model according to an embodiment of this application is shown.

[0122] On the one hand, the image processing performance of neural network-based models is strongly correlated with the receptive field of the network. The receptive field of a single first image processing model is relatively limited. To further improve the receptive field, a second image processing model can be designed with one layer comprising multiple first image processing models connected in parallel, and the partitioning parameters of each first image processing model in the same layer can be different. For example, the partitioning parameters of each first image processing model in the same layer can be increased sequentially, thereby increasing the receptive field. Figure 7 As shown, each layer can be configured to include three parallel first image processing models, with partition parameters of 4x4, 8x8, and 16x16 for each first image processing model.

[0123] Those skilled in the art will understand that the receptive field can also be increased using other existing technologies. For example, one existing technology increases the receptive field by reserving "holes." When using this method, the partitioning parameters can remain unchanged, and when the image to be processed is divided into multiple partitions, some pixels are skipped, so that the pixels in the same partition have a larger area in the image to be processed. This application does not limit the method of increasing the receptive field.

[0124] On the other hand, the partitions obtained from a single-layer model are independent of each other. To achieve interaction between different partitions, a second image processing model can be designed, comprising multiple layers connected in series. Through multiple series paths, the final image processing effect is improved progressively layer by layer. In this embodiment, as... Figure 7 As shown, three layers can be set, with the partitioning parameters of the first image processing model in each layer being 4x4, 8x8, and 16x16, respectively. In practical applications, the partitioning parameters of the first image processing model in each layer can also be different, and this application does not impose any restrictions on this.

[0125] By using parallel pathways to increase the receptive field and using serial pathways to enhance the interaction between different regions, image processing performance can be improved.

[0126] In one possible implementation, for any layer of the second image processing model, when the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the image to be processed by the first image processing model of the next layer; when the layer includes multiple first image processing models, the average of the processed images obtained by the multiple first image processing models is used as the image to be processed by the first image processing model of the next layer.

[0127] For example, a layer of the second image processing model may include one first image processing model or multiple first image processing models connected in parallel. The overall output of the layer may be different for these two different cases.

[0128] When a layer of the second image processing model includes a first image processing model, the processed image obtained by the first image processing model can be directly used as the image to be processed by the first image processing model in the next layer. If the layer is already the last layer, the processed image obtained by the first image processing model can be directly used as the final processed image output.

[0129] When a layer of the second image processing model includes multiple first image processing models, the average of the processed images obtained by the multiple first image processing models can be used as the image to be processed by the first image processing model of the next layer. If the layer is already the last layer, the average of the processed images obtained by the multiple first image processing models can be used as the final processed image output.

[0130] In this way, the image processing effect of the second image processing model can be improved step by step.

[0131] Tables 1 and 2 show the processing results of the images obtained using the first image processing model and the second image processing model of the present application, and the processing results of the images obtained using the prior art image processing method, when applied to a target detection scenario.

[0132] Table 1

[0133] AP1 AP2 AP3 AP4 mAP FPS Benchmark 1 68.59 75.47 86.55 70.18 75.20 - Benchmark 2 76.04 79.14 87.94 67.16 77.57 164.32 Ours 11 78.84 78.08 89.94 69.33 79.05 554.71 Ours 33 79.53 78.84 89.87 70.51 79.69 63.24

[0134] Table 2

[0135] AP1 AP2 AP3 mAP FPS Benchmark 1 50.33 40.30 53.99 48.21 - Benchmark 2 48.81 42.99 58.12 49.97 164.32 Ours 11 49.10 44.19 59.75 51.01 554.71 Ours 33 49.58 45.89 60.75 52.07 63.24

[0136] In Tables 1 and 2, reference 1 indicates that the input detector is an image obtained by an image signal processor; reference 2 indicates that the input detector is an image obtained using a prior art solution; Ours 11 indicates that the input detector is an image obtained using a first image processing model of an embodiment of this application; Ours 33 indicates that the input detector is an image obtained using a second image processing model of an embodiment of this application, wherein the second image processing model includes three layers, each layer including three first image processing models.

[0137] Table 1 shows the detection results of the images obtained by each scheme on the two-stage detector. Table 2 shows the detection results of the images obtained by each scheme on the one-stage detector. Both detectors are existing detectors in the prior art, but their detection principles are different.

[0138] Average precision (AP) is a metric that represents the detection accuracy of a detector in a specific class. A higher AP indicates higher detection accuracy in that class. Tables 1 and 2 show the detection accuracy for three classes (AP1, AP2, and AP3).

[0139] Mean average precision (mAP) is the arithmetic mean of the average accuracy rates for each class. Generally, the higher the value, the higher the detection accuracy of the detector. In Tables 1 and 2, the arithmetic mean of AP1, AP2, and AP3 can be used.

[0140] Frames per second (FPS) is a metric representing computational speed, indicating the number of executions per second. Tables 1 and 2 show the computation speed of a first / second image processing model on a given graphics processing platform for a 3072x4096 resolution image. However, reference 1 does not perform post-processing on the image output from the image signal processor, therefore its corresponding computational speed cannot be statistically analyzed.

[0141] As can be seen from the two tables above, the embodiments of this application can achieve good results on detectors with different design principles.

[0142] Figures 8-12 This paper presents a comparison between the target detection results of the processed image obtained according to the embodiments of this application and the target detection results of the image obtained by the prior art.

[0143] The comparison focuses on the target detection results of images obtained by an image signal processor, images obtained by the prior art, and images obtained using the first image processing model of this application. For ease of comparison, in... Figure 8 , Figure 9 , Figure 12 The differences in target detection results among the three methods are marked with arrows. It can be seen that... Figure 8 In the original image, two objects were detected; in the image obtained by the prior art, three objects were detected; and in the image obtained by the first image processing model of this application, four objects were detected.

[0144] exist Figure 9 In the original image, two objects were detected, but the bounding box of one of the objects deviated significantly from the actual object. In the prior art, one object was detected in the image. However, the first image processing model of this application detected two objects, and the bounding box of each object matched the actual object's bounding box to a high degree.

[0145] exist Figure 10In the original image, no object was detected in the target detection result. In the image obtained by the prior art, no object was detected in the target detection result. In the image obtained by the first image processing model of this application embodiment, one object was detected in the target detection result.

[0146] exist Figure 11 In the original image, one object was detected. In the prior art, one object was detected in the target detection result of the image, while in the image obtained using the first image processing model of this application, two objects were detected.

[0147] exist Figure 12 In the original image, four objects were detected, including three pedestrians and one vehicle. In the prior art, five objects were detected, including two pedestrians and three vehicles. The first image processing model of this application detected six objects, including three pedestrians and three vehicles.

[0148] Therefore, the image obtained by the first image processing model in this application embodiment is superior to the comparative scheme in terms of false detection, false negative detection, and the degree of matching between the detection box and the actual object.

[0149] Embodiments of this application provide an image processing apparatus. Figure 13 An exemplary structural diagram of an image processing apparatus according to an embodiment of this application is shown.

[0150] like Figure 13 As shown, the apparatus is applied to a first image processing model on a terminal device, the apparatus comprising:

[0151] The splitting module 141 is used to split the image to be processed into multiple partitions based on partitioning parameters, wherein the partitioning parameters indicate the size of each partition.

[0152] The statistics module 142 is used to perform histogram statistics for each partition and obtain the histogram corresponding to each partition;

[0153] Prediction module 143 is used to determine the mapping parameters corresponding to each partition based on the mapping parameter lookup table and the histogram corresponding to each partition;

[0154] The mapping module 144 is used to adjust the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain the processed image.

[0155] The splitting module 141 can be implemented by the partitioning module 100 mentioned above, the statistics module 142 can be implemented by the partitioning histogram calculation module 200 mentioned above, the prediction module 143 can be implemented by the mapping parameter prediction model 301 mentioned above, and the mapping module 144 can be implemented by the mapping module 400 mentioned above.

[0156] In one possible implementation, the horizontal axis of the histogram represents a preset range of pixel values, and the vertical axis represents the number of pixels. The step of determining the mapping parameters for each partition based on the mapping parameter lookup table and the histogram corresponding to each partition includes: for any given partition, obtaining the feature vector corresponding to that partition based on the number of pixels in each pixel value range included in the histogram corresponding to that partition; and finding a set of mapping parameters in the mapping parameter lookup table that matches the feature vector, using these as the mapping parameters corresponding to that partition.

[0157] In one possible implementation, the set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to be processed to the processed image, and a second mapping parameter corresponding to the intercept of the tangent line of the mapping curve.

[0158] In one possible implementation, adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain the processed image includes: for any pixel in any partition, determining the product of the first mapping parameter corresponding to the partition and the pixel value of the pixel; and obtaining the pixel value of the pixel in the processed image based on the sum of the product and the second mapping parameter corresponding to the partition.

[0159] In one possible implementation, the device further includes:

[0160] The construction module is used to construct multiple sample feature vectors based on partitioning parameters. Each sample feature vector corresponds to all combinations of the number of pixels in each pixel value range of the sample histogram under the partitioning parameters.

[0161] The input module is used to input the multiple sample feature vectors into the mapping parameter prediction model respectively, and the mapping parameter prediction model outputs the mapping parameters corresponding to each sample feature vector;

[0162] The generation module is used to generate the mapping parameter lookup table based on the multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector. The mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters.

[0163] The construction module, input module, and generation module can be implemented by the mapping parameter prediction module 300 mentioned above.

[0164] In one possible implementation, the receptive field value of the mapping parameter prediction model is less than a first threshold.

[0165] In one possible implementation, the terminal device further includes a second image processing model, which includes at least one layer, each layer including at least one first image processing model. When the second image processing model includes multiple layers, the layers are connected in series; when each layer includes multiple first image processing models, the first image processing models are connected in parallel, and the partitioning parameters of the first image processing models in the same layer are different.

[0166] In one possible implementation, for any layer of the second image processing model, when the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the image to be processed by the first image processing model of the next layer; when the layer includes multiple first image processing models, the average of the processed images obtained by the multiple first image processing models is used as the image to be processed by the first image processing model of the next layer.

[0167] An embodiment of this application provides a terminal device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0168] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0169] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0170] Figure 14 An exemplary structural diagram of a terminal device according to an embodiment of this application is shown.

[0171] like Figure 14As shown, the terminal device may include at least one of the following: mobile phone, foldable electronic device, handheld computer, ultra-mobile personal computer (UMPC), personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device. This application embodiment does not impose any special limitation on the specific type of the terminal device.

[0172] The terminal device may include a processor 110, a memory 121, and a communication module 160. It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0173] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0174] The processor can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0175] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently, such as partition parameters in the embodiments of this application. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0176] Memory 121 can be used to store computer executable program code, including instructions. Memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as a mapping parameter lookup table), etc. The data storage area may store data acquired or created during the use of the terminal device (such as histograms, etc.). In addition, memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 implements the various methods performed by the terminal device by executing instructions stored in memory 121 and / or instructions stored in memory disposed in the processor.

[0177] The communication module 160 can be used to receive data (such as image data and enhanced images in the embodiments of this application) from other devices or equipment via wireless / wired communication, and to output data to other devices or equipment. For example, it can provide wireless communication solutions including WLAN (such as Wi-Fi network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR) technology, etc.

[0178] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), compact disk read-only memory (CD-ROM), digital video disk (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0179] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0180] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smallport, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0181] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0182] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0183] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0185] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0186] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0187] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that, The method is applied to a first image processing model on a terminal device, and the method includes: The image to be processed is divided into multiple partitions based on partitioning parameters, wherein the partitioning parameters indicate the size of each partition. Perform histogram statistics for each partition to obtain the histogram corresponding to each partition; The mapping parameters for each partition are determined based on the mapping parameter lookup table and the histogram corresponding to each partition. The pixel values ​​of the corresponding partitions in the image to be processed are adjusted using the mapping parameters to obtain the processed image; The horizontal axis of the histogram represents a preset range of pixel values, and the vertical axis represents the number of pixels. Determining the mapping parameters for each partition based on the mapping parameter lookup table and the histogram corresponding to each partition includes: for any given partition, obtaining the feature vector corresponding to that partition based on the number of pixels in each pixel value range included in the histogram corresponding to that partition; finding a set of mapping parameters matching the feature vector in the mapping parameter lookup table, and using this set as the mapping parameters corresponding to that partition; the set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to the processed image, and a second mapping parameter corresponding to the intercept of the tangent line of the mapping curve.

2. The method according to claim 1, characterized in that, The step of adjusting the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain the processed image includes: For any pixel in any partition, determine the product of the first mapping parameter corresponding to that partition and the pixel value of that pixel; The pixel value of the pixel in the processed image is obtained by summing the product and the second mapping parameter corresponding to the partition.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Multiple sample feature vectors are constructed based on partitioning parameters. Each sample feature vector corresponds to all combinations of the number of pixels in each pixel value range of the sample histogram under the partitioning parameters. The multiple sample feature vectors are respectively input into the mapping parameter prediction model, and the mapping parameter prediction model outputs the mapping parameters corresponding to each sample feature vector. The mapping parameter lookup table is generated based on the multiple sample feature vectors and the mapping parameters corresponding to each sample feature vector. The mapping parameter lookup table records each sample feature vector and its corresponding set of mapping parameters.

4. The method according to claim 3, characterized in that, The receptive field value of the mapping parameter prediction model is less than the first threshold.

5. The method according to claim 4, characterized in that, The terminal device also includes a second image processing model, which comprises at least one layer, and each layer includes at least one first image processing model. When the second image processing model includes multiple layers, the layers are connected in series. When each layer includes multiple first image processing models, the first image processing models are connected in parallel, and the partitioning parameters of the first image processing models in the same layer are different.

6. The method according to claim 5, characterized in that, For any layer of the second image processing model, when the layer includes a first image processing model, the processed image obtained by the first image processing model is used as the image to be processed by the first image processing model of the next layer. When the layer includes multiple first image processing models, the average of the processed images obtained by the multiple first image processing models is used as the image to be processed by the first image processing model of the next layer.

7. An image processing apparatus, characterized in that, The apparatus is applied to a first image processing model on a terminal device, and the apparatus includes: A splitting module is used to split the image to be processed into multiple partitions based on partitioning parameters, wherein the partitioning parameters indicate the size of each partition. The statistics module is used to perform histogram statistics for each partition and obtain the histogram corresponding to each partition; The prediction module is used to determine the mapping parameters corresponding to each partition based on the mapping parameter lookup table and the histogram corresponding to each partition; The mapping module is used to adjust the pixel values ​​of the corresponding partitions in the image to be processed using the mapping parameters to obtain the processed image; The horizontal axis of the histogram represents a preset range of pixel values, and the vertical axis represents the number of pixels. Determining the mapping parameters for each partition based on the mapping parameter lookup table and the histogram corresponding to each partition includes: for any given partition, obtaining the feature vector corresponding to that partition based on the number of pixels in each pixel value range included in the histogram corresponding to that partition; finding a set of mapping parameters matching the feature vector in the mapping parameter lookup table, and using this set as the mapping parameters corresponding to that partition; the set of mapping parameters includes a first mapping parameter corresponding to the slope of the tangent line of the mapping curve from the image to the processed image, and a second mapping parameter corresponding to the intercept of the tangent line of the mapping curve.

8. A terminal device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-6 when executing the instructions.

9. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-6.

10. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that, When the computer-readable code is run in an electronic device, the processor in the electronic device performs the method according to any one of claims 1-6.

Images