Image processing method, apparatus, device and storage medium

Abstract

An image processing method, executed by an electronic device and comprising: performing object detection on an image to be processed, so as to determine a pixel-region of interest in the image to be processed; performing color temperature value estimation on the basis of a non-pixel-region of interest other than the pixel-region of interest in the image to be processed, so as to determine an estimated light source color temperature value of the image to be processed; on the basis of the estimated light source color temperature value of the image to be processed and historical light source color temperature values of reference images, determining a white balance gain for the image to be processed, wherein the reference images and the image to be processed are acquired by a same image acquisition device, the reference images comprising at least one of K image frames acquired by the image acquisition device before the image to be processed, and K being a positive integer; and, on the basis of the white balance gain for the image to be processed, performing color correction at least on the pixel-region of interest in the image to be processed.

Description

Image processing method, device, apparatus and storage medium

[0001] Related Applications

[0002] The present application claims priority to the Chinese patent application No. 202411352197.5, filed on September 26, 2024, and entitled "Image white balance processing method, device, apparatus and storage medium", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of image processing, and more particularly, to an image processing method, device, apparatus and storage medium. BACKGROUND

[0004] With the progress of image processing technology, image shooting devices such as cameras and mobile phones are widely used, and the quality of images will be affected by light sources in the environment, resulting in color cast phenomenon of images.

[0005] In related technologies, the method of white balance is mainly used to solve the problem of image color cast, but the quality of the color corrected image in related technologies is still not good enough. SUMMARY

[0006] Therefore, the embodiments of the present application provide an image processing method, device, apparatus and storage medium, which can make the color of the concerned area of the color corrected image more accurate.

[0007] In a first aspect, the embodiments of the present application provide an image processing method, which is executed by an electronic device, and includes: performing target detection on a to-be-processed image to determine a concerned pixel area in the to-be-processed image; performing color temperature value estimation on a non-concerned pixel area in the to-be-processed image except the concerned pixel area to determine a light source color temperature value estimation value of the to-be-processed image; determining a white balance gain for the to-be-processed image according to the light source color temperature value estimation value of the to-be-processed image and a historical light source color temperature value of a reference image; the reference image and the to-be-processed image are collected by the same image collection device; the reference image includes at least one of K frames of images collected by the image collection device before the to-be-processed image, K being a positive integer; and performing color correction on at least the concerned pixel area in the to-be-processed image according to the white balance gain for the to-be-processed image.

[0008] In a second aspect, an embodiment of the present application provides a color correction device, comprising a region determination module, a color temperature value estimation module, a gain determination module, and a color correction module. The region determination module is configured to perform target detection on a to-be-processed image, and determine a region of interest in the to-be-processed image. The color temperature value estimation module is configured to perform color temperature value estimation on a non-region of interest in the to-be-processed image, and determine a light source color temperature value estimation value of the to-be-processed image. The gain determination module is configured to determine a white balance gain for the to-be-processed image according to the light source color temperature value estimation value of the to-be-processed image and a historical light source color temperature value of a reference image. The reference image and the to-be-processed image are captured by the same image capturing device. The reference image comprises at least one of K frames of images captured by the image capturing device before the to-be-processed image, where K is a positive integer. The color correction module is configured to perform color correction on at least the region of interest in the to-be-processed image according to the white balance gain for the to-be-processed image.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory. The memory stores computer readable instructions. When the computer readable instructions are executed by the processor, the method described above is implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer readable instructions. When the computer readable instructions are executed by a processor, the method described above is implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises computer readable instructions. When the computer readable instructions are executed by a processor, the steps of the method described above are implemented.

[0012] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the disclosed drawings.

[0014] FIG. 1 shows a block diagram of an electronic device according to an embodiment of the present application;

[0015] FIG. 2 shows a flowchart of an image processing method according to an embodiment of the present application;

[0016] FIG. 3 shows a flowchart of step S110 in FIG. 2;

[0017] FIG. 4 shows a flowchart of step S120 in FIG. 2;

[0018] FIG. 5 shows a schematic diagram of a Planck curve in a CIE xy chromaticity coordinate system according to an embodiment of the present application;

[0019] FIG. 6 shows a schematic diagram of a color gamut range of an RGB camera under different color temperatures according to an embodiment of the present application;

[0020] FIG. 7 shows a flowchart of step S130 in FIG. 2;

[0021] FIG. 8 shows a schematic diagram of a structure of an electronic device according to an embodiment of the present application;

[0022] FIG. 9 shows another flowchart of an image processing method according to an embodiment of the present application;

[0023] FIG. 10 shows a flowchart of step S180 in FIG. 9;

[0024] FIG. 11 shows a schematic diagram of a structure of a target detection model according to an embodiment of the present application;

[0025] FIG. 12 shows an application diagram of an electronic device for performing a method according to an embodiment of the present application;

[0026] FIG. 13 shows an image obtained by performing color correction on an image of a palm collected under different color temperatures in the related art;

[0027] FIG. 14 shows a schematic diagram of a palm payment process performed by a processor according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0029] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.

[0030] The block diagrams in the drawings show only the functionality of the features and can not imply that the functionality must be implemented in the particular correlated order indicated. For example, a step or functionality can be implemented prior to, after, or as part of another step or functionality. Further, both the steps and the functionality can be combined or divided into separate steps or functionality.

[0031] The flow diagrams depicted in the figures are merely exemplary and do not necessarily include all of the steps or operations, nor do they necessarily indicate the order in which the steps or operations can be performed. For example, some operations can be performed in parallel, some operations can be omitted, and some operations can be combined or partially combined with other operations. In the following description, references to "some embodiments" or "some implementations" refer to a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or a different subset than "some other embodiments" and can be combined with other embodiments without conflict.

[0032] It should be noted that "a plurality of" means two or more.

[0033] The following describes exemplary applications of the image processing method provided by the application. The image processing method provided by the application can be applied to an electronic device as shown in FIG. 1.

[0034] Referring to FIG. 1, FIG. 1 is a structural schematic diagram of an electronic device 100 provided by an embodiment of the application. The electronic device 100 can be a terminal device or a server. The electronic device can be used to execute the image processing method provided by the application.

[0035] In some embodiments, the server can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0036] The terminal device can be a smart phone, a tablet computer, a notebook computer, a vehicle-mounted terminal, a payment terminal (for example, a palm payment terminal, a face payment terminal), an access control device, etc., but is not limited thereto, and the terminal device is provided with an image acquisition device to acquire an image to obtain a to-be-processed image.

[0037] The electronic device 100 shown in FIG. 1 includes at least one processor 110, a memory 150, and at least one network interface 120. The various components in the electronic device 100 are coupled together by a bus system 140. It can be understood that the bus system 140 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all the various buses are marked as the bus system 140 in FIG. 1.

[0038] The processor 110 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0039] The memory 150 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical drives, etc. The memory 150 can optionally include one or more storage devices that are physically located away from the processor 110.

[0040] The memory 150 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 150 described in the embodiments of the present application is intended to include any suitable type of memory.

[0041] In some embodiments, the memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily described below.

[0042] The operating system 151 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0043] The network communication module 152 is configured to communicate with other computing devices via one or more (wired or wireless) network interfaces 120, such as Bluetooth, WiFi, Universal Serial Bus (USB), and the like.

[0044] In some embodiments, the image processing method provided by the embodiments of the present application can be implemented in a software manner. FIG. 1 shows an image processing apparatus 155 stored in a memory 150, which can be software in the form of programs and plug-ins, and includes the following software modules: an image region determination module 1551, a color temperature value estimation module 1552, a gain determination module 1553, and a color correction module 1554. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0045] In the following, the method provided by the embodiments of the present application will be described in conjunction with exemplary applications and implementations of the electronic device provided by the embodiments of the present application.

[0046] Referring to FIG. 2, FIG. 2 shows an image processing method provided by the embodiments of the present application, which can be applied to an electronic device, which can be the terminal device or the server described above. The method includes the following steps.

[0047] Step S110: performing target detection on the to-be-processed image to determine a pixel region of interest in the to-be-processed image.

[0048] The to-be-processed image refers to an image that needs to be color corrected. It can be an image collected in real time by an image collection apparatus. The to-be-processed image can be an RGB image or an infrared image.

[0049] For example, the to-be-processed image can be a collected face image, a palm image, or an image collected for a body part, a vehicle, an animal, a pedestrian, a city building, or natural scenery.

[0050] Target detection is an important task in computer vision, and its purpose is to identify specific categories of objects and the positions of the objects in an image.

[0051] The pixel region of interest in the to-be-processed image refers to the pixel region in which a semantic object of interest (the semantic object of interest is also referred to as the target of interest) is located in the to-be-processed image. The target detection on the to-be-processed image can identify whether the semantic object of interest exists in the to-be-processed image and the position information of the semantic object of interest in the to-be-processed image. The position information of the semantic object of interest in the to-be-processed image can be indicated by the position of a detection box surrounding the pixel region in which the semantic object is located. The detection box can be a rectangular detection box.

[0052] The attention target can be set as needed, and the attention target can be a palm print, a face, etc. The set attention target can be one or multiple.

[0053] The other regions in the to-be-processed image except the attention pixel region are non-attention pixel regions, wherein the attention pixel region usually contains information that is most concerned by the user or the application program, such as a face, a specific object, or an identifier, etc.; the non-attention region usually contains background information or other secondary elements.

[0054] In the target detection on the to-be-processed image, the determined attention pixel region in the to-be-processed image can be one or multiple, and if it is multiple, the semantic objects in each attention region can be different.

[0055] In the above step S110, the to-be-processed image can be subjected to target detection by a target detection model, and the attention pixel region in the to-be-processed image can be determined according to the detection result output by the target detection model. The target detection model can be a deep learning model, such as Mask R-CNN (Mask Region Convolutional Neural Network), U-Net (U-shaped Convolutional Neural Network), etc. It can be understood that the process of target detection is equivalent to semantic recognition of the image, and it can recognize whether the image includes an attention target and, in the case where the attention target exists in the image, the position of the attention target in the image.

[0056] In an implementation manner, please refer to FIG. 3, the step S110 includes steps S112-S114:

[0057] Step S112: the to-be-processed image is subjected to target detection by using a target detection model to obtain a detection result, and the detection result includes a semantic object recognized from the to-be-processed image and position information of the semantic object in the to-be-processed image.

[0058] The target detection model can include a feature extraction network and a classification layer, and when the to-be-processed image is subjected to target detection by using the target detection model, the specific process is as follows:

[0059] The feature extraction network in the target detection model is used to extract features of the to-be-processed image. When the feature extraction network in the target detection model is a convolutional neural network (CNN), the image block encoding sequence is arranged in a two-dimensional matrix form in sequence and input into the CNN. First, the input image block encoding sequence is convolved by multiple convolutional layers, each convolutional layer including multiple convolutional kernels, and the convolutional kernels slide on the image block encoding sequence to perform convolution calculation and extract features of different scales and directions. Then, the output of the convolutional layer is down-sampled by a pooling layer (such as maximum pooling) to reduce the size of the feature map while retaining important feature information. After multiple layers of convolution and pooling operations, the image features of the to-be-processed image are obtained. If the feature extraction network is a Transformer network, each image block encoding vector in the image block encoding sequence is added with position encoding information to retain the spatial position relationship of the image blocks. Then, these vectors are input into the multi-head self-attention mechanism of the Transformer network to calculate the attention scores between each vector and other vectors, and the context representation of each vector is obtained by weighted summation. Then, the context representation is nonlinearly transformed by a feedforward neural network, and the multi-head self-attention and feedforward neural network operations are repeated multiple times to finally obtain the image features of the to-be-processed image. The multi-head self-attention mechanism is a key component in the Transformer network, and in the feature extraction process of the target detection model, each image block encoding vector in the image block encoding sequence is added with position encoding information and then input into the mechanism to calculate the attention scores between each vector and other vectors, and the context representation of each vector is obtained by weighted summation, which helps the model capture long-distance dependencies and context information in the image.

[0060] Then, the classification layer in the target detection model is used to classify the to-be-processed image based on the image features of the to-be-processed image, and the semantic objects of the to-be-processed image and the position information of the semantic objects in the to-be-processed image are obtained.

[0061] In some embodiments, before inputting the target detection model, the method can further include: dividing the to-be-processed image into blocks to obtain an image block sequence of the to-be-processed image, and then encoding the image block sequence of the to-be-processed image to obtain an image block encoding sequence of the to-be-processed image. The step S112 includes: using the feature extraction network in the target detection model to extract features of the image block encoding sequence of the to-be-processed image to obtain image features of the to-be-processed image. Then, the classification layer in the target detection model is used to detect the target of the to-be-processed image based on the image features of the to-be-processed image, and the semantic objects of the to-be-processed image and the position information of the semantic objects in the to-be-processed image are obtained.

[0062] The image block coding sequence is obtained by coding each image block in the image block sequence after dividing the sample image or the image to be processed into the image block sequence. The coding manner can be linear coding and position coding of the image block to obtain the final image coding sequence, or fusing the linear coding and the position coding of the image block to obtain the final image coding sequence, or calculating the color gradient of each pixel point in the image block, determining the HOG feature of the image block, and splicing to obtain the final image coding sequence. The sequence is used for feature extraction in the feature extraction network of the target detection model, which helps the model to capture local features more carefully, improves the detection accuracy, and at the same time, reduces the calculation burden and speeds up the training speed. The color gradient is calculated according to the color value of each pixel point in the image block, and reflects the direction and amplitude of the color change of the pixel point. By calculating the color gradient, the gradient direction and gradient amplitude of the pixel point can be obtained, and then the HOG feature of the image block can be determined.

[0063] In the process of dividing the image to be processed into the image block sequence, the image to be processed can be divided into image blocks of a fixed size, and arranged in a spatial order of the image blocks to obtain the image block sequence.

[0064] The image to be processed is divided into a plurality of image blocks, and the feature extraction network in the subsequent target detection model can perform feature extraction on the plurality of image blocks in parallel. In this way, the target detection efficiency of the target detection model can be improved. Moreover, the resolution of the image block is smaller than that of the entire image to be processed, and compared with feature extraction in units of the image to be processed, the calculation complexity of feature extraction in units of the image block is lower, while the detail information in the image to be processed can be maintained.

[0065] In an implementation manner, coding each image block in the image block sequence can be linear coding and position coding of the image block to obtain the final image coding sequence, or fusing the linear coding and the position coding of the image block to obtain the final image coding sequence.

[0066] For example, each image block in the image block sequence can be position coded to obtain the position coding of each image block, and the position coding of the image block is used to represent the position of the image block in the image to be processed. Each image block in the image block sequence can be linear coded to obtain the linear coding of each image block. For each image block in the image block sequence, the position coding and the linear coding of the image block are fused to obtain the image coding of the image block. The image codings of the plurality of image blocks are combined according to the positions of the image blocks in the image block sequence to obtain the image coding sequence of the image block sequence.

[0067] In the position encoding of each image block in the image block sequence, for each image block, the position information (e.g., coordinates) of the image block relative to the overall position of the image is calculated, and then the position information of the image block is converted into an encoded form. Then, the encoded form of the position information of each image block can be combined with the pixel values or feature representation of the image block itself to obtain the position encoding of the image block. In the linear encoding of each image block in the image block sequence, the image block can be subjected to linear transformation to obtain the linearly transformed result, which is the linear encoding feature of the image block.

[0068] In the linear encoding of each image block in the image block sequence, principal component analysis (PCA) is used for linear transformation. The specific steps are as follows: first, the image block is unfolded into a one-dimensional vector, then the covariance matrix of all image block vectors is calculated, then the covariance matrix is subjected to eigenvalue decomposition, the eigenvectors corresponding to the first k largest eigenvalues are selected as principal components, and each image block vector is projected onto the k principal components to obtain the projection coefficient vector, which is the linear encoding feature of the image block. The value of k is determined according to experiments to balance the feature dimension and the degree of information retention.

[0069] The above-mentioned fusion of the position encoding and the linear encoding of the image block can be concatenation, multiplication, or addition of the position encoding and the linear encoding of the image block.

[0070] By encoding the position of the image block, the subsequent feature extraction network can better understand and utilize the position information of each image block in the image to be processed, so as to capture the spatial information and structure in the image to be processed, which helps the feature extraction network to better understand and utilize the spatial relationship and context information in the image to be processed. Linear encoding of the image block can enable the feature extraction network to capture continuous features and patterns in the image to be processed, and improve the understanding and classification accuracy of the content in the image to be processed.

[0071] In another implementation, encoding each image block in the image block sequence can further include: performing color gradient calculation on each pixel in the image block according to the color value of each pixel in the image block to obtain the gradient direction and gradient amplitude of the pixel; determining the HOG feature (Histogram of Oriented Gradients) of the image block according to the gradient direction and gradient amplitude of each pixel in the image block; and splicing the HOG features of each image block in the image block sequence according to the order of the image blocks in the image block sequence to obtain the image block encoding sequence. The gradient direction histogram is a feature representation method determined according to the gradient direction and gradient amplitude of each pixel in the image block. Splicing the HOG features of each image block in the image block sequence according to the order of the image blocks in the image block sequence can obtain the image block encoding sequence, which can effectively capture the semantics of the image to be processed and has less calculation, thereby reducing the calculation consumption of the terminal device.

[0072] By dividing the image to be processed into image blocks, extracting the HOG features from each image block, and combining the HOG features of each image block into a large one-dimensional vector (image block encoding sequence), the image block encoding sequence contains the color gradient information in the image to be processed, and the semantics of the image to be processed can be effectively captured during subsequent feature extraction based on the image block encoding sequence. Moreover, the HOG features of the image blocks are extracted with less calculation, which can reduce the calculation consumption of the terminal device when the scheme of the present application is applied to the terminal device.

[0073] Step S114: If it is determined according to the detection result that the image to be processed includes the target semantic object, the pixel region in which the target semantic object in the image to be processed is located is determined as the pixel region of interest according to the position information of the target semantic object in the image to be processed.

[0074] The target semantic object is the semantic object of interest mentioned above. If the position information of the target semantic object in the image to be processed indicates the position information of the detection box of the target semantic object, the pixel region in which the target semantic object in the image to be processed is located can be regarded as the pixel region within the bounding box, i.e., the pixel region of interest.

[0075] The pixel region of interest in which the target semantic object in the image to be processed is located can be a pixel region that needs to be focused on subsequently, for example, for further image processing. If the image to be processed does not include the target semantic object, the image to be processed can not need to be color corrected subsequently, which can avoid unnecessary color correction of the image to be processed. Subsequently, the next image of the image to be processed can be obtained as a new image to be processed, and the above process can be performed.

[0076] Step S120: Color temperature value estimation is performed on the non-focus pixel region in the to-be-processed image except the focus pixel region, and a light source color temperature value estimation value of the to-be-processed image is determined.

[0077] The light source color temperature value estimation value of the to-be-processed image is the color temperature of the light source in the environment where the to-be-processed image comes from, which is determined through color temperature value estimation.

[0078] The color temperature is a unit of measurement indicating the color component contained in light, which is an important parameter for describing the color characteristics of a light source. The colors of light emitted by light sources of different color temperatures are different. The color temperature is defined based on the principle of black body radiation. When the color of light emitted by a light source is the same as the color of a black body heated to a certain temperature, the temperature of the black body is defined as the color temperature of the light source. If the color of the semantic object of interest (i.e., the target semantic object in the foregoing) is similar to the color of the light emitted by the light source, for example, if the target semantic object is a palm, a face, or the like, and its color is yellow or similar to yellow, and the color of the light emitted by the light source is also yellow, the difference between the two colors is small. If the focus pixel region in the to-be-processed image where the target semantic object is located is used for color temperature value estimation, the accuracy of the determined light source color temperature value estimation value will be low. Therefore, in the present application, to avoid the color difference between the target semantic object itself and the light source being small and causing inaccurate color temperature value estimation, the non-focus pixel region in the to-be-processed image is used for color temperature value estimation.

[0079] In some embodiments, before step S120 is performed, the method can further include: performing mask processing on each pixel point in the focus pixel region in the to-be-processed image, to set the color value of each pixel point in the focus pixel region to a specified color value. The specified color value can be a color value whose influence on color temperature value estimation is less than an influence threshold, for example, the specified color value can be black.

[0080] When the focus pixel region is set to a specified color (such as black), although the pixels in the focus pixel region still exist in the to-be-processed image, because their colors have been changed, their color attributes no longer reflect the actual color information of the focus region. The pixel region with the specified color value can not be used for color temperature value estimation subsequently.

[0081] In an implementation manner, step S120 can be: using a pre-trained color temperature recognition model to perform color temperature value estimation on the non-focus pixel region in the to-be-processed image except the focus pixel region, to obtain the light source color temperature value estimation value of the to-be-processed image.

[0082] The color temperature recognition model is trained based on a plurality of color temperature sample images with color temperature labels. A training process of the color temperature recognition model includes: performing feature extraction on the color temperature sample images by using a feature extraction network in the color temperature recognition model to obtain image features; performing classification based on the image features by using a classification network in the color temperature recognition model to obtain color temperature prediction results of the color temperature sample images, obtaining a loss value based on the color temperature prediction results and the color temperature labels, and adjusting model parameters of the color temperature recognition model based on the loss value. The neural network used in the color temperature recognition model can be, but is not limited to, ResNet (Residual Network), VGG (Visual Geometry Group Network), Inception (Inception Convolutional Neural Network), etc.

[0083] In another implementation manner, referring to FIG. 4, the step S120 includes:

[0084] Step S122: performing white point detection processing on the non-focus pixel region to obtain a reference white point in the non-focus pixel region.

[0085] It is worth mentioning that the image subjected to the white point detection processing is an RGB image. If the image to be processed is not an RGB image, the image to be processed can be converted into an RGB image first. The RGB image is an image constructed by using an RGB color mode. The RGB color mode is a color standard in the industry. By changing three color channels of red (R), green (G), and blue (B) and superimposing them on each other, various colors of images can be obtained.

[0086] The method for performing the white point detection processing can include a maximum RGB method.

[0087] In the white point detection processing of the non-focus region of the to-be-processed image by using the maximum RGB method, specifically, for each pixel in the non-focus region of the to-be-processed image, according to the values of the three channels R, G and B of each pixel, in the pixels in the non-focus region of the to-be-processed image whose sum of the values of the three channels is greater than a preset threshold, the first N pixel points in the order of the maximum value of the values of the three channels of the pixel are taken as the reference white point, where N is an integer greater than 1. Specifically, from all the pixels in the non-focus region of the to-be-processed image, the first N pixels in the order of max(R, G, B) are selected when the sum of the values of the three channels R, G and B is greater than the preset threshold, such as (255, 255, 255), (255, 255, 254) or (254, 255, 255), and so on. Here, max(R, G, B) means selecting the maximum value in the three channels of red, green and blue. In an ideal case, a pixel close to white usually has high values in all three channels, and the values are close to each other. Therefore, if the max(R, G, B) value of a pixel is close to 255, and the values of the three channels R, G and B are not much different, then this pixel is likely to be a pixel close to white. The preset threshold can be obtained by statistical analysis of a large amount of experimental data, and the value range can be 50K-100K.

[0088] Step S124: determining the estimated value of the light source color temperature of the to-be-processed image according to the color value of the reference white point.

[0089] In an implementation manner, the above step S124 can be: searching for the estimated value of the light source color temperature corresponding to the color value of the reference white point from a preset corresponding relationship table, where the preset corresponding relationship table stores a plurality of color temperatures and the color value corresponding to each color temperature.

[0090] In an implementation manner, the above step S124 includes:

[0091] Step S124a: converting the color value of the reference white point to a target chromaticity space to obtain the chromaticity coordinates corresponding to the color value of the reference white point.

[0092] The target chromaticity space refers to the CIE xy chromaticity coordinate system, which is derived from the CIE XYZ color space. In the CIE XYZ color space, which is technically in the RGB color space, three ideal primary colors are selected by mathematical methods to replace the actual three primary colors, so that each color can be described by three numerical values.

[0093] The CIE XYZ color space is defined by three color matching functions X, Y, and Z, where X represents the response of a red primary color, Y represents the response of a green primary color, which is also generally considered to be the component closest to the perceived intensity of the human eye, and Z represents the response of a blue primary color. In the CIE xy chromaticity coordinate system, colors are represented as points in a two-dimensional plane (xy coordinate plane), and the points on the xy coordinate plane represent the chromaticity properties of the colors, while ignoring the intensity information. The xy coordinate plane is converted from the CIE XYZ color space by the following conversion formula: x = X / (X+Y+Z); y = Y / (X+Y+Z).

[0094] In the case of a known color value of a reference white point, the color value of the reference white point can be converted to obtain the coordinates of the color value of the reference white point in the CIE XYZ color space, and the coordinates of the color value of the reference white point in the CIE XYZ color space are converted to a plane coordinate value in the CIE xy chromaticity coordinate system by using the conversion formula, and the plane coordinate value is the chromaticity coordinate corresponding to the color value of the reference white point.

[0095] Step S124b: determining the reference point closest to the chromaticity coordinate of the reference white point on the Planck curve.

[0096] The Planck curve is obtained based on the Planck law, which describes the relationship between the energy distribution of black body radiation and temperature, as shown in FIG. 5. When the color of black body radiation is converted to the CIE xy chromaticity coordinate system, the points corresponding to the color of the black body radiation in the CIE xy chromaticity coordinate system form a continuous curve, which is the Planck curve. That is, the points on the Planck curve represent the radiation colors of the black body at different temperatures. When the temperature of the black body is low (for example, about 2000K), the color of the radiation is biased towards red. As the temperature increases, the color of the black body radiation gradually changes to yellow, and then to white (for example, about 5500K). When the temperature of the black body is very high (for example, about 10000K), the color of the radiation is biased towards blue.

[0097] As shown in FIG. 6, the color gamut range of the RGB camera under different color temperature conditions is shown. The multiple curves in FIG. 6 represent the relative intensity changes of the red (R) and blue (B) color channels under different color temperatures. The horizontal axis in FIG. 6 represents the intensity of the red channel, and the vertical axis represents the intensity of the blue channel. As can be seen from FIG. 6, as the color temperature changes, the intensity of the red and blue channels also changes. At a higher color temperature (such as 8500K), the intensity of the blue channel is larger. When the color temperature decreases, the intensity of the red channel gradually becomes larger and the intensity of the blue channel gradually becomes smaller.

[0098] The color temperature characteristics of many light sources can be approximately described by black body radiation under natural light or artificial light source. That is, the color temperature of many common light sources (such as sunlight, incandescent lamp, fluorescent lamp, etc.) can find an approximate point on the Planck curve to represent. Therefore, when there is a color value of a reference white point, it is converted into CIE xy chromaticity coordinates, and the nearest point on the Planck curve to it is found as a reference point, and the color temperature corresponding to the reference point can be used as the color temperature of the reference white point.

[0099] Step S124c: determining the light source color temperature value estimate of the image to be processed according to the color temperature corresponding to the reference point.

[0100] Wherein, the above-mentioned reference white point is one or more, and the corresponding reference point is also one or more.

[0101] If the reference white point is one, the color temperature of the reference point can be determined as the light source color temperature value estimate of the image to be processed.

[0102] If the reference white point is multiple, in one implementation, the color temperature corresponding to each reference white point can be weighted to obtain the light source color temperature value estimate of the image to be processed.

[0103] In another implementation, the median or mean of the color temperature corresponding to each reference white point can be taken as the light source color temperature value estimate of the image to be processed.

[0104] In yet another implementation, the multiple reference white points can also be clustered to obtain a target category with the most reference white points, and the median of the color temperature corresponding to the multiple reference white points belonging to the target category is taken as the light source color temperature value estimate of the image to be processed.

[0105] K-means clustering algorithm can be used to cluster the multiple reference white points. The specific steps are as follows:

[0106] Step 1, initialize the cluster center: randomly select K reference white points as the initial cluster center, and the value of K can be determined according to experience or experiment, for example, K=3.

[0107] Step 2, assign the reference white point to the cluster: calculate the distance (such as Euclidean distance) between each reference white point and the K cluster centers, and assign the reference white point to the category where the nearest cluster center is located.

[0108] Step 3, update the cluster center: for each category, calculate the centroid of all reference white points in the category, and take the centroid as the new cluster center.

[0109] Repeat steps 2 and 3 until the cluster center no longer changes significantly or the preset number of iterations is reached.

[0110] Step 4, selecting a target category: selecting a category with the most reference white points as the target category, and obtaining a median value of color temperatures corresponding to the plurality of reference white points belonging to the target category as an estimated value of the light source color temperature of the to-be-processed image.

[0111] By using the above method, the estimated value of the light source color temperature of the to-be-processed image is accurately obtained based on the principle that the color temperature characteristics of the actual light source are similar to those of a black body.

[0112] Step S130: determining a white balance gain for the to-be-processed image according to the estimated value of the light source color temperature of the to-be-processed image and the historical light source color temperature of the reference image.

[0113] The reference image and the to-be-processed image are captured by the same image capturing device; the reference image includes at least one of K images captured by the image capturing device before the to-be-processed image, and K is a positive integer.

[0114] The reference image can be an image that does not include a target semantic object, and the target semantic object is a semantic object of interest. For example, the image that is captured at the latest time before the to-be-processed image and does not include the target semantic object.

[0115] In a feasible implementation, it is assumed that the estimated value of the light source color temperature of the to-be-processed image is T est , and the historical light source color temperature of the reference image is T hist . First, a weighted average value T avg is calculated, and the formula is T avg = ω1T est + ω2T hist , where ω1 and ω2 are weight coefficients, ω1+ω2=1, ω1 is determined according to the confidence of the estimated value of the light source color temperature of the to-be-processed image, and ω2 is determined according to the correlation between the number of reference images and the capture time. Then, a white balance gain is determined through a preset color temperature-gain mapping table. The mapping table is obtained based on a large amount of experimental data and records gain values of red (G R ), green (G G ), and blue (G B ) channels corresponding to different color temperature values. According to the calculated T avg , the gain value corresponding to the closest color temperature value in the mapping table is searched, and is used as the white balance gain of the to-be-processed image.

[0116] In a feasible implementation, referring to FIG. 7, the above step S130 can include: step S132: determining a reference light source color temperature according to the estimated value of the light source color temperature of the to-be-processed image and the historical light source color temperature of the reference image.

[0117] The step S132 can be specifically: performing median filtering on the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image to obtain a reference light source color temperature value of the to-be-processed image.

[0118] The median filtering is a nonlinear filtering technique, and the median filtering on the current estimated color temperature and the historical color temperature can smooth the color temperature change and avoid the dramatic fluctuation of the subsequent white balance adjustment caused by the abnormality of a single estimation value.

[0119] The step S132 can also be specifically: performing weighted summation on the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image to obtain a reference light source color temperature value.

[0120] When setting the weights of the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image, the weight of the light source color temperature value estimation value of the to-be-processed image can be determined according to the number of reference images, and the historical light source color temperature value of each reference image is set with a weight according to the collection interval length between the reference image and the to-be-processed image, wherein the number of reference images is positively correlated with the weight of the light source color temperature value estimation value of the to-be-processed image, that is, the more the number of reference images, the smaller the weight of the light source color temperature value estimation value of the to-be-processed image, and the collection interval length corresponding to the reference image is negatively correlated with the weight of the historical light source color temperature value of the reference image, that is, the longer the collection interval length corresponding to the reference image, the smaller the weight of the historical light source color temperature value of the reference image, wherein the sum of the weight of the light source color temperature value estimation value of the to-be-processed image and the weight of the historical light source color temperature value of the reference image is 1.

[0121] The reference light source color temperature value of the to-be-processed image is obtained by using the weighted summation, which can make the final color temperature estimation more stable. Even if the current estimation value is affected by noise, the historical data can help to smooth the fluctuations. Thus, the color temperature mutation caused by the environmental light mutation can be avoided.

[0122] The step S132 can also be: determining whether the absolute value of the difference between the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image is less than a preset difference threshold value. If yes, the light source color temperature value estimation value of the to-be-processed image is determined as the reference light source color temperature value; if not, the light source color temperature value estimation value of the to-be-processed image is weighted and summed to obtain a reference light source color temperature value.

[0123] When there are multiple reference images, the way of determining whether the difference between the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image is less than a preset difference threshold value can be: determining whether the average value or the median value of the absolute value of the difference between the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image is less than a preset difference threshold value.

[0124] Step S134: determining the white balance gain of the to-be-processed image according to the reference light source color temperature value of the to-be-processed image.

[0125] The target of white balance is to make the image look natural under different lighting conditions, i.e. an originally white object can be shown as white under any light source.

[0126] To achieve the effect of white balance, it is usually required to obtain the white balance gain of the red (R), green (G) and blue (B) channels.

[0127] In an implementation, the gain ratio of the R, G and B channels relative to a reference channel can be calculated. For example, a channel is selected as the reference channel, and the gain ratio of the other two channels relative to the reference channel is calculated. Specifically, after the white point detection is performed on the non-focus pixel region of the to-be-processed image to obtain a plurality of reference white points, a target reference white point closest to the point on the Planck curve corresponding to the reference light source color temperature value can be selected from the plurality of reference white points. The channel with the largest intensity value among the red (R), green (G) and blue (B) channels can be selected as the reference channel based on the intensity values of the channels of the target reference white point, or the middle channel (green) can be selected as the reference channel. After the reference channel is selected, the ratio between the intensity value of each channel of the target reference white point and the intensity value of the reference channel can be determined as the gain of the channel.

[0128] For example, if the pixel value of a target reference white point is (220, 210, 180), if the selected reference channel is the R channel, the gain of the G channel G_gain=210 / 220, and the gain of the B channel B_gain=180 / 220; if the selected reference channel is the G channel, the gain of the R channel R_gain=220 / 210, and the gain of the B channel B_gain=180 / 210.

[0129] In another implementation, step S134 can also be determining the white balance gain of the to-be-processed image according to the reference light source color temperature value of the to-be-processed image and a reference color temperature. Specifically, the reference color temperature is the color temperature of neutral white light, such as any value in the range of 5400K to 5600K. For example, the reference color temperature can be set to 5500K, and then the gain of each color channel can be calculated according to the difference between the reference light source color temperature value and the reference color temperature.

[0130] For example, the gain of the G channel G_gain can be determined according to the formula R_gain=1+(T-T0) / P×α, and the gain of the B channel B_gain can be determined according to the formula B_gain=1-(6500K-5500K) / P×β.

[0131] Wherein, T is a reference light source color temperature value corresponding to the to-be-processed image; T0 is a reference color temperature; P is a reference color temperature difference value, which is a preset value; and a and β are preset constants.

[0132] Specifically, P can be adjusted according to different application scenarios and experimental data, and the value range can be 500K-2000K; a and β are constants obtained through a large number of experiments, the value range of a can be 0.1-0.3, and the value range of β can be 0.2-0.4.

[0133] It is worth mentioning that when the reference light source color temperature value is greater than 5500K, similar to overcast or shadowed sunlight, the spectrum is biased to blue, and therefore, the red channel needs to increase more gain to compensate for this shift, while the blue channel may need to reduce the gain; and when the reference light source color temperature value is less than 5500K, similar to the morning or evening sunlight, the spectrum is biased to red, and therefore, the blue channel needs to increase more gain to compensate for this shift, while the red channel may need to reduce the gain. Therefore, the gain calculation method of the red channel and the blue channel can be set based on the above principle.

[0134] For example, if the reference light source color temperature value is 6500K, similar to overcast or shadowed sunlight, the spectrum is biased to blue, and at this time, the gain R_gain of the R channel is 1+(6500K-5500K) / 1000K×a; the gain B_gain of the B channel is 1-(6500K-5500K) / 1000K×β, wherein 1000K in the above gain calculation formula is a fixed value, and a and β are constant values.

[0135] It should be noted that the above gain calculation method is only illustrative, and there can be more calculation methods, which will not be described one by one here.

[0136] Step S140: performing color correction on at least the pixel region of interest in the to-be-processed image according to the white balance gain for the to-be-processed image.

[0137] Specifically, for each pixel of at least the pixel region of interest in the to-be-processed image, the gain of each channel included in the white balance gain for the to-be-processed image is multiplied by the color value of the corresponding channel of the pixel point to obtain a color-corrected image.

[0138] For example, if the gain of the R channel is R_gain, the gain of the G channel is G_gain, and the gain of the B channel is B_gain, and the color channel intensity value of a pixel point is (R1, G1, B1), then the color channel intensity value of the pixel point after color correction is (R2, G2, B2), wherein R2=R1×R_gain, G2=G1×G_gain, and B2=B1×B_gain.

[0139] Since the concerned pixel region usually contains information that is most concerned by the user or the application, such as a face, a palm, a specific object or an identification, etc., and the color of the face, the palm, etc. is usually yellow or white, and the color of the light of the light source is usually yellow as well, the difference between the two colors is small. Therefore, if the concerned pixel region in which the target semantic object in the to-be-processed image is located is used to estimate the color temperature value, the accuracy of the estimated light source color temperature value is not high. By using the above method of the present application, the color temperature value is estimated according to the non-concerned pixel region in the to-be-processed image, and the light source color temperature value estimation value of the to-be-processed image is determined, which can avoid the situation that the color temperature value estimation is not accurate due to the small difference between the actual color of the object presented in the concerned pixel region and the color of the light source in the environment from which the to-be-processed image is derived, so that the light source color temperature value estimation value can more truly reflect the color temperature of the light source in the environment from which the to-be-processed image is derived. Then, the white balance gain for the to-be-processed image is determined in combination with the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image collected before the to-be-processed image, which can realize the use of the historical light source color temperature value of the reference image to assist in determining the white balance gain of the to-be-processed image, thereby improving the accuracy of the white balance gain determination. Finally, the color of the concerned pixel region in the processed image is corrected according to the white balance gain for the to-be-processed image, which can make the color of the concerned pixel region in the processed image more accurate.

[0140] In an implementation manner, the concerned pixel region is a pixel region reflecting a biometric feature; and the method further includes: performing identity recognition according to the color-corrected concerned pixel region to obtain an identity recognition result.

[0141] Specifically, in one embodiment, for the color-corrected concerned pixel region, a local binary pattern (LBP) algorithm is used to extract texture features, the concerned pixel region is divided into a plurality of small sub-regions, the LBP feature histogram of each sub-region is calculated, and then the histograms of all the sub-regions are connected into a feature vector F. A plurality of feature templates F i (i = 1, 2,..., n) of a plurality of registered users are stored in a database. The Euclidean distance D i between the feature vector F and each feature template F i is calculated. If there is a D j smaller than a preset distance threshold D th , it is considered that the identity recognition is successful, and an identity recognition result indicating the user information corresponding to the feature template F j is obtained. If all D i are greater than or equal to D th , an identity recognition result indicating that the identity verification fails is obtained.

[0142] In the identity recognition according to the color-corrected focus pixel region, a feature available for the identity recognition can be extracted from the color-corrected focus pixel region, similarity calculation is performed between the extracted feature and a plurality of feature templates stored in a database, if there is a feature template with similarity greater than a preset threshold, an identity recognition result indicating user information corresponding to the feature template is obtained, and if the similarity is not greater than the preset threshold, an identity recognition result indicating that the identity verification fails is obtained.

[0143] It is worth mentioning that the above method can be applied to electronic devices such as access control devices, payment devices, safes, and the like, which need to perform identity verification. After obtaining the identity recognition result, corresponding operations can be performed based on the functions of the electronic device, such as access control management based on the identity recognition result, payment processing based on the identity recognition result, and the like. In this case, referring to FIG. 8, the electronic device can include an image acquisition device, which can include at least one of an RGB camera and an infrared camera.

[0144] To ensure that the user may be blocked from ambient light when performing identity recognition, resulting in the electronic device being unable to accurately identify the image during image acquisition, in an embodiment of the present application, the electronic device is further provided with a light source for light compensation.

[0145] The color temperature of the light source can be set to 5000K-7000K, and further, the color temperature of the light source can be 5500K, 6000K, or 6500K, etc.

[0146] In an embodiment, the method is applied to a payment device, and the focus pixel region is a palm print pixel region or a face region. After step S140, the method further includes performing payment processing based on the identity recognition result.

[0147] By using the above method, the focus pixel region in the image to be processed can be more accurately color-corrected after the payment device acquires the image to be processed, and the color of the focus pixel region is more natural during subsequent payment, which can avoid the failure of identity recognition due to the influence of light in the environment on the image quality, resulting in payment failure. This reduces the possibility of misidentification or identification failure due to lighting conditions.

[0148] Referring to FIG. 9, in an embodiment, the image processing method further includes steps S150-S190. It can be understood that steps S150-S190 can also form a separate protection subject: a target detection model training method, which can also be referred to as an image processing method, for training a target detection model. The target detection model can be used in the image processing method of the above embodiments, and can also be used in other scenarios requiring target detection.

[0149] Step S150: Obtain training samples.

[0150] The training samples include sample images and label information of the sample images, and the label information indicates annotated semantic objects and sample position information of the annotated semantic objects in the sample images.

[0151] The sample position information of the annotated semantic objects in the sample images indicates a detection box of the sample semantic objects.

[0152] The above-mentioned manner of obtaining training samples can be various, for example, a professional can annotate a plurality of images one by one to determine semantic objects in the images and position information of the images.

[0153] The training samples can also be obtained by using a language model and / or an image recognition model to preliminarily annotate a large amount of data, and screening training samples based on the annotation results. Specifically, a pre-trained language model (such as BERT) can be used to process the text description of a large amount of images, extract semantic information in the text, and generate preliminary semantic labels for the images. At the same time, an image recognition model (such as Faster R-CNN) is used to detect objects in the images and identify the object categories and position information in the images. The semantic labels generated by the language model and the detection results of the image recognition model are fused to obtain preliminary annotation results. For each annotation result, a confidence score C is calculated, which comprehensively considers the prediction probability of the language model and the detection confidence of the image recognition model. Images with a confidence score C greater than a preset threshold C th and complete annotation information (including object categories and position information) are selected as training samples.

[0154] The training samples can also be selected from a target open source data set. The target open source data set can be a GVN data set (Grounded Vision Narratives, a data set for studying visual narratives and understanding multi-modal interaction), an MSCOCO (Microsoft Common Objects in Context, a large-scale image recognition and image-text data set).

[0155] The training samples can include positive samples and negative samples. The sample images in the positive samples include target semantic objects, and the annotated semantic objects in the sample labels of the sample images are target semantic objects. The sample images in the negative samples do not include target semantic objects, and the annotated semantic objects in the sample labels of the negative samples are non-target semantic objects.

[0156] For example, if the target semantic object is a palm, the sample images in the positive samples are images including a palm, and the sample images in the negative samples are images not including a palm; if the target semantic object is a face, the sample images in the positive samples are images including a face, and the sample images in the negative samples are images not including a face; if the target semantic object is a vehicle, the positive samples are images including a vehicle, and the negative samples are images not including a vehicle.

[0157] By setting the positive samples and the negative samples, the feature extraction network can better understand the information or semantics in the image, thereby helping to improve the judgment and decision-making ability of the model. Moreover, due to the existence of the negative samples, the feature extraction network can better generalize when facing new, unseen data. For example, when there are some noise or abnormal features similar to the negative samples in the subsequent data, the model can effectively identify and exclude them, thereby more accurately extracting key features.

[0158] For example, if the target semantic object is a palm, the step S150 can be: a plurality of images of different objects collected by the image collection device can be obtained, and the collection objects can include a left palm, a right palm, a vehicle, a face, a human body, furniture, and the like, wherein the ratio of the number of images including a palm to the number of images not including a palm is within a preset ratio range, such as a range of 0.9-1.1, or the number of images including a palm is the same as the number of images not including a palm. After obtaining the plurality of images, the plurality of images can be cleaned, and the remaining images after cleaning are labeled to obtain sample images. The images including a palm are positive samples, and the images not including a palm are negative samples.

[0159] In an implementation manner, the sample images include at least one of an RGB image and an infrared image.

[0160] Step S160: performing image feature extraction on the sample images by using a feature extraction network in the target detection model to obtain image features of the sample images.

[0161] The feature extraction network refers to a neural network used for feature encoding of a two-dimensional image. The image feature encoder can be constructed by a convolutional neural network, a pooling neural network, or a Transformer network.

[0162] In this case, if the feature extraction network is a convolutional neural network, the step S160 can be: performing convolution and pooling operations on each image block in the image block sequence by using a convolutional layer and a pooling layer to extract features of the sample images.

[0163] When the feature extraction network is a Transformer network, the step S160 can be that, for the image block sequence, a convolutional neural network (CNN) can be used to extract features of the image block sequence, and then the extracted feature vectors are taken as inputs of the Transformer network to further extract context features, so as to obtain the features of the sample image. The image block sequence can also be subjected to feature extraction by using a multi-layer self-attention mechanism and an attention mechanism after normalization of pixel values in the image block, so as to obtain the features of the sample image.

[0164] It should be understood that the above process of feature extraction is only illustrative, and the corresponding feature extraction process is different when the network structure of the feature extraction network is different, which is not limited in the embodiments of the present application.

[0165] Step S170: performing target detection on the sample image based on the image features of the sample image by using a classification layer in the target detection model, to obtain a sample semantic object in the sample image and position information of the sample semantic object in the sample image.

[0166] For the process of performing target detection on the sample image, reference can be made to the foregoing description of step S110 in the embodiments, which will not be repeated here.

[0167] Step S180: determining a model loss of the target detection model based on the sample semantic object, the position information of the sample semantic object in the sample image, the labeled semantic object, and the sample position information of the labeled semantic object in the sample image.

[0168] Please refer to FIG. 10, in an implementable manner, the step S180 includes:

[0169] Step S182: determining a position detection loss based on the sample position information of the labeled semantic object in the sample image and the position information of the sample semantic object in the sample image.

[0170] Specifically, the loss calculation can be performed according to the sample position information of the labeled semantic object in the sample image and the position information of the sample semantic object in the sample image by using a cross-entropy loss function, an IoU loss function, a weighted cross-entropy loss function, a mean square error loss function, a Manhattan distance loss function, and a cosine similarity loss function, to obtain the position detection loss.

[0171] In an embodiment, for the position detection loss, a cross-entropy loss L ce , an IoU loss L iou , a weighted cross-entropy loss L wce , a mean square error loss L mse , a Manhattan distance loss L mdCosine similarity loss L cs .

[0172] Cross-entropy loss L ce : used to measure the difference between the semantic object of the sample and the classification probability distribution of the labeled semantic object, the formula is where y i is the real class label (0 or 1) of the label, p i is the class probability predicted by the model.

[0173] IoU loss L iou : calculate the intersection over union (IoU) between the bounding box of the sample semantic object and the bounding box of the labeled semantic object, and then calculate the loss using L iou = 1-IoU.

[0174] Weighted cross-entropy loss L wce : on the basis of cross-entropy loss, different classes of samples are weighted to deal with the problem of sample imbalance, the formula is where w i is the weight coefficient.

[0175] Mean squared error loss L mse : calculate the mean squared error between the position coordinates of the sample semantic object and the position coordinates of the labeled semantic object, the formula is where and are the predicted position coordinates, and are the real position coordinates.

[0176] Manhattan distance loss L md : calculate the Manhattan distance between the position coordinates of the sample semantic object and the position coordinates of the labeled semantic object, the formula is

[0177] Cosine similarity loss L cs : calculate the cosine similarity between the feature vector of the sample semantic object and the feature vector of the labeled semantic object, and then calculate the loss using .

[0178] Final position detection loss L pos = α1L ce + α2L iou + α3L wce + α4L mse + α5L md + α6L cs , where α1, α2, …, α6 are weight coefficients determined by experiments, used to balance the contribution of different loss functions.

[0179] Step S184: determining a semantic detection loss based on the labeled semantic object labeled for the sample image and the sample semantic object.

[0180] Specifically, the loss calculation can be performed according to the labeled semantic object labeled for the sample image and the sample semantic object by using a cross-entropy loss function, an IoU loss function, a weighted cross-entropy loss function, a mean square error loss function, a Manhattan distance loss function, and a cosine similarity loss function, to obtain the semantic detection loss.

[0181] Further, for the semantic detection loss, the cross-entropy loss L ce ′, the IoU loss L iou ′, the weighted cross-entropy loss L wce ′, the mean square error loss L mse ′, the Manhattan distance loss L md ′, and the cosine similarity loss L cs ′ are also calculated respectively, and the calculation method is similar to the corresponding loss function in the position detection loss. The final semantic detection loss L sem = β1L ce ′+ β2L iou ′+ β3L wce ′+ β4L mse ′+ β5L md ′+ β6L cs ′, where β1, β2, …, β6 are weighting coefficients determined by experiments, for balancing the contributions of different loss functions.

[0182] Step S186: performing weighted processing based on the position detection loss and the semantic detection loss to obtain a model loss of the target detection model.

[0183] The weighting coefficients of the position detection loss and the semantic detection loss can be set according to actual needs, and the weighting coefficients of the two can be the same or different.

[0184] For example, if the position detection loss is L1 and the semantic detection loss is L2, the model loss L totle can be defined as: L totle = λ1×L1+ λ2×L2.

[0185] Where λ1 is the weighting coefficient of the position detection loss, and λ2 is the weighting coefficient of the semantic detection loss. λ1 and λ2 can be determined by experiments to obtain the best values, so as to balance the importance of different tasks in the training process.

[0186] Specifically, the grid search method can be used to determine the optimal values of λ1and λ2. The grid search is a method for determining the weighting coefficients of the position detection loss and the semantic detection loss in the target detection model. By setting the value range of the weighting coefficients and dividing them into several discrete points to form a parameter grid, each set of values in the parameter grid is trained and evaluated on the validation set, and the set of values with the highest average precision mean (mAP) on the validation set is selected as the optimal weighting coefficients to balance the importance of different tasks. First, set the value range of λ1and λ2, for example, λ1∈[0, 1], λ2∈[0, 1], and λ1+ λ2= 1. Then divide the value range into several discrete points to form a parameter grid. For each set of (λ1, λ2) values in the parameter grid, train the target detection model using the same training samples and training settings, and evaluate the performance of the model on the validation set. The evaluation indicator can use the average precision mean (mAP). Select the set of (λ1, λ2) values with the highest mAP on the validation set as the optimal weighting coefficients. The average precision mean is an indicator for evaluating the performance of the target detection model. When training the target detection model, the average precision mean of the model is evaluated on the validation set, and the set of weighting coefficients with the highest mAP on the validation set is selected as the optimal weighting coefficients to balance the contributions of the position detection loss and the semantic detection loss.

[0187] Step S190: updating the model parameters of the target detection model based on the model loss.

[0188] After obtaining the model loss, the model parameters can be adjusted according to the model loss to minimize the model loss. The training end condition can be that the number of iterations of the target detection model reaches a preset number, or the model loss is less than a preset loss threshold.

[0189] By using the above steps S150-S190, the target detection model is trained using training samples with annotation information, so that the target detection model can learn the correct classification labels and target box positions, ensuring that the subsequent target detection model can accurately identify the target objects in the image and accurately locate the positions of these objects.

[0190] In an implementation manner, before performing the above step S160, the above method further comprises: performing block processing on the sample image to obtain an image block sequence; performing image encoding on each image block in the image block sequence to obtain an image block encoding sequence; and the above step S160 can further comprise: using the feature extraction network in the target detection model to perform feature extraction on the image block encoding sequence to obtain the image features of the sample image.

[0191] In the target detection task, since the object can appear at any position in the image, by dividing the sample image into blocks, encoding each image block, and extracting features from the encoded sequence of image blocks by the feature extraction network in the target detection model, the target detection model can capture more detailed local features. For example, in the pedestrian detection task or the palm detection task, even if the pedestrian or palm only occupies a small part of the image, by dividing the image into blocks, the target detection model can focus more on these key blocks, thereby improving the detection accuracy.

[0192] Further, since each image block requires less computational resources relative to the entire image, it can be more easily placed in a graphics processor for efficient parallel computing. At the same time, since each image block can be processed independently, parallel processing can be implemented in a graphics processor or a distributed computing environment, each focusing on the local feature information of the sample image while reducing the computational burden and speeding up the training.

[0193] The process of dividing the sample image into blocks and the subsequent feature extraction process can be referred to the specific description of step S110 above, which will not be repeated here.

[0194] As shown in FIG. 11, a process of training using sample images collected by an infrared camera or an RGB camera is shown. Each sample image has label information. For each sample image, the feature extraction network in the target detection model is used to extract features from the sample image, and the classification layer of the target detection model is used to detect the target based on the image features of the sample image, to obtain the sample semantic object in the sample image and the position information of the sample semantic object in the sample image. Then, based on the sample position information of the labeled semantic object in the sample image and the position information of the sample semantic object in the sample image, the position detection loss L1 is determined. Based on the labeled semantic object and the sample semantic object of the sample image, the semantic detection loss L2 is determined, and finally the position detection loss and the semantic detection loss are weighted to obtain the model loss L of the target detection model. totle Then, the model parameters of the target detection model can be adjusted based on the model loss.

[0195] With the image processing method of the present application applied to a palm payment device as an example for illustration, please refer to (A) shown in FIG. 12, the palm payment device includes a processor (not shown in the figure), an image acquisition device (such as an RGB camera and an infrared camera), and a light source, wherein the image acquisition device is connected with the processor, and the image acquisition device performs image acquisition to obtain a to-be-processed image, and the light source can be used to emit light of a specified color temperature (such as 6500K), as shown in (B) of FIG. 12, the light source can be a light guide ring, used to instruct the user to cover his palm on the position where the light guide ring is located when performing palm payment, the optical axis of the image acquisition device is located in the light guide ring, and the number of the image acquisition device can be one or more, when being multiple, the multiple image acquisition devices can include an RGB image acquisition device and an infrared image acquisition device.

[0196] The image acquisition device in the palm payment device can acquire images to obtain a to-be-processed image, and send the obtained to-be-processed image to the processor. In the multiple palm images acquired by the image acquisition device in the palm payment device, due to the very complex light conditions in the actual environment, including natural light, indoor light, and light of the light guide ring of the palm payment device, etc. The color temperature, intensity and direction of these light sources can be different, resulting in different degrees of color cast in the acquired palm images.

[0197] If the image processing method of the present application is not performed before the payment processing, it can cause the image after color correction according to the determined white balance gain to still have a serious color cast problem, as shown in FIG. 13, the palm image after color correction still has a color cast. Obviously, if the palm image shown in FIG. 13 is directly subjected to identity verification, it can cause inaccurate image feature extraction, and finally cause the palm to be incorrectly identified, and thus the payment processing cannot be completed. In FIG. 13(A), the palm is roughly dark green, and a sample point is expressed as 3d6656 in hexadecimal color code; the background is roughly light gray, and a sample point is expressed as 474948 in hexadecimal color code. In FIG. 13(B), the palm is roughly dark blue-purple, and a sample point in the middle is expressed as 2c1f4d in hexadecimal color code; the background is roughly purple-red, and a sample point is expressed as A62b85 in hexadecimal color code. In FIG. 13(C), the palm is roughly dark indigo, and a sample point is expressed as 32316a in hexadecimal color code; the background is roughly purple-red, and a sample point is expressed as b12182 in hexadecimal color code. In FIG. 13(D), the palm is roughly dark green, and a sample point is expressed as 457f67 in hexadecimal color code; the background is roughly dark gray, and a sample point is expressed as 4f5354 in hexadecimal color code.

[0198] Based on this, please refer to FIG. 14, the processor of the application can specifically include a palm detection module, a color correction module, and a payment processing module (not shown in the figure), each module can perform the following processing process:

[0199] When the processor receives the image to be processed, it can call the palm detection module to perform the following steps:

[0200] Palm detection module: The palm detection module performs real-time palm detection on the images collected by the palm payment device. If there is no pixel region of interest (i.e., palm pixel region) in the collected image, the currently collected image is taken as the reference image, and the color temperature value of the reference image is estimated to determine the light source color temperature of the reference image as the historical light source color temperature value of the reference image.

[0201] The palm detection module continues to perform palm detection on the subsequently collected images. If it is determined that there is a palm pixel region in the image, the color temperature value of the non-pixel region of interest in the image is estimated to determine the light source color temperature value estimate corresponding to the image.

[0202] Wherein, in the process of palm detection by the palm detection module, the following process is performed:

[0203] The real-time collected image is processed in blocks to obtain an image block sequence of the real-time collected image. Then, the image block sequence of the real-time collected image is encoded to obtain an image block encoding sequence of the real-time collected image. Thereafter, the feature extraction network in the target detection model (e.g., palm detection model) is used to perform palm detection on the image block encoding sequence of the real-time collected image. If it is determined that there is a palm pixel region in the real-time collected image, the image can be taken as the image to be processed, and the color correction module can be called to perform the following steps:

[0204] White point detection step: The palm pixel region in the image to be processed is subjected to mask processing to obtain a mask-processed image to be processed. The mask-processed image to be processed is subjected to white point detection processing to obtain a reference white point of the non-palm pixel region.

[0205] The light source color temperature value estimation step: the color value of the reference white point is converted from the RGB space to the CIE XYZ color space, and then to the CIE xy color space to obtain the chromaticity coordinates corresponding to the color value of the reference white point. The chromaticity coordinates corresponding to the color value of the reference white point indicate the position of the reference white point in the CIE xy chromaticity diagram. On the Planck curve, the reference point closest to the chromaticity coordinates of the reference white point is determined; according to the color temperature corresponding to the reference point, the light source color temperature value estimation value of the to-be-processed image is determined. The light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image are median filtered to obtain the reference light source color temperature value of the to-be-processed image. The reference image and the to-be-processed image are collected by the image collection device in the brush palm payment device; the reference image includes at least one of the K frames of images collected by the image collection device before the to-be-processed image, and K is a positive integer.

[0206] The gain calculation step: according to the reference light source color temperature value of the to-be-processed image, the white balance gain of the to-be-processed image is determined.

[0207] The color correction step: according to the white balance gain of the to-be-processed image, at least the palm pixel region in the to-be-processed image is color corrected to obtain the processed palm pixel region.

[0208] After color correcting the palm region, the payment processing module can also be called to perform identity recognition based on the processed palm pixel region to obtain an identity recognition result including user information, and perform payment processing based on the identity recognition result, such as finding a corresponding payment account according to the user information and performing payment processing based on the payment account.

[0209] It is worth mentioning that after the above color correction step is executed, the color correction module can also execute the following steps: determining whether the algorithm involved in the color correction module converges according to the processed palm pixel region. If it does not converge, at least one of the algorithm and the parameter involved in the above steps can be adjusted, and the above steps are repeated again until the algorithm involved in the color correction module converges to obtain the final processed palm pixel region. The method of determining whether the algorithm involved in the color correction module converges can be: calculating the average color temperature of the processed palm pixel region and comparing it with a specified color temperature (such as 6500K). If the difference between the average color temperature and the specified color temperature is less than a preset threshold, it indicates that the algorithm involved in the color correction module has converged.

[0210] By using the above process, the historical light source color temperature value corresponding to the image (i.e., the reference image) collected without the palm as a reference, the historical light source color temperature value of the reference image can accurately reflect the color temperature of the light source in the environment, and the image collected with the palm as the to-be-processed image, and the color temperature value estimation is performed according to the non-palm pixel area in the to-be-processed image, and the light source color temperature estimation value of the to-be-processed image is determined, which can avoid the inaccurate color temperature estimation caused by the small difference between the actual color of the palm in the palm pixel area and the color of the light source in the environment where the to-be-processed image comes from. Therefore, the light source color temperature estimation value of the to-be-processed image can also truly reflect the color temperature of the light source in the environment where the to-be-processed image comes from.

[0211] The historical light source color temperature value of the reference image and the color temperature estimated from the image collected with the palm are comprehensively referenced to determine the white balance gain for the palm in the to-be-processed image, which can assist in determining the white balance gain for the palm in the to-be-processed image using the historical light source color temperature value of the reference image, thereby improving the accuracy of the white balance gain determination. Finally, according to the white balance gain for the to-be-processed image, the color of the palm pixel area is corrected, which can make the color of the processed palm pixel area close to its true color, and eliminate the color cast problem caused by the environment color temperature.

[0212] It should be understood that although each step in the flowchart involved in the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0213] Please refer to Fig. 1 again, another embodiment of the present application provides a color correction device 155, which comprises a region determination module 1551, a color temperature value estimation module 1552, a gain determination module 1553, and a color correction module 1554. The region determination module 1551 is configured to perform target detection on a to-be-processed image, and determine a region of interest in the to-be-processed image. The color temperature value estimation module 1552 is configured to perform color temperature value estimation on a non-region of interest in the to-be-processed image, and determine an estimated value of a light source color temperature of the to-be-processed image. The gain determination module 1553 is configured to determine a white balance gain for the to-be-processed image according to the estimated value of the light source color temperature of the to-be-processed image and a historical light source color temperature of a reference image. The reference image and the to-be-processed image are captured by the same image capturing device. The reference image comprises at least one of K frames of images captured by the image capturing device before the to-be-processed image, where K is a positive integer. The color correction module 1554 is configured to perform color correction on at least the region of interest in the to-be-processed image according to the white balance gain for the to-be-processed image.

[0214] In an implementation manner, the gain determination module 1553 comprises a color temperature obtaining sub-module and a gain determination sub-module. The color temperature obtaining sub-module is configured to perform median filtering on the estimated value of the light source color temperature of the to-be-processed image and the historical light source color temperature of the reference image, to obtain a reference light source color temperature of the to-be-processed image. The gain determination sub-module is configured to determine the white balance gain for the to-be-processed image according to the reference light source color temperature of the to-be-processed image.

[0215] In an implementation manner, the region of interest is a pixel region reflecting a biological feature. The color correction device 155 further comprises an identity recognition module. The color correction module is further configured to perform color correction on at least the region of interest in the to-be-processed image according to the white balance gain for the to-be-processed image, to obtain a color-corrected region of interest. The identity recognition module is configured to perform identity recognition according to the color-corrected region of interest, to obtain an identity recognition result.

[0216] In an implementation manner, the image processing method is applied to a payment device. The region of interest is a palm print pixel region or a face region. The color correction device 155 further comprises a payment processing module configured to perform payment processing based on the identity recognition result.

[0217] In an implementation manner, the color temperature value estimation module 1552 comprises a white point detection sub-module and a color temperature value estimation sub-module. The white point detection sub-module is configured to perform white point detection processing on the non-region of interest, to obtain a reference white point in the non-region of interest. The color temperature value estimation sub-module is configured to determine the estimated value of the light source color temperature of the to-be-processed image according to a color value of the reference white point.

[0218] In an implementation, the color temperature value estimation submodule is further configured to convert the color value of the reference white point to a target chroma space to obtain a chroma coordinate corresponding to the color value of the reference white point; determine a reference point closest to the chroma coordinate of the reference white point on a Planck curve; and determine the estimated value of the light source color temperature of the image to be processed according to the color temperature corresponding to the reference point.

[0219] In an implementation, the region determination module 1551 includes a detection submodule and a region determination submodule. The detection submodule is configured to perform target detection on the image to be processed by using a target detection model to obtain a detection result, the detection result including a semantic object recognized from the image to be processed and position information of the semantic object in the image to be processed. The region determination submodule is configured to, when it is determined that the image to be processed includes a target semantic object according to the detection result, determine a pixel region in which the target semantic object in the image to be processed is located as the pixel region of interest according to the position information of the target semantic object in the image to be processed.

[0220] In an implementation, the color correction apparatus 155 further includes a sample acquisition module, a feature extraction module, a detection module, a loss determination module, and a model update module. The sample acquisition module is configured to acquire a training sample, the training sample including a sample image and label information of the sample image, the label information indicating a labeled semantic object labeled for the sample image and sample position information of the labeled semantic object in the sample image. The feature extraction module is configured to perform image feature extraction on the sample image by using a feature extraction network in the target detection model to obtain image features of the sample image. The detection module is configured to perform target detection on the sample image by using a classification layer in the target detection model to obtain a sample semantic object in the sample image and position information of the sample semantic object in the sample image. The loss determination module is configured to determine a model loss of the target detection model based on the sample semantic object and the position information of the sample semantic object in the sample image, and the labeled semantic object labeled for the sample image and the sample position information of the labeled semantic object in the sample image. The model update module is configured to update model parameters of the target detection model based on the model loss.

[0221] In an implementation, the loss determination module includes a position detection loss determination submodule, a semantic loss determination submodule, and a model loss determination submodule. The position detection loss determination submodule is configured to determine a position detection loss based on the sample position information of the labeled semantic object in the sample image and the position information of the sample semantic object in the sample image. The semantic loss determination submodule is configured to determine a semantic detection loss based on the labeled semantic object labeled for the sample image and the sample semantic object. The model loss determination submodule is configured to perform weighted processing based on the position detection loss and the semantic detection loss to obtain the model loss of the target detection model.

[0222] In an implementation, the color correction device 155 further comprises an image blocking module and an image coding module. The image blocking module is configured to block the sample image to obtain a sequence of image blocks. The image coding module is configured to code each image block in the sequence of image blocks to obtain a sequence of image block codes. The feature extraction module is further configured to extract features of the sample image from the sequence of image block codes by using a feature extraction network in the target detection model.

[0223] In an implementation, the sample image comprises at least one of an RGB image and an infrared image.

[0224] The sample acquisition module, the feature extraction module, the detection module, the loss determination module, and the model updating module, and the optional image blocking module and the image coding module can form a separate protection subject: a target detection model training device, which can also be referred to as an image processing device, for training a target detection model. The target detection model can be used in the image processing method or the image processing device of each of the above embodiments, and can also be used in other scenarios requiring target detection.

[0225] Each of the modules in the image processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to each of the modules. It should be noted that the device embodiments in the present application correspond to the foregoing method embodiments, and the specific principles in the device embodiments can be referred to the content in the foregoing method embodiments, which will not be described herein.

[0226] The computer program product or the computer program provided in the embodiments of the present application includes executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the executable instructions from the computer readable storage medium, and the processor executes the executable instructions to enable the electronic device to perform the method provided in the embodiments of the present application.

[0227] The computer readable storage medium provided in the embodiments of the present application stores computer readable instructions, and the computer readable instructions include executable instructions. When the computer readable instructions are executed by the processor, the processor will execute the foregoing method steps provided in the embodiments of the present application.

[0228] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices including one or any combination of the above memories.

[0229] In some embodiments, the computer readable instructions can be in the form of programs, procedures, modules, scripts, or code, and can be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages; and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0230] By way of example, computer readable instructions can be, but are not limited to, those made of files in a file system, those made of part of files that store other programs or data, e.g., one or more scripts in a Hyper Text Markup Language (HTML) document, those made of a single file dedicated to the program in question, or those made of multiple coordinated files, e.g., files that store one or more modules, sub programs, or code portions.

[0231] By way of example, computer readable instructions can be deployed to be executed on one computer, or on multiple computers of a system, or on multiple computers distributed among multiple locations and interconnected by a communication network.

[0232] To sum up, the present application provides an image processing method, device, equipment, computer readable storage medium and computer program product. The electronic device detects a target in a to-be-processed image and determines a pixel region of interest. Since the object of interest of a user is different in different image scenes, accurately determining the pixel region of interest is the basis for subsequent targeted processing. Then, a light source color temperature estimation value is obtained by performing color temperature estimation according to a non-pixel region of interest. Because the color of the pixel region of interest is usually similar to the color of the light source, if the color temperature is estimated by using the pixel region of interest, the estimation result will be disturbed by the color of the object of interest itself, resulting in inaccuracy. However, by using the non-pixel region of interest, the disturbance can be avoided, and the light source color temperature estimation can more truly reflect the color temperature of the actual light source. Then, a white balance gain is determined in combination with a historical light source color temperature value of a reference image. The historical light source color temperature value of the reference image can provide additional reference information for color temperature estimation of the current to-be-processed image, help correct possible estimation errors, and improve the accuracy of white balance gain determination. Finally, color correction is performed on at least the pixel region of interest according to the gain, so that the color of the pixel region of interest in the processed image is more accurate, the demand of the user for the color accuracy of the object of interest is met, and the visual quality and usability of the image are improved.

[0233] Further, the electronic device performs median filtering on the light source color temperature value estimation of the to-be-processed image and the historical light source color temperature value of the reference image to obtain a reference light source color temperature value, and determines the white balance gain according to the reference light source color temperature value. Median filtering is a nonlinear filtering technique that can effectively suppress the influence of outliers. In the actual image acquisition process, due to the complexity of the environment and the instability of the sensor, there may be an abnormal situation of a single color temperature estimation value. If such an abnormal value is directly used to determine the white balance gain, it will lead to a sharp fluctuation in subsequent white balance adjustment, resulting in poor image color correction effect. The median filtering smooths the color temperature change by processing multiple color temperature values, so that the final reference light source color temperature value is more stable and reliable, thereby further improving the stability and accuracy of the white balance gain determination, and making the color correction effect more natural and uniform.

[0234] When the pixel region of interest is a pixel region reflecting a biometric feature, the electronic device performs identity recognition according to the color-corrected pixel region of interest to obtain an identity recognition result. The color-accurate pixel region of interest is crucial for biometric feature extraction. In the identity recognition process, the accuracy of the biometric feature directly affects the recognition result. If the image is color-biased, the extracted biometric feature will be biased, thereby reducing the accuracy and reliability of identity recognition. The present method restores the color of the pixel region of interest to a state close to the true state through accurate color correction, which enables more accurate extraction of the biometric feature, thereby improving the accuracy and reliability of identity recognition, providing stronger support for scenarios requiring identity verification, such as access control systems and payment systems, and ensuring the safety and normal operation of the system.

[0235] When the electronic device is a payment device and the pixel region of interest is a palmprint pixel region or a face region, after obtaining the identity recognition result, the electronic device performs payment processing based on the result. In the payment scenario, accurate identity recognition is the key to ensuring payment safety and smooth progress. If the image quality is affected by light, leading to identity recognition errors, the payment process may fail, causing inconvenience and economic loss to the user. The present method reduces the possibility of misrecognition or recognition failure caused by light conditions through accurate color correction and identity recognition, improves the success rate and safety of payment processing, and ensures the safety of the user's funds and payment experience.

[0236] Further, the electronic device performs white point detection processing on the non-focus pixel region to obtain a reference white point, and determines an estimated value of the light source color temperature of the to-be-processed image according to a color value of the reference white point. In an image, a white point usually has high brightness and relatively uniform color distribution, and its color value can better reflect the characteristics of the light source. Through white point detection processing, suitable reference white points can be screened out from the non-focus pixel region, and the color values of these reference white points can better represent the color of the actual light source. Estimating the color temperature based on the color values of these reference white points can more accurately reflect the color temperature characteristics of the light source, and provide a more reliable basis for subsequent determination of the white balance gain, thereby improving the accuracy of color correction.

[0237] Further, the electronic device converts the color value of the reference white point to a target chrominance space to obtain a chrominance coordinate, determines a reference point on a Planck curve that is closest to the chrominance coordinate, and determines the estimated value of the light source color temperature of the to-be-processed image according to a color temperature corresponding to the reference point. The Planck curve describes the relationship between the color and temperature of black body radiation, and the color temperature characteristics of many common light sources can be approximately described by black body radiation. After converting the color value of the reference white point to the target chrominance space, the corresponding reference point on the Planck curve can be more accurately found, and the color temperature corresponding to the reference point can more accurately represent the color temperature of the actual light source. In this way, the light source color temperature can be more accurately estimated, thereby improving the accuracy of white balance gain determination, making the color correction effect better, and making the color performance of the image under different light sources more natural and accurate.

[0238] Further, the electronic device performs target detection on the to-be-processed image using a target detection model to obtain a detection result, and determines the focus pixel region according to position information of a target semantic object included in the to-be-processed image if the detection result indicates that the to-be-processed image includes the target semantic object. After being trained, the target detection model can learn the features and patterns of the target semantic object, thereby accurately identifying the target semantic object and its position in the image. Accurately determining the focus pixel region is the basis for subsequent targeted processing, provides an accurate area range for subsequent color temperature estimation and color correction, avoids processing unnecessary areas, improves the accuracy and efficiency of the entire image processing process, and reduces the waste of computing resources.

[0239] Further, the electronic device obtains training samples including sample images and label information, extracts features of the sample images using a feature extraction network in the target detection model, and then uses a classification layer to perform target detection to obtain sample semantic objects and position information. Based on these information and the labeled information, the model loss is determined, and finally the parameters of the target detection model are updated according to the model loss. By using training samples with labeled information, the target detection model can learn the correct classification label and target box position. During the training process, the model continuously adjusts its parameters to minimize the model loss, so that the model can better adapt to different image scenes and target objects. The target detection model trained in this way can accurately identify target objects in images and accurately locate the positions of these objects in practical applications, improving the accuracy and reliability of target detection and providing a more accurate basis for subsequent image processing.

[0240] Further, the electronic device determines a position detection loss based on the position information of the labeled semantic objects and the sample semantic objects, determines a semantic detection loss based on the labeled semantic objects and the sample semantic objects, and then performs weighted processing on the two losses to obtain the model loss of the target detection model. The position detection loss reflects the accuracy of the model in positioning the position of the target object, and the semantic detection loss reflects the accuracy of the model in identifying the category of the target object. By considering both losses and balancing their contributions through weighted processing, the target detection model can learn sample information more comprehensively during the training process, not only focusing on the position of the target object, but also focusing on its semantic information. This can improve the performance and accuracy of the model, enabling the model to more accurately identify and locate target objects in practical applications.

[0241] Before extracting image features of the sample image using the feature extraction network in the target detection model, the electronic device performs block processing on the sample image to obtain an image block sequence, encodes each image block to obtain an image block encoding sequence, and then extracts features of the image block encoding sequence using the feature extraction network. Block processing can decompose a large sample image into multiple small image blocks, each with relatively low resolution and reduced computational complexity. At the same time, encoding of the image blocks can enable the feature extraction network to better understand and utilize the information of the image blocks, including position information and local features. This enables the target detection model to capture local features more meticulously, with less computational resources required for each image block, allowing parallel processing in a graphics processing unit or distributed computing environment, reducing computational burden and speeding up training, while focusing on local feature information of the sample image to improve the accuracy of target detection and enable the model to more accurately identify and locate target objects.

[0242] When the sample image includes at least one of an RGB image and an infrared image, the electronic device can train the target detection model using different types of sample images. The RGB image contains rich color information and can reflect the appearance characteristics of the target object; the infrared image can provide thermal radiation information of the target object, and has better detection effect for some target objects with special performance in the infrared wave band. Rich training data of the model enables the model to learn more extensive features, improves the generalization ability and adaptability of the model, and can accurately perform target detection when facing different types of images to be processed, thereby expanding the application range of the model.

[0243] The electronic device performs a target detection model training method, obtains training samples, extracts features of sample images using a feature extraction network in the target detection model, performs target detection using a classification layer, determines a model loss based on the detection result and the annotation information, and finally updates the model parameters. Through such a training process, the target detection model can continuously learn and optimize its own parameters, so that it can accurately identify and locate the target object. In practical applications, accurate target detection is very important in many fields, such as security monitoring, autonomous driving, etc., which can improve the safety and reliability of the system.

[0244] Further, the electronic device determines a position detection loss based on the position information of the annotated and sample semantic objects, determines a semantic detection loss based on the annotated and sample semantic objects, and obtains a model loss of the target detection model by weighted processing of the two. The position detection loss and the semantic detection loss respectively reflect the performance of the model from different angles, and the weighted processing can balance the contributions of the two, so that the model can learn the sample information more comprehensively during the training process. In this way, the performance and accuracy of the model can be improved, so that the model can more accurately identify and locate the target object in practical applications, providing a more reliable basis for subsequent image processing and analysis.

[0245] Before extracting image features of the sample image using the feature extraction network in the target detection model, the electronic device performs block processing on the sample image to obtain an image block sequence, encodes the image blocks to obtain an image block code sequence, and then extracts features of the image block code sequence using the feature extraction network. Block processing and encoding can decompose the sample image into multiple small and easy-to-process units, each of which contains local feature information. The feature extraction network can process these units in parallel, improving the processing efficiency. At the same time, this method can also enable the model to capture local features more meticulously, improve the accuracy of target detection, and enable the model to accurately identify and locate the target object even when facing complex image scenes.

[0246] Before step S120 is performed, each pixel point in the pixel region of interest in the image to be processed is subjected to mask processing, and the color value thereof is set to a specified color value (such as black). Since the color of the pixel region of interest is less different from the color of the light source, if mask processing is not performed, the color information thereof will interfere with color temperature estimation. After mask processing is performed, the region will not be used for color temperature estimation subsequently, interference of the pixel region of interest on color temperature estimation is avoided, the light source color temperature value estimation more accurately reflects the color temperature of the actual light source, a more reliable basis is provided for subsequent determination of the white balance gain, and the effect of color correction is improved.

[0247] When obtaining the training samples, a large amount of data is preliminarily labeled by using the language model and / or the image recognition model, and the training samples are screened based on the labeling results. The professional personnel label the images one by one, and the labeling efficiency is low. In this way, the language model and the image recognition model are used to preliminarily label a large amount of images quickly, and the labeling efficiency is improved. Meanwhile, the high-quality training samples are screened by calculating the confidence scores, the quality of the training samples is ensured, the target detection model can learn more accurate features and patterns in the training process, and the performance and accuracy of the model are improved.

[0248] In the palm swiping payment device, the light source is used for light compensation, and the color temperature of the light source is set to be between 5000K and 7000K (such as 5500K, 6000K or 6500K). When the user performs palm swiping payment, the ambient light may be blocked, resulting in poor image quality. The light source with a suitable color temperature can provide stable and uniform illumination conditions, so that the collected image is clearer and more accurate in color. In this way, the accuracy of image feature extraction is improved when color correction and identity recognition are performed, so that the success rate of identity recognition and the security of payment processing are improved, and the payment experience of the user is ensured.

[0249] In the color correction process, the color correction module determines whether the algorithm converges, adjusts the algorithm and parameters and repeats the steps again if the algorithm does not converge, and converges. In the actual color correction process, due to the complexity of image data and the limitation of the algorithm, the ideal correction effect may not be achieved at one time. By determining whether the algorithm converges and adjusting, the result of color correction can be continuously optimized, so that the color of the processed image is more accurate and natural, and the quality and usability of the image are improved.

[0250] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the description.

[0251] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An image processing method performed by an electronic device, comprising: performing target detection on a to-be-processed image to determine a region of interest in the to-be-processed image; performing color temperature value estimation on a non-region of interest in the to-be-processed image to determine a light source color temperature value estimation value of the to-be-processed image; determining a white balance gain for the to-be-processed image according to the light source color temperature value estimation value of the to-be-processed image and a historical light source color temperature value of a reference image; the reference image and the to-be-processed image are captured by a same image capturing device; the reference image comprises at least one of K frames of images captured by the image capturing device before the to-be-processed image, K being a positive integer; and performing color correction on at least the region of interest in the to-be-processed image according to the white balance gain for the to-be-processed image.

2. The method of claim 1, wherein the determining the white balance gain for the to-be-processed image according to the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image comprises: performing median filtering on the light source color temperature value estimation value of the to-be-processed image and the historical light source color temperature value of the reference image to obtain a reference light source color temperature value of the to-be-processed image; and determining the white balance gain for the to-be-processed image according to the reference light source color temperature value of the to-be-processed image.

3. The method of claim 1 or 2, wherein the region of interest is a pixel region reflecting a biometric feature; and the method further comprises: performing identity recognition according to the color corrected region of interest to obtain an identity recognition result.

4. The method of claim 3, wherein the electronic device is a payment device, and the region of interest is a palm print pixel region or a face region; and after the performing identity recognition according to the color corrected region of interest to obtain an identity recognition result, the method further comprises: performing payment processing based on the identity recognition result.

5. The method of any one of claims 1 to 4, wherein the determining the light source color temperature value estimation value of the to-be-processed image according to the non-region of interest in the to-be-processed image comprises: performing white point detection processing on the non-region of interest to obtain a reference white point in the non-region of interest; and determining the light source color temperature value estimation value of the to-be-processed image according to a color value of the reference white point.

6. The method of claim 5, wherein the determining the light source color temperature value estimation value of the to-be-processed image according to the color value of the reference white point comprises: converting the color value of the reference white point to a target chroma space to obtain a chroma coordinate corresponding to the color value of the reference white point; determining a reference point closest to the chroma coordinate of the reference white point on a Planck curve; and determining the light source color temperature value estimation value of the to-be-processed image according to a color temperature corresponding to the reference point.

7. The method of any one of claims 1 to 6, wherein the performing target detection on a to-be-processed image to determine a region of interest in the to-be-processed image comprises: ​ performing target detection on the to-be-processed image by using a target detection model to obtain a detection result, the detection result including a semantic object identified from the to-be-processed image and position information of the semantic object in the to-be-processed image; if it is determined according to the detection result that the to-be-processed image includes a target semantic object, determining, according to the position information of the target semantic object in the to-be-processed image, a pixel region in which the target semantic object is located in the to-be-processed image as a pixel region of interest.

8. The method of claim 7, wherein the target detection model is trained in the following manner: obtaining a training sample, the training sample including a sample image and label information of the sample image, the label information indicating a labeled semantic object labeled for the sample image and sample position information of the labeled semantic object in the sample image; performing image feature extraction on the sample image by using a feature extraction network in the target detection model to obtain image features of the sample image; performing target detection on the sample image by using a classification layer in the target detection model to obtain a sample semantic object in the sample image and position information of the sample semantic object in the sample image; determining a model loss of the target detection model based on the sample semantic object and the position information of the sample semantic object in the sample image, and the labeled semantic object labeled for the sample image and the sample position information of the labeled semantic object in the sample image; updating a model parameter of the target detection model based on the model loss.

9. The method of claim 8, wherein the determining the model loss of the target detection model based on the sample semantic object and the position information of the sample semantic object in the sample image, and the labeled semantic object labeled for the sample image and the sample position information of the labeled semantic object in the sample image comprises: determining a position detection loss based on the sample position information of the labeled semantic object in the sample image and the position information of the sample semantic object in the sample image; determining a semantic detection loss based on the labeled semantic object labeled for the sample image and the sample semantic object; performing weighted processing based on the position detection loss and the semantic detection loss to obtain the model loss of the target detection model.

10. The method of claim 8 or 9, wherein before the performing image feature extraction on the sample image by using the feature extraction network in the target detection model to obtain the image features of the sample image, the method further comprises: performing block processing on the sample image to obtain a sequence of image blocks; performing image encoding on each image block in the sequence of image blocks to obtain a sequence of image block encodings; the performing image feature extraction on the sample image by using the feature extraction network in the target detection model to obtain the image features of the sample image comprises: performing feature extraction on the sequence of image block encodings by using the feature extraction network in the target detection model to obtain the image features of the sample image.

11. The method of any one of claims 8-10, wherein the sample image comprises at least one of an RGB image and an infrared image.

12. An image processing apparatus comprising: a region determination module configured to perform target detection on a to-be-processed image to determine a region of interest in the to-be-processed image; a color temperature value estimation module configured to perform color temperature value estimation on a non-region of interest in the to-be-processed image other than the region of interest to determine a light source color temperature value estimation of the to-be-processed image; a gain determination module configured to determine a white balance gain for the to-be-processed image according to the light source color temperature value estimation of the to-be-processed image and a historical light source color temperature value of a reference image, the reference image and the to-be-processed image being captured by a same image capturing device, the reference image comprising at least one of K frames of images captured by the image capturing device before the to-be-processed image, K being a positive integer; and a color correction module configured to perform color correction on at least the region of interest in the to-be-processed image according to the white balance gain for the to-be-processed image.

13. An electronic device comprising: a processor; a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, implement the method of any one of claims 1-11.

14. A computer readable storage medium having computer readable instructions stored thereon, the computer readable instructions, when executed by a processor, implement the method of any one of claims 1-11.

15. A computer program product comprising computer readable instructions, the computer readable instructions, when executed by a processor, implement the steps of the method of any one of claims 1-11. ​

Images