Image white balance processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image white balance processing method and device, equipment and a storage medium, and the method comprises the steps: carrying out the target detection of a to-be-processed image, and determining a focus pixel region in the to-be-processed image; performing color temperature estimation according to a non-concerned pixel region except the concerned pixel region in the to-be-processed image, and determining an estimated light source color temperature corresponding to the to-be-processed image; determining a white balance gain for the to-be-processed image according to the estimated light source color temperature corresponding to the to-be-processed image and the historical light source color temperature corresponding to the reference image; the reference image and the to-be-processed image are acquired by the same image acquisition device; the reference image comprises at least one of the first K frames of images of the to-be-processed image, and K is a positive integer; and according to the white balance gain for the to-be-processed image, at least carrying out white balance processing on a concerned pixel region in the to-be-processed image. By adopting the method, the color of the region of interest in the image after white balance processing can be more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly, to an image white balance processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the progress of image processing technology, image shooting devices such as cameras and mobile phones are widely used, and the quality of images will be affected by light sources in the environment, resulting in color cast phenomenon of images.

[0003] In the related art, the method of white balance is mainly used to solve the problem of image color cast, but the quality of the image after white balance processing in the related art is still not good enough. SUMMARY

[0004] Therefore, the embodiments of the present application provide an image white balance processing method, device, equipment and storage medium, which can make the color of the concerned area of the image after white balance processing more accurate.

[0005] In a first aspect, the embodiments of the present application provide an image white balance processing method, comprising: performing target detection on a to-be-processed image to determine a concerned pixel area in the to-be-processed image; performing color temperature estimation on a non-concerned pixel area in the to-be-processed image except the concerned pixel area to determine an estimated light source color temperature corresponding to the to-be-processed image; determining a white balance gain for the to-be-processed image according to the estimated light source color temperature corresponding to the to-be-processed image and a historical light source color temperature corresponding to a reference image; the reference image and the to-be-processed image are collected by the same image collection device; the reference image includes at least one of the first K frames of images of the to-be-processed image, and K is a positive integer; and performing white balance processing on at least the concerned pixel area in the to-be-processed image according to the white balance gain for the to-be-processed image.

[0006] Secondly, embodiments of this application provide a white balance processing device, including a region determination module, a color temperature estimation module, a gain determination module, and a white balance processing module. The region determination module is used to perform target detection on the image to be processed, determining the region of interest in the image to be processed; the color temperature estimation module is used to estimate the color temperature of the non-interested pixel regions in the image to be processed based on the non-interested pixel regions, excluding the region of interest, to determine the estimated light source color temperature corresponding to the image to be processed; the gain determination module is used to determine the white balance gain for the image to be processed based on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to a reference image; the reference image and the image to be processed are acquired by the same image acquisition device; the reference image includes at least one of the previous K frames of the image to be processed, where K is a positive integer; the white balance processing module is used to perform white balance processing on at least the region of interest in the image to be processed based on the white balance gain for the image to be processed.

[0007] In one possible implementation, the gain determination module includes a color temperature acquisition submodule and a gain determination submodule; the color temperature acquisition submodule is used to perform median filtering on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image to obtain the reference light source color temperature of the image to be processed; the gain determination submodule is used to determine the white balance gain of the image to be processed based on the reference light source color temperature of the image to be processed.

[0008] In one possible implementation, the region of interest is a region of pixels reflecting biometric features; the device further includes an identity recognition module, and the white balance processing module is further configured to perform white balance processing on at least the region of interest in the image to be processed based on the white balance gain for the image to be processed, to obtain the processed region of interest; the identity recognition module is configured to perform identity recognition based on the processed region of interest, to obtain an identity recognition result.

[0009] In one possible implementation, the method is applied to a payment device, where the focus pixel area is a palm print pixel area or a face area; the device further includes a payment processing module for performing payment processing based on the identity recognition result.

[0010] In one possible implementation, the color temperature estimation module includes a white point detection submodule and a color temperature estimation submodule; the white point detection submodule is used to perform white point detection processing on the non-interest pixel region to obtain a reference white point in the non-interest pixel region; the color temperature estimation submodule is used to determine the estimated light source color temperature corresponding to the image to be processed based on the color value of the reference white point.

[0011] In one possible implementation, the color temperature estimation submodule is further configured to convert the color value of the reference white point to the target color space to obtain the chromaticity coordinates corresponding to the color value of the reference white point; determine the reference point closest to the chromaticity coordinates of the reference white point on the Planck curve; and determine the estimated light source color temperature corresponding to the image to be processed based on the color temperature corresponding to the reference point.

[0012] In one possible implementation, the region determination module includes a detection submodule and a region determination submodule. The detection submodule is used to perform target detection on the image to be processed using a target detection model to obtain a detection result. The detection result includes a semantic object identified from the image to be processed and the location information of the semantic object in the image to be processed. The region determination submodule is used to determine the pixel region in the image to be processed where the target semantic object is located as a region of interest when it is determined from the detection result that the image to be processed includes a target semantic object.

[0013] In one possible implementation, the apparatus includes: a sample acquisition module, a feature extraction module, a detection module, a loss determination module, and a model update module; the sample acquisition module is used to acquire training samples, the training samples including sample images and label information of the sample images, the label information indicating the labeled semantic objects annotated for the sample images and the sample position information of the labeled semantic objects in the sample images; the feature extraction module is used to extract image features from the sample images using the feature extraction network in the target detection model to obtain the image features of the sample images; the detection module is used to perform target detection on the sample images using the classification layer in the target detection model to obtain the sample semantic objects in the sample images and the position information of the sample semantic objects in the sample images; the loss determination module is used to determine the model loss of the target detection model based on the sample semantic objects and the position information of the sample semantic objects in the sample images, as well as the labeled semantic objects annotated for the sample images and the sample position information of the labeled semantic objects in the sample images; the model update module is used to update the model parameters of the target detection model based on the model loss.

[0014] In one possible implementation, the loss determination module includes: a location detection loss determination submodule, a semantic loss determination submodule, and a model loss determination submodule; the location detection loss determination submodule is used to determine the location detection loss based on the sample location information of the labeled semantic object in the sample image and the location information of the sample semantic object in the sample image; the semantic loss determination submodule is used to determine the semantic detection loss based on the labeled semantic object annotated for the sample image and the sample semantic object; the model loss determination submodule is used to perform weighted processing based on the location detection loss and the semantic detection loss to obtain the model loss of the target detection model.

[0015] In one possible implementation, the device further includes: an image segmentation module and an image encoding module; the image segmentation module is used to segment the sample image into blocks to obtain an image block sequence; the image encoding module is used to encode each image block in the image block sequence to obtain an image block encoding sequence; the feature extraction module is further used to use the feature extraction network in the target detection model to extract features from the image block encoding sequence to obtain the image features of the sample image.

[0016] In one possible implementation, the sample image includes at least one of RGB images and infrared images.

[0017] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; the memory stores computer-readable instructions, which, when executed by the processor, implement the above-described method.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the above-described method.

[0019] Fifthly, embodiments of this application provide a computer program product including computer-readable instructions that, when executed by a processor, implement the steps of the above-described method.

[0020] In this application, firstly, color temperature estimation is performed on the non-interested pixel areas in the image to be processed, excluding the pixel areas of interest, to determine the estimated light source color temperature corresponding to the image to be processed. This avoids inaccurate color temperature estimation due to the small difference between the actual color of the object in the pixel areas of interest and the color of the light source in the environment from which the image to be processed originates. This ensures that the estimated light source color temperature more accurately reflects the color temperature of the light source in the environment from which the image to be processed originates. Then, the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature of a reference image acquired before the image to be processed are combined to jointly determine the white balance gain for the image to be processed. This allows the historical light source color temperature of the reference image to assist in determining the white balance gain, thereby improving the accuracy of the white balance gain determination. Finally, based on the white balance gain for the image to be processed, white balance processing is performed on at least the pixel areas of interest in the image to be processed, making the colors of the pixel areas of interest in the processed image more accurate. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A block diagram of an electronic device provided in an embodiment of this application is shown;

[0023] Figure 2 A schematic flowchart of an image white balance processing method proposed in an embodiment of this application is shown;

[0024] Figure 3 It shows Figure 2 A flowchart illustrating step S110;

[0025] Figure 4 It shows Figure 2 A flowchart illustrating step S120;

[0026] Figure 5 This paper shows a schematic diagram of the Planck curve proposed in the embodiments of this application in the CIE xy chromaticity coordinate system;

[0027] Figure 6 A schematic diagram illustrating the color gamut range of an RGB camera under different color temperatures, as proposed in an embodiment of this application, is shown.

[0028] Figure 7 It shows Figure 2 A flowchart illustrating step S130;

[0029] Figure 8 This paper shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0030] Figure 9 This paper shows another schematic flowchart of an image white balance processing method proposed in an embodiment of this application;

[0031] Figure 10 It shows Figure 9 A flowchart illustrating step S180;

[0032] Figure 11 This paper shows a schematic diagram of the structure of a target detection model provided in an embodiment of the present application;

[0033] Figure 12 An application schematic diagram of an electronic device for performing the methods of embodiments of this application is shown;

[0034] Figure 13 The image shown is an example of a related technique where images of a hand captured at different color temperatures are processed with white balance.

[0035] Figure 14 This illustration shows a schematic diagram of a palm payment process performed using a processor, as provided in an embodiment of this application. Detailed Implementation

[0036] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0037] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the actual situation. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0040] It should be noted that "multiple" as mentioned in this article refers to two or more.

[0041] The following describes an exemplary application of the image white balance processing method provided in this application. The image white balance processing method provided in this application can be applied to, for example... Figure 1 In the electronic device shown.

[0042] See Figure 1 , Figure 1 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application. The electronic device 100 may be a terminal device or a server. The electronic device may be used to execute the image white balance processing method provided in this application.

[0043] In some embodiments, the server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0044] Terminal devices can be smartphones, tablets, laptops, in-vehicle terminals, payment terminals (such as palm payment terminals and facial recognition payment terminals), access control devices, etc., but are not limited to these. The terminal devices are equipped with image acquisition devices to acquire images to be processed.

[0045] Figure 1 The illustrated electronic device 100 includes at least one processor 110, a memory 150, and at least one network interface 120. Various components within the electronic device 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to implement communication between these components. In addition to a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 1 The general labeled all buses as Bus System 140.

[0046] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0047] The memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 150 may optionally include one or more storage devices physically located away from the processor 110.

[0048] The memory 150 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 150 described in this application embodiment is intended to include any suitable type of memory.

[0049] In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0050] Operating system 151 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0051] The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 120 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0052] In some embodiments, the image white balance processing method provided in this application can be implemented in software. Figure 1An image white balance processing device 155 stored in memory 150 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: an image region determination module 155, a color temperature estimation module 1552, a gain determination module 1553, and a white balance processing module 1554. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0053] The following will describe the method provided in the embodiments of this application by referring to exemplary applications and implementations of the electronic devices provided in the embodiments of this application.

[0054] Please read Figure 2 , Figure 2 The present application also provides an image white balance processing method, which can be applied to an electronic device, such as the aforementioned terminal device or server. The method includes:

[0055] Step S110: Perform target detection on the image to be processed to determine the region of interest in the image.

[0056] The image to be processed refers to the image that requires white balance processing. It can be an image captured in real-time by an image acquisition device. The image to be processed can be an RGB image or an infrared image.

[0057] For example, the image to be processed can be a captured face image, a hand image, or an image captured of body parts, vehicles, animals, pedestrians, urban buildings, or natural scenery.

[0058] Object detection is an important task in computer vision, which aims to identify specific categories of objects and their locations within an image.

[0059] In this context, the region of interest in the image to be processed refers to the pixel region where the semantic object of interest (i.e., the target of interest) is located. Target detection in the image to be processed can identify whether the semantic object of interest exists and its location within the image. The location of the semantic object can be indicated by the position of the detection box surrounding the pixel region containing the semantic object. The detection box can be a rectangular box.

[0060] The focus target can be set as needed; the focus target can be palm prints, faces, etc. You can set one or more focus targets.

[0061] In this context, the regions in the image to be processed other than the regions of interest are considered non-interested regions.

[0062] The areas of interest typically contain information that the user or application cares about most, such as faces, specific objects, or logos; while the areas of non-interest typically contain background information or other secondary elements.

[0063] When performing object detection on an image to be processed, the identified regions of interest in the image can be one or more. If there are multiple regions of interest, the semantic objects in each region of interest may be different.

[0064] In step S110 above, an object detection model can perform object detection on the image to be processed, and determine the region of interest in the image based on the detection results output by the object detection model. The object detection model can be a deep learning model, such as Mask R-CNN (Mask Region Convolutional Neural Network) or U-Net (U-shaped Convolutional Neural Network), etc., capable of object detection. It can be understood that the object detection process is equivalent to semantic recognition of the image, identifying whether the image contains the target of interest and, if so, its location within the image.

[0065] In one possible implementation, please refer to Figure 3 As shown, step S110 includes steps S112-S114:

[0066] Step S112: Use the object detection model to perform object detection on the image to be processed and obtain the detection results. The detection results include the semantic objects identified from the image to be processed and the position information of the semantic objects in the image to be processed.

[0067] Object detection models can include feature extraction networks and classification layers. The specific process of using an object detection model to detect objects in an image is as follows:

[0068] The feature extraction network in the object detection model is used to extract features from the image to be processed, obtaining the image features of the image to be processed. Then, the classification layer in the object detection model is used to classify the image to be processed based on the image features, obtaining the semantic objects of the image to be processed and the location information of the semantic objects in the image to be processed.

[0069] In some embodiments, before inputting the object detection model, the method may further include: dividing the image to be processed into blocks to obtain an image block sequence of the image to be processed, and then encoding the image block sequence of the image to be processed to obtain an image block encoding sequence of the image to be processed. Step S112 above includes: using the feature extraction network in the object detection model to extract features from the image block encoding sequence of the image to be processed to obtain image features of the image to be processed. Then, using the classification layer in the object detection model, object detection is performed on the image to be processed based on the image features of the image to be processed, obtaining the semantic objects of the image to be processed and the location information of the semantic objects in the image to be processed.

[0070] When dividing the image to be processed into blocks to obtain an image block sequence, the image to be processed can be divided into image blocks of fixed size and arranged according to the spatial order of the image blocks (for example, image blocks located in different rows in the image to be processed are arranged in the order of top to bottom, and image blocks located in the same row in the image to be processed are arranged in the order of left to right) to obtain the image block sequence.

[0071] Dividing the image to be processed into multiple image patches allows the feature extraction network in the subsequent object detection model to extract features from multiple image patches in parallel. This improves the object detection efficiency of the object detection model. Moreover, since the resolution of an image patch is smaller than that of the entire image to be processed, the computational complexity of feature extraction is lower than that of feature extraction on a per-image-patch basis, while also preserving the detailed information in the image to be processed.

[0072] In one possible implementation, encoding each image block in the image block sequence can specifically involve performing linear encoding and positional encoding on the image blocks to obtain the final image encoding sequence, or performing linear encoding and positional encoding on the image blocks and then fusing them to obtain the final image encoding sequence.

[0073] For example, each image block in the image block sequence can be positionally encoded to obtain a positional code for each image block, which is used to characterize the position of the image block in the image to be processed; each image block in the image block sequence can be linearly encoded to obtain a linear code for each image block; for each image block in the image block sequence, the positional code and the linear code of the image block are fused to obtain the image code of the image block; according to the position of the image block in the image block sequence, the image codes of multiple image blocks are combined to obtain the image code sequence of the image block sequence.

[0074] In the process of positional encoding of each image block in the image block sequence, for each image block, the positional information (e.g., coordinates) of the image block relative to the overall image is calculated. Then, the positional information of the image block is converted into an encoded form. Subsequently, the encoded form of the positional information of each image block can be merged with its own pixel value or feature representation to obtain the positional code of the image block. In the process of linear encoding of each image block in the image block sequence, a linear transformation process can be applied to the image block to obtain the linearly transformed result, which is the linear encoding feature of the image block.

[0075] The above-mentioned methods for fusing positional and linear codes of image blocks can include concatenating, multiplying, or adding positionally to the positional and linear codes of the image blocks.

[0076] By encoding the positions of image patches, subsequent feature extraction networks can better understand and utilize the positional information of each patch in the image to be processed. This allows them to capture the spatial information and structure within the image, helping the feature extraction network to better understand and utilize spatial relationships and contextual information. Linear encoding of image patches enables the feature extraction network to capture continuous features and patterns in the image to be processed, improving the accuracy of understanding and classifying the content.

[0077] In another possible implementation, encoding each image block in the image encoding sequence can specifically involve: calculating the color gradient of each pixel in the image block based on the color value of each pixel to obtain the gradient direction and gradient magnitude of the pixel; determining the HOG feature (Histogram of Oriented Gradients) of the image block based on the gradient direction and gradient magnitude of each pixel in the image block; and concatenating the HOG features of each image block in the image block sequence according to the order of the image blocks in the image block sequence to obtain the image block encoding sequence.

[0078] By dividing the image to be processed into blocks and extracting HOG features from each block, the HOG features of each block are combined into a large one-dimensional vector (image block encoding sequence). This image block encoding sequence contains color gradient information from the image to be processed, enabling effective capture of the semantics of the image during subsequent feature extraction based on the image block encoding sequence. Furthermore, extracting HOG features from image blocks requires minimal computation, which is particularly beneficial when the proposed solution is applied to terminal devices, reducing the computational burden on the device.

[0079] Step S114: If the detection results determine that the image to be processed contains a target semantic object, the pixel region in which the target semantic object is located in the image to be processed is determined as the region of interest based on the location information of the target semantic object in the image to be processed.

[0080] The target semantic object is the semantic object of interest mentioned above. If the location information of the target semantic object in the image to be processed indicates the location information of the detection box of the target semantic object, then the pixel region of the target semantic object in the image to be processed can be regarded as the pixel region within the bounding box, that is, the region of interest.

[0081] The region of interest in the image containing the target semantic object can be a region of interest that needs to be focused on later, for example, for further image processing. If the image does not contain the target semantic object, then white balance processing of the image does not need to be performed, avoiding ineffective white balance processing. The next image can be obtained as the new image to be processed, and the above process can be followed. Step S120: Estimate the color temperature of the non-interested pixel regions in the image to be processed, excluding the region of interest, to determine the estimated light source color temperature corresponding to the image.

[0082] The estimated light source color temperature corresponding to the image to be processed is the color temperature of the light source in the environment from which the image to be processed originates, determined by color temperature estimation.

[0083] Color temperature is a unit of measurement representing the color components in light. It is an important parameter describing the color characteristics of a light source, and light sources with different color temperatures emit light of different colors. Color temperature is defined based on the principle of blackbody radiation. When the color of the light emitted by a light source is the same as the color of the radiation from a blackbody heated to a certain temperature, that temperature of the blackbody is defined as the color temperature of the light source. If the color of the semantic object of interest (i.e., the target semantic object mentioned above) is small compared to the color of the light emitted by the light source—for example, if the target semantic object is a hand or a face, its color is yellow or similar to yellow, and the light from the light source is usually also yellow, the difference between the two colors is small. Therefore, if the pixel region of interest containing the target semantic object in the image to be processed is used for color temperature estimation, the accuracy of the estimated light source color temperature will be low. Therefore, in this application, to avoid inaccurate color temperature estimation due to the small difference between the color of the target semantic object itself and the color of the light source, the non-interest pixel region in the image to be processed is used for color temperature estimation.

[0084] In some embodiments, before performing step S120, the method may further include: performing masking processing on each pixel in the pixel region of interest in the image to be processed, so as to set the color value of each pixel in the pixel region of interest to a specified color value, wherein the specified color value may have an influence on the color temperature estimation that is less than an influence threshold, for example, the specified color value may be black.

[0085] When the region of interest is set to a specified color (such as black), although the pixels within that region still exist in the image to be processed, their color attributes no longer reflect the actual color information of the region of interest because their color has been changed. Therefore, the pixel region with the specified color value can be excluded from subsequent color temperature estimation.

[0086] In one possible implementation, step S120 may be: using a pre-trained color temperature recognition model to estimate the color temperature of non-interest pixel regions in the image to be processed, excluding the pixel regions of interest, to obtain the estimated light source color temperature corresponding to the image to be processed.

[0087] The color temperature recognition model is trained on multiple color temperature sample images with color temperature labels. The training process of the color temperature recognition model is as follows: the feature extraction network in the color temperature recognition model extracts features from the color temperature sample images to obtain image features; the classification network in the color temperature recognition model classifies the image features to obtain the color temperature prediction result of the color temperature sample image; a loss value is obtained based on the color temperature prediction result and the color temperature label; and the model parameters of the color temperature recognition model are adjusted based on the loss value. The neural network used in the color temperature recognition model can be, but is not limited to, ResNet (Residual Network), VGG (Visual Geometry Group Network), and Inception (Inception Convolutional Neural Network).

[0088] In another possible implementation, please refer to Figure 4 The above step S120 includes:

[0089] Step S122: Perform white point detection processing on the non-interest pixel areas to obtain reference white points in the non-interest pixel areas.

[0090] It is worth mentioning that the image used for white point detection is an RGB image. If the image to be processed is not an RGB image, it can be converted to an RGB image first. An RGB image is an image constructed using the RGB color mode, which is an industry color standard. Various colors are obtained by changing the three color channels of red, green, and blue and superimposing them.

[0091] The methods used for white point detection processing may include the maximum RGB method.

[0092] When using the maximum RGB method to detect white points in non-interested regions of an image, specifically, for each pixel in the non-interested region, the top N pixels with the highest RGB channel values ​​in the region are selected as reference white points, where N is an integer greater than 1. The highest pixel value refers to selecting pixels from all pixels in the non-interested region that have N highest max(R,G,B) values, and whose sum of RGB channel values ​​is greater than a preset threshold, such as (255,255,255), (255,255,254), or (254,255,255), etc. Here, max(R,G,B) represents selecting the maximum value among the red, green, and blue channels. Ideally, pixels close to white usually have high values ​​in all three channels, and these values ​​are relatively close. Therefore, if a pixel's max(R,G,B) value is close to 255, and the values ​​of the R, G, and B channels are not significantly different, then this pixel is likely a pixel close to white.

[0093] Step S124: Determine the estimated light source color temperature corresponding to the image to be processed based on the color value of the reference white point.

[0094] In one possible implementation, step S124 may be to look up the estimated light source color temperature corresponding to the color value of the reference white point from a preset correspondence table, wherein the preset correspondence table stores multiple color temperatures and the color value corresponding to each color temperature.

[0095] In one possible implementation, step S124 includes:

[0096] Step S124a: Convert the color value of the reference white point to the target color space to obtain the chromaticity coordinates corresponding to the color value of the reference white point.

[0097] The target color space refers to the CIE xy chromaticity coordinate system, which is derived from the CIE XYZ color space. The CIE XYZ color space builds upon the RGB color space by mathematically selecting three ideal primary colors to replace the actual three primary colors, allowing each color to be described by three numerical values.

[0098] The CIE XYZ color space is defined by three color matching functions: X, Y, and Z. X represents the response of the red primary color; Y represents the response of the green primary color, which is generally considered to be the component closest to the intensity perceived by the human eye; and Z represents the response of the blue primary color. In the CIE xy chromaticity coordinate system, color is represented as a point in a two-dimensional plane (the xy coordinate plane). Points on this plane represent the chromaticity attribute of the color, ignoring intensity information. The xy coordinate plane is derived from the CIE XYZ color space using the following conversion formulas: where x = X / (X+Y+Z); y = Y / (X+Y+Z).

[0099] Given the color value of the reference white point, the color value of the reference white point can be converted to obtain the coordinates of the reference white point in the CIE XYZ color space. Then, the coordinates of the reference white point in the CIE XYZ color space can be converted to CIE xy chromaticity coordinates using a conversion formula to obtain a planar coordinate value. This planar coordinate value is the chromaticity coordinate corresponding to the color value of the reference white point.

[0100] Step S124b: On the Planck curve, determine the reference point that is closest to the chromaticity coordinates of the reference white point.

[0101] Among them, the Planck curve is obtained based on Planck's law, which describes the relationship between the energy distribution of blackbody radiation and temperature, such as... Figure 5 As shown, when the color of blackbody radiation is converted to the CIE xy chromaticity coordinate system, the points corresponding to the color of blackbody radiation in the CIE xy chromaticity coordinate system form a continuous curve. This curve is the Planck curve, meaning that the points on the Planck curve represent the radiation color of the blackbody at different temperatures. When the temperature of the blackbody is low (e.g., around 2000 K), its radiation color leans towards red. As the temperature increases, the color of the blackbody radiation gradually changes to yellow, then to white (e.g., around 5500 K). When the temperature of the blackbody is very high (e.g., around 10000 K), its radiation color leans towards blue.

[0102] like Figure 6 As shown, the color gamut range of the RGB camera under different color temperature conditions is illustrated. Figure 5 The multiple curves in the image represent the relative intensity changes of the red (R) and blue (B) color channels at different color temperatures.Figure 5 The horizontal axis represents the intensity of the red channel, and the vertical axis represents the intensity of the blue channel. From... Figure 5 As can be seen, the intensity of the red and blue channels changes with the color temperature. At higher color temperatures (such as 8500K), the intensity of the blue channel is greater; while as the color temperature decreases, the intensity of the red channel gradually increases while the intensity of the blue channel gradually decreases.

[0103] Under natural or artificial light, the color temperature characteristics of many light sources can be approximately described by blackbody radiation. That is, the color temperature of many common light sources (such as sunlight, incandescent lamps, and fluorescent lamps) can be approximated by a point on the Planck curve. Therefore, when a reference white point has a color value, it is converted to CIE xy chromaticity coordinates, and the point on the Planck curve closest to it is found as the reference point. The color temperature corresponding to this reference point can be used as the color temperature of the reference white point.

[0104] Step S124c: Determine the estimated light source color temperature of the image to be processed based on the color temperature corresponding to the reference point.

[0105] Among them, there are one or more reference white dots, and correspondingly, there are also one or more reference points.

[0106] If there is only one reference white point, then the color temperature of the reference point can be determined as the estimated light source color temperature corresponding to the image to be processed.

[0107] If there are multiple reference white points, in one possible implementation, the estimated light source color temperature of the image to be processed can be obtained by weighted calculation based on the color temperature corresponding to each reference white point.

[0108] In another possible implementation, the median or mean of the color temperature corresponding to each reference white point can be calculated as the estimated light source color temperature corresponding to the image to be processed.

[0109] In another possible implementation, multiple reference white points can be clustered to obtain the target category with the most reference white points. The median of the color temperatures corresponding to the multiple reference white points belonging to the target category is then calculated to obtain the estimated light source color temperature corresponding to the image to be processed.

[0110] By employing the above method, the principle of the color temperature characteristics of the actual light source being similar to blackbody radiation is realized, thereby accurately obtaining the estimated light source color temperature corresponding to the image to be processed.

[0111] Step S130: Determine the white balance gain for the image to be processed based on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image.

[0112] The reference image and the image to be processed are acquired by the same image acquisition device; the reference image includes at least one of the first K frames of the image to be processed, where K is a positive integer.

[0113] The reference image can be an image that does not include the target semantic object, and the target semantic object is the semantic object of interest. For example, it can be the latest image acquired before the image to be processed that does not include the target semantic object.

[0114] In one possible implementation, please refer to Figure 7 The above step S130 may include: step S132: determining a reference light source color temperature based on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image.

[0115] Specifically, step S132 can be: performing median filtering on the estimated light source color temperature of the image to be processed and the historical light source color temperature of the reference image to obtain the reference light source color temperature of the image to be processed.

[0116] Median filtering is a non-linear filtering technique that smooths out color temperature changes by applying median filtering to the current estimated color temperature and historical color temperatures, thus avoiding drastic fluctuations in subsequent white balance adjustments caused by anomalies in a single estimated value.

[0117] The above step S132 can also be specifically: weighted summation of the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image to obtain a reference light source color temperature.

[0118] When assigning weights to the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image, the weight of the estimated light source color temperature corresponding to the image to be processed can be determined based on the number of reference images, and the weight of the historical light source color temperature corresponding to each reference image can be assigned based on the acquisition interval between each reference image and the image to be processed. The number of reference images is positively correlated with the weight of the estimated light source color temperature corresponding to the image to be processed, that is, the more reference images there are, the smaller the weight of the estimated light source color temperature corresponding to the image to be processed. The acquisition interval corresponding to the reference image is negatively correlated with the weight of the historical light source color temperature corresponding to the reference image, that is, the longer the acquisition interval corresponding to the reference image, the smaller the weight of the historical light source color temperature corresponding to the reference image. The sum of the weight of the estimated light source color temperature corresponding to the image to be processed and the weight of the historical light source color temperature corresponding to the reference image is 1.

[0119] By employing a weighted summation method to obtain the reference light source color temperature of the image to be processed, the final color temperature estimate becomes more stable. Even if the current estimate is affected by noise, historical data can help smooth out these fluctuations, thus avoiding color temperature abrupt changes caused by sudden changes in ambient light.

[0120] Step S132 above can also involve determining whether the absolute value of the difference between the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image is less than a preset difference threshold. If it is less, the estimated light source color temperature corresponding to the image to be processed is determined as the reference light source color temperature; if it is not less, a weighted sum is performed on the estimated light source color temperatures corresponding to the image to be processed to obtain a reference light source color temperature.

[0121] When there are multiple reference images, the method for determining whether the difference between the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image is less than a preset difference threshold can be to determine whether the average or median of the absolute values ​​of the difference between the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image is less than a preset difference threshold.

[0122] Step S134: Determine the white balance gain of the image to be processed based on the color temperature of the reference light source of the image to be processed.

[0123] The goal of white balance is to make an image look natural under different lighting conditions, that is, to make an object that is originally white appear white under any light source.

[0124] To achieve white balance, it is usually necessary to obtain white balance gain for the three channels: red (R), green (G), and blue (B).

[0125] In one implementation, this can be achieved by calculating the gain ratio of the R, G, and B channels relative to a reference channel. For example, one channel is selected as a reference, and then the gain ratio of the other two channels relative to that reference channel is calculated. Specifically, step S134 above can be as follows: after detecting white points in the non-interest pixel region of the image to be processed to obtain multiple reference white points, the target reference white point closest to the point on the Planck curve corresponding to the color temperature of the reference light source is selected from the multiple reference white points. Based on the intensity values ​​of each color channel of the target reference white point, the channel with the largest intensity value among the red (R), green (G), and blue (B) channels can be selected as the reference channel, or the middle channel (green) can be selected as the reference channel. After selecting the reference channel, the gain of each channel is determined by the ratio between the intensity value of each channel of the target reference white point and the intensity value of the reference channel.

[0126] For example, if the pixel value of a target reference white point is (220, 210, 180), if the selected reference channel is the R channel, then the gain of the G channel is G_gain = 210 / 220, and the gain of the B channel is B_gain = 180 / 220; if the selected reference channel is the G channel, then the gain of the R channel is R_gain = 220 / 210, and the gain of the B channel is B_gain = 180 / 210.

[0127] In another approach, step S110 can also involve determining the white balance gain of the image to be processed based on the reference light source color temperature and the reference color temperature. Specifically, the reference color temperature is the color temperature of neutral white light, such as any value between 5400K and 5600K. For example, the reference color temperature can be set to 5500K. Subsequently, the gain of each color channel can be calculated based on the difference between the reference light source color temperature and the reference color temperature.

[0128] For example, the gain of channel G can be determined by the formula: R_gain=1+(T-T0) / P×α; the gain of channel B can be determined by the formula: B_gain=1-(6500K-5500K) / P×β.

[0129] Where T is the reference light source color temperature corresponding to the image to be processed; T0 is the reference color temperature; P is the reference color temperature difference, which is a preset value; α and β are preset constants.

[0130] It's worth noting that when the reference light source color temperature is greater than 5500K, similar to sunlight on a cloudy day or in the shade, the spectrum is biased towards blue. Therefore, the red channel needs more gain to compensate for this shift, while the blue channel may need less gain. Conversely, when the reference light source color temperature is less than 5500K, similar to sunlight in the morning or evening, the spectrum is biased towards red. Therefore, the blue channel needs more gain to compensate for this shift, while the red channel may need less gain. Thus, the gain calculation method for the red and blue channels can be set based on the above principles.

[0131] For example, if the color temperature of the reference light source is 6500K, it is similar to sunlight on a cloudy day or in the shade, with the spectrum biased towards blue. In this case, the gain of the R channel is R_gain = 1 + (6500K - 5500K) / 1000K × α; the gain of the B channel is B_gain = 1 - (6500K - 5500K) / 1000K × β, where 1000K in the above gain calculation formula is a fixed value, and α and β are constant values.

[0132] It should be noted that the above gain calculation method is only illustrative, and there are many other calculation methods, which will not be elaborated here.

[0133] Step S140: Perform white balance processing on at least the pixel region of interest in the image to be processed, based on the white balance gain for the image to be processed.

[0134] Specifically, for each pixel in the region of interest in the image to be processed, the gain of each channel included in the white balance gain for the image to be processed is multiplied by the color value of the corresponding channel of that pixel to obtain the white balance processed image.

[0135] For example, if the gain of the R channel is R_gain, the gain of the G channel is G_gain, the gain of the B channel is B_gain, and the intensity values ​​of each color channel of a pixel are (R1, G1, B1), then the intensity values ​​of each color channel of the pixel after white balance processing are (R2, G2, B2), where R2 = R1 × R_gain, G2 = G2 × G_gain, and B2 = B1 × B_gain.

[0136] Since the region of interest typically contains information of most interest to the user or application, such as faces, hands, specific objects, or logos, and the colors of faces and hands are usually yellow or white, and the light from a light source is also typically yellow with little difference between the two, using the region of interest containing the target semantic object in the image to be processed for color temperature estimation would lead to low accuracy in the estimated light source color temperature. By employing the method described in this application, color temperature estimation is performed on the non-region of interest in the image to be processed, thus determining the estimated light source color temperature corresponding to the image. This avoids the situation where the actual color of the object presented in the region of interest is very similar to the color of the light source in the environment from which the image originates, leading to inaccurate color temperature estimation. This allows the estimated light source color temperature to more accurately reflect the color temperature of the light source in the environment from which the image originates. Subsequently, by combining the estimated light source color temperature corresponding to the image to be processed with the historical light source color temperature of a reference image acquired before the image to be processed, the white balance gain for the image to be processed is determined. This allows the historical light source color temperature of the reference image to be used to assist in determining the white balance gain of the image to be processed, thereby improving the accuracy of the white balance gain determination. Finally, based on the white balance gain for the image to be processed, white balance processing should be performed on at least the region of interest in the image to be processed, which can make the color of the region of interest in the processed image more accurate.

[0137] In one possible implementation, the pixel region of interest is a pixel region that reflects biological characteristics; the above step S140 includes: performing white balance processing on at least the pixel region of interest in the image to be processed according to the white balance gain for the image to be processed, to obtain the processed pixel region of interest.

[0138] After performing step S140, the method further includes: performing identity recognition based on the processed region of interest to obtain the identity recognition result.

[0139] In the process of identity recognition based on the processed pixel region of interest, features that can be used for identity recognition can be extracted from the processed pixel region of interest. The extracted features are then compared with multiple feature templates stored in the database. If there is a feature template with a similarity greater than a preset threshold, the identity recognition result of the user information corresponding to the feature template is obtained. If the similarity is not greater than the preset threshold, the identity recognition result indicating identity verification failure is obtained.

[0140] It is worth mentioning that the above method can be applied to electronic devices that require identity verification, such as access control devices, payment devices, and safes. After obtaining the identity verification result, corresponding operations can be performed based on the functions of the electronic device, such as access control management based on the identity verification result, or payment processing based on the identity verification result. In this case, please refer to 8, the electronic device may include an image acquisition device, which may include at least one of an RGB camera and an infrared camera.

[0141] To ensure accurate image recognition during user identification, where ambient light may be obstructed, preventing the electronic device from accurately identifying the image, in one embodiment of this application, the electronic device is further equipped with a light source for supplementary lighting.

[0142] The color temperature of the light source can be set between 5000K and 7000K. Furthermore, the color temperature of the light source can be 5500K, 6000K, or 6500K, etc.

[0143] In one possible implementation, the method is applied to a payment device, focusing on a palm print pixel area or a face area; after performing step S150, the method further includes: performing payment processing based on the identity recognition result.

[0144] By employing the above method, payment devices can more accurately white balance the relevant pixel areas in the image after acquiring it. This results in more natural colors in these areas during subsequent payment processing, preventing image quality issues caused by ambient lighting that could lead to incorrect identification and payment failure. This reduces the likelihood of misidentification or failure due to lighting conditions.

[0145] Please see Figure 9 In one possible implementation, the method further includes:

[0146] Step S150: Obtain training samples.

[0147] The training samples include sample images and label information for the sample images. The label information indicates the labeled semantic objects for the sample images and the sample location information of the labeled semantic objects in the sample images.

[0148] Among them, the sample location information of the labeled semantic object in the sample image indicates the detection box of the sample semantic object.

[0149] There are several ways to obtain training samples. For example, professionals can annotate multiple images one by one to determine the semantic objects and location information in the images. Alternatively, language models and / or image recognition models can be used to initially annotate a large amount of data, and training samples can be selected based on the annotation results. Another method is to select training samples from a target open-source dataset, such as the GVN dataset (Grounded Vision Narrators, a dataset used to study visual narratives and understand multimodal interactions) or MSCOCO (Microsoft Common Objects in Context, a large-scale image recognition and image-text dataset).

[0150] Training samples can include positive samples and negative samples. Among them, the sample images in the positive samples are images that include the target semantic object, and the labeled semantic object in the sample label of the sample image is the target semantic object; the sample images in the negative samples are images that do not include the target semantic object, and the labeled semantic object in the sample label of the negative samples is a non-target semantic object.

[0151] For example, if the target semantic object is a hand, then the sample images in the positive samples are images that include the hand, and the sample images in the negative samples are images that do not include the hand; if the target semantic object is a face, then the sample images in the positive samples are images that include the face, and the sample images in the negative samples are images that do not include the face; if the target semantic object is a vehicle, then the positive samples are images that include the vehicle, and the negative samples are images that do not include the vehicle.

[0152] By setting positive and negative samples, the feature extraction network can better understand the information or semantics in the image, thereby improving the model's judgment and decision-making abilities. Furthermore, the presence of negative samples allows the feature extraction network to generalize better when faced with new and unseen data. For example, if subsequent noise or anomalous features similar to negative samples exist, the model can effectively identify and exclude them, thus extracting key features more accurately.

[0153] For example, if the target semantic object is a hand, then step S150 above can be: acquiring multiple images of different objects captured by an image acquisition device. These objects can include the left hand, right hand, vehicles, faces, bodies, and furniture, etc. The ratio of the number of images including the hand to the number of images excluding the hand is within a preset range, such as 0.9-1.1, or the number of images including the hand is the same as the number of images excluding the hand. After acquiring multiple images, the images can be cleaned, and the remaining images after cleaning can be labeled to obtain sample images. Images including the hand are positive samples, and images excluding the hand are negative samples.

[0154] In one possible implementation, the sample image includes at least one of an RGB image and an infrared image.

[0155] Step S160: Use the feature extraction network in the object detection model to extract image features from the sample image to obtain the image features of the sample image.

[0156] A feature extraction network is a neural network used to encode features of a two-dimensional image. This image feature encoder can be constructed using convolutional neural networks, pooling neural networks, or Transformer networks, etc.

[0157] In this case, if the feature extraction network is a convolutional neural network, step S130 above can be to perform convolution and pooling operations on each image block in the image block sequence through convolutional layers and pooling layers to extract features of the sample image.

[0158] When the feature extraction network is a Transformer network, step S130 above can be as follows: For an image patch sequence, a convolutional neural network (CNN) can be used to extract features from the image patch sequence. Then, the extracted feature vector is used as the input of the Transformer network to further extract contextual features, thereby obtaining the features of the sample image. Alternatively, the pixel values ​​in the image patch can be normalized, and then a multi-layer self-attention mechanism and an attention mechanism can be used to extract features from the image patch sequence to obtain the features of the sample image.

[0159] It should be understood that the above-described feature extraction process is only illustrative. The corresponding feature extraction process will be different when the network structure of the feature extraction network is different, and it is not specifically limited in the embodiments of this application.

[0160] Step S170: Utilize the classification layer in the object detection model to perform object detection on the sample image based on the image features of the sample image, thereby obtaining the sample semantic object and the location information of the sample semantic object in the sample image.

[0161] For details on the process of object detection in the sample image, please refer to the specific description of step S110 in the foregoing embodiments, which will not be repeated here.

[0162] Step S180: Based on the sample semantic object and its position information in the sample image, as well as the labeled semantic object and its sample position information in the sample image, determine the model loss of the target detection model.

[0163] Please see Figure 10 In one possible implementation, step S180 includes:

[0164] Step S182: Determine the location detection loss based on the sample location information of the labeled semantic object in the sample image and the location information of the sample semantic object in the sample image.

[0165] Specifically, the location detection loss can be calculated based on the sample location information of the labeled semantic object in the sample image and the sample semantic object in the sample image, using the cross-entropy loss function, IoU loss function, weighted cross-entropy loss function, mean squared error loss function, Manhattan distance loss function, and cosine similarity loss function.

[0166] Step S184: Determine the semantic detection loss based on the labeled semantic objects and sample semantic objects annotated for the sample images.

[0167] Specifically, semantic detection loss can be obtained by calculating the loss based on the labeled semantic objects and sample semantic objects in the sample images using cross-entropy loss function, IoU loss function, weighted cross-entropy loss function, mean squared error loss function, Manhattan distance loss function, and cosine similarity loss function.

[0168] Step S186: Perform weighted processing based on the location detection loss and semantic detection loss to obtain the model loss of the target detection model.

[0169] The weighting coefficients for location detection loss and semantic detection loss can be set according to actual needs; the weighting coefficients for the two can be the same or different.

[0170] For example, if the location detection loss is L1 and the semantic detection loss is L2, then the model loss L totle It can be defined as: L totle =λ1×L1+λ2×L2;

[0171] Here, λ1 is the weighting coefficient for the location detection loss, and λ2 is the weighting coefficient for the semantic detection loss. The optimal values ​​of λ1 and λ2 can be determined experimentally to balance the importance of different tasks during training.

[0172] Step S190: Update the model parameters of the target detection model based on the model loss.

[0173] After obtaining the model loss, the model parameters can be adjusted based on the model loss to minimize it. The training termination condition can be that the number of iterations of the object detection model reaches a preset number, or the model loss is less than a preset loss threshold.

[0174] By employing the steps S150-S190 described above, the target detection model is trained using training samples with labeled information. This enables the target detection model to learn the correct classification labels and target box positions, ensuring that the subsequent target detection model can accurately identify target objects in the image and precisely locate the positions of these objects.

[0175] In one possible implementation, before performing step S160, the method further includes: dividing the sample image into blocks to obtain an image block sequence; encoding each image block in the image block sequence to obtain an image block encoding sequence; step S160 may also be: using the feature extraction network in the target detection model to extract features from the image block encoding sequence to obtain image features of the sample image.

[0176] In object detection tasks, since objects can appear anywhere in an image, by dividing the sample image into blocks and encoding each image block, the feature extraction network in the object detection model can extract features from the encoded sequence of the image blocks. This allows the object detection model to capture these local features more precisely. For example, in the object detection model performing pedestrian detection or hand detection tasks, even if the pedestrian or hand only occupies a small part of the image, by dividing the image into blocks, the object detection model can focus more on these key blocks, thereby improving detection accuracy.

[0177] Furthermore, since each image patch requires fewer computational resources compared to the entire image, it can be more easily placed into a graphics processor for efficient parallel computation. At the same time, since each image patch can be processed independently, parallel processing can be implemented in a graphics processor or distributed computing environment, so as to focus on the local feature information of the sample image while reducing the computational burden and speeding up the training.

[0178] For details regarding the process of segmenting the sample image into blocks and the subsequent feature extraction process, please refer to the specific description of step S110 above, which will not be repeated here.

[0179] like Figure 11The diagram illustrates the training process using sample images acquired by an infrared or RGB camera. These sample images contain label information. For each sample image, a feature extraction network in the object detection model extracts its features. The classification layer of the image detection model then performs object detection based on these features, obtaining the semantic objects and their locations within the sample images. Subsequently, a location detection loss L1 is determined based on the labeled semantic objects and the sample semantic objects themselves. A semantic detection loss L2 is then determined based on the labeled semantic objects and the sample semantic objects. Finally, a weighted average of the location and semantic detection losses is applied to obtain the model loss L of the object detection model. totle Subsequently, the model parameters of the target detection model can be adjusted based on the model loss.

[0180] The application of the image white balance processing method of this application to a palm payment device will be used as an example for illustration. Please refer to [link / reference]. Figure 12 As shown in (A), the palm payment device includes a processor (not shown), an image acquisition device (e.g., an RGB camera and an infrared camera), and a light source. The image acquisition device is connected to the processor and acquires images to obtain the image to be processed. The light source can be used to emit light with a specified color temperature (e.g., 6500K). Figure 12 As shown in (B), the light source can be a light guide ring, which is used to instruct the user to cover the position of the light guide ring with their palm when making a palm payment. The optical axis of the image acquisition device is located inside the light guide ring, and the number of image acquisition devices can be one or more. When there are multiple devices, the multiple image acquisition devices can include RGB image acquisition devices and infrared image acquisition devices.

[0181] The image acquisition device in a palm-swipe payment device can capture images, obtain images to be processed, and send the obtained images to the processor. When using the image acquisition device to capture multiple palm images, the lighting conditions in the actual environment are very complex, including natural light, indoor lighting, and light from the light guide ring of the palm-swipe payment device. These light sources may have different color temperatures, intensities, and directions, resulting in varying degrees of color casts in the captured palm images.

[0182] If, before payment processing, the white balance is applied to some images according to the determined white balance gain, the resulting images may still have significant color casts. Figure 13 The image shown still shows a hand with color cast even after white balance processing. Clearly, if one directly applies white balance... Figure 13Using the hand image shown for identity verification may result in inaccurate image feature extraction, ultimately leading to the inability to correctly identify the hand and thus preventing payment processing. Therefore, please refer to [link / reference needed]. Figure 14 As shown, the processor of this application may specifically include a palm detection module, a white balance processing module, and a payment processing module (not shown in the figure). Each module can perform the following processing procedures:

[0183] When the processor receives the image to be processed, it can call the hand detection module to perform the following steps:

[0184] Palm detection module: The palm detection module performs palm detection on the image collected by the palm payment device in real time. If there is no target pixel area (i.e., palm pixel area) in the collected image, the currently collected image is used as a reference image, and the color temperature of the reference image is estimated to determine the light source color temperature of the reference image, which is used as the historical light source color temperature corresponding to the reference image.

[0185] The palm detection module continues to detect palms in subsequently acquired images. If a palm pixel region is determined to exist in the image, color temperature is estimated based on the non-interest pixel region in the image to determine the estimated light source color temperature corresponding to the image.

[0186] The hand detection module processes hand detection according to the following procedure:

[0187] The real-time acquired images are divided into blocks to obtain a sequence of image blocks. This sequence is then encoded to obtain an encoded sequence. Subsequently, a feature extraction network in a target detection model (e.g., a hand detection model) is used to detect a hand on the encoded sequence. If a hand pixel region is determined to exist in the real-time acquired image, this image can be used as the image to be processed. The white balance processing module can then be invoked to perform the following steps:

[0188] White spot detection steps: First, mask the palm pixel region in the image to be processed, obtaining the masked image. Then, perform white spot detection on the masked image to obtain reference white spots in the non-palm pixel regions.

[0189] The light source color temperature estimation steps are as follows: The color value of the reference white point is converted from RGB space to CIE XYZ color space, and then to CIE xy color space to obtain the chromaticity coordinates corresponding to the reference white point's color value. These chromaticity coordinates indicate the position of the reference white point in the CIE xy chromaticity diagram. On the Planck curve, the reference point closest to the reference white point's chromaticity coordinates is determined. Based on the color temperature corresponding to the reference point, the estimated light source color temperature of the image to be processed is determined. The estimated light source color temperature of the image to be processed and the historical light source color temperature of the reference image are subjected to median filtering to obtain the reference light source color temperature of the image to be processed. Both the reference image and the image to be processed are acquired by the image acquisition device in the mobile payment device. The reference image includes at least one of the first K frames of the image to be processed, where K is a positive integer.

[0190] Gain calculation steps: Determine the white balance gain of the image to be processed based on the color temperature of the reference light source.

[0191] White balance processing steps: Based on the white balance gain for the image to be processed, perform white balance processing on at least the palm pixel area in the image to be processed to obtain the processed palm pixel area.

[0192] After white balance processing of the palm area, the payment processing module can be called to perform identity recognition based on the processed palm pixel area, obtain the identity recognition result including user information, and perform payment processing based on the identity recognition result, such as finding the corresponding payment account based on the user information and performing payment processing based on the payment account.

[0193] It is worth mentioning that after performing the above white balance processing steps, the white balance processing module can also perform the following steps: Based on the processed palm pixel region, determine whether the algorithm involved in the white balance processing module has converged. If it has not converged, at least one of the algorithms and parameters involved in the aforementioned steps can be adjusted, and the above steps can be repeated until the algorithm involved in the white balance processing module converges, obtaining the final processed palm pixel region. The method for determining whether the algorithm involved in the white balance processing module has converged can be: calculate the average color temperature of the processed palm pixel region and compare it with a specified color temperature (e.g., 6500K). If the difference between the average color temperature and the specified color temperature is less than a preset threshold, it indicates that the algorithm involved in the white balance processing module has converged.

[0194] By employing the above process, the historical light source color temperature corresponding to the image acquired without a hand (i.e., the reference image) is used as a reference. The historical light source color temperature corresponding to the reference image can accurately reflect the color temperature of the light source in the environment. The image acquired with a hand is used as the image to be processed, and the color temperature is estimated based on the non-hand pixel area in the image to be processed to determine the estimated light source color temperature corresponding to the image to be processed. This avoids the situation where the actual color of the hand itself presented in the hand pixel area is very similar to the color of the light source in the environment from which the image to be processed originates, resulting in inaccurate color temperature estimation. Therefore, the estimated light source color temperature corresponding to the image to be processed can also truly reflect the color temperature of the light source in the environment from which the image to be processed originates.

[0195] By combining the historical light source color temperature of the reference image with the estimated color temperature of the image captured with a hand present, the white balance gain for the hand in the image to be processed is determined. This allows the use of the historical light source color temperature of the reference image to assist in determining the white balance gain for the hand in the image to be processed, thereby improving the accuracy of the white balance gain determination. Finally, based on the white balance gain for the image to be processed, white balance processing is performed on the hand pixel area, which restores the color of the processed hand pixel area to a color close to its true skin tone, eliminating the color cast problem caused by the ambient color temperature.

[0196] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0197] Please refer to it again. Figure 1Another embodiment of this application provides a white balance processing device 155, including a region determination module 1551, a color temperature estimation module 1552, a gain determination module 1553, and a white balance processing module 1554. The region determination module 1551 is used to perform target detection on the image to be processed, determining the region of interest in the image to be processed; the color temperature estimation module 1552 is used to estimate the color temperature of the non-interested pixel regions in the image to be processed based on the non-interested pixel regions, excluding the region of interest, to determine the estimated light source color temperature corresponding to the image to be processed; the gain determination module 1553 is used to determine the white balance gain for the image to be processed based on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image; the reference image and the image to be processed are acquired by the same image acquisition device; the reference image includes at least one of the first K frames of the image to be processed, where K is a positive integer; the white balance processing module 1554 is used to perform white balance processing on at least the region of interest in the image to be processed based on the white balance gain for the image to be processed.

[0198] In one possible implementation, the gain determination module 553 includes a color temperature acquisition submodule and a gain determination submodule; the color temperature acquisition submodule is used to perform median filtering on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image to obtain the reference light source color temperature of the image to be processed; the gain determination submodule is used to determine the white balance gain of the image to be processed based on the reference light source color temperature of the image to be processed.

[0199] In one possible implementation, the pixel region of interest is a pixel region reflecting biometric features; the white balance processing device 155 further includes an identity recognition module, which is further configured to perform white balance processing on at least the pixel region of interest in the image to be processed based on the white balance gain for the image to be processed, to obtain the processed pixel region of interest; the identity recognition module is configured to perform identity recognition based on the processed pixel region of interest, to obtain an identity recognition result.

[0200] In one possible implementation, the method is applied to a payment device, with the pixel area of ​​interest being either a palm print pixel area or a face area; the white balance processing device 155 also includes a payment processing module for performing payment processing based on the identity recognition result.

[0201] In one possible implementation, the color temperature estimation module 1552 includes a white point detection submodule and a color temperature estimation submodule; the white point detection submodule is used to perform white point detection processing on non-interest pixel areas to obtain reference white points in the non-interest pixel areas; the color temperature estimation submodule is used to determine the estimated light source color temperature corresponding to the image to be processed based on the color value of the reference white point.

[0202] In one possible implementation, the color temperature estimation submodule is further configured to convert the color value of the reference white point to the target color space to obtain the chromaticity coordinates corresponding to the color value of the reference white point; determine the reference point closest to the chromaticity coordinates of the reference white point on the Planck curve; and determine the estimated light source color temperature corresponding to the image to be processed based on the color temperature corresponding to the reference point.

[0203] In one possible implementation, the region determination module 1551 includes a detection submodule and a region determination submodule. The detection submodule is used to perform target detection on the image to be processed using a target detection model to obtain a detection result. The detection result includes the semantic object identified from the image to be processed and the position information of the semantic object in the image to be processed. The region determination submodule is used to determine the pixel region in the image to be processed as the region of interest based on the position information of the target semantic object in the image to be processed when it is determined from the detection result that the image to be processed includes a target semantic object.

[0204] In one possible implementation, the white balance processing device 155 further includes: a sample acquisition module, a feature extraction module, a detection module, a loss determination module, and a model update module; the sample acquisition module is used to acquire training samples, the training samples including sample images and label information of the sample images, the label information indicating the labeled semantic objects for the sample images and the sample position information of the labeled semantic objects in the sample images; the feature extraction module is used to extract image features from the sample images using the feature extraction network in the object detection model to obtain image features of the sample images; the detection module is used to perform object detection on the sample images using the classification layer in the object detection model to obtain sample semantic objects in the sample images and the position information of the sample semantic objects in the sample images; the loss determination module is used to determine the model loss of the object detection model based on the sample semantic objects and the position information of the sample semantic objects in the sample images, as well as the labeled semantic objects for the sample images and the sample position information of the labeled semantic objects in the sample images; the model update module is used to update the model parameters of the object detection model based on the model loss.

[0205] In one possible implementation, the loss determination module includes: a location detection loss determination submodule, a semantic loss determination submodule, and a model loss determination submodule; the location detection loss determination submodule is used to determine the location detection loss based on the sample location information of the labeled semantic object in the sample image and the location information of the sample semantic object in the sample image; the semantic loss determination submodule is used to determine the semantic detection loss based on the labeled semantic object and the sample semantic object labeled for the sample image; the model loss determination submodule is used to perform weighted processing based on the location detection loss and the semantic detection loss to obtain the model loss of the target detection model.

[0206] In one possible implementation, the white balance processing device 155 further includes: an image segmentation module and an image encoding module; the image segmentation module is used to segment the sample image into blocks to obtain an image block sequence; the image encoding module is used to encode each image block in the image block sequence to obtain an image block encoding sequence; the feature extraction module is further used to extract features from the image block encoding sequence using the feature extraction network in the target detection model to obtain image features of the sample image.

[0207] In one possible implementation, the sample image includes at least one of an RGB image and an infrared image.

[0208] Each module in the aforementioned image white balance processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles of the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0209] This application provides a computer program product or computer program that includes executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, causing the electronic device to perform the method described in this application.

[0210] This application provides a computer-readable storage medium storing computer-readable instructions, wherein the stored instructions are executable and, when executed by a processor, will cause the processor to perform the aforementioned method steps provided in this application.

[0211] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0212] In some embodiments, computer-readable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0213] As an example, computer-readable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0214] As an example, computer-readable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image white balance processing method, characterized in that, include: Target detection is performed on the image to be processed to determine the region of interest in the image; The color temperature of the light source corresponding to the image to be processed is determined by estimating the color temperature of the non-interested pixel regions in the image to be processed, excluding the interest pixel regions. Based on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image, the white balance gain for the image to be processed is determined; the reference image and the image to be processed are acquired by the same image acquisition device. The reference image includes at least one of the first K frames of the image to be processed, where K is a positive integer; Based on the white balance gain for the image to be processed, at least the pixel regions of interest in the image to be processed are subjected to white balance processing.

2. The method according to claim 1, characterized in that, The step of determining the white balance gain for the image to be processed based on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image includes: The estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image are subjected to median filtering to obtain the reference light source color temperature of the image to be processed. The white balance gain of the image to be processed is determined based on the color temperature of the reference light source of the image to be processed.

3. The method according to claim 1, characterized in that, The pixel region of interest is a pixel region that reflects biological characteristics; The step of performing white balance processing on at least the region of interest in the image to be processed based on the white balance gain for the image to be processed includes: Based on the white balance gain for the image to be processed, at least the pixel region of interest in the image to be processed is subjected to white balance processing to obtain the processed pixel region of interest. After performing white balance processing on at least the region of interest in the image to be processed based on the white balance gain for the image to be processed, the method further includes: Based on the processed region of interest, identity recognition is performed to obtain the identity recognition result.

4. The method according to claim 3, characterized in that, The method is applied to a payment device, and the focus pixel area is either a palm print pixel area or a face area. The After performing identity recognition based on the processed pixel region of interest and obtaining the identity recognition result, the method further includes: Payment is processed based on the identity verification result.

5. The method according to claim 1, characterized in that, The step of estimating the color temperature of the light source corresponding to the image to be processed based on the non-interested pixel regions in the image to be processed, excluding the pixel regions of interest, includes: White point detection processing is performed on the non-interested pixel region to obtain a reference white point in the non-interested pixel region; Based on the color value of the reference white point, the estimated light source color temperature corresponding to the image to be processed is determined.

6. The method according to claim 5, characterized in that, The step of determining the estimated light source color temperature corresponding to the image to be processed based on the color value of the reference white point includes: The color value of the reference white point is converted to the target color space to obtain the chromaticity coordinates corresponding to the color value of the reference white point. On the Planck curve, determine the reference point that is closest to the chromaticity coordinates of the reference white point; Based on the color temperature corresponding to the reference point, the estimated light source color temperature corresponding to the image to be processed is determined.

7. The method according to claim 1, characterized in that, The step of performing target detection on the image to be processed, and determining the region of interest in the image to be processed, includes: An object detection model is used to detect objects in the image to be processed, and the detection results are obtained. The detection results include semantic objects identified from the image to be processed and the location information of the semantic objects in the image to be processed. If the detection result determines that the image to be processed contains a target semantic object, the pixel region in which the target semantic object is located in the image to be processed is determined as the region of interest based on the location information of the target semantic object in the image to be processed.

8. The method according to claim 7, characterized in that, The target detection model is trained in the following way: Obtain training samples, which include sample images and label information of the sample images. The label information indicates the labeled semantic objects that are labeled for the sample images and the sample position information of the labeled semantic objects in the sample images. The feature extraction network in the target detection model is used to extract image features from the sample image to obtain the image features of the sample image; The classification layer in the target detection model is used to perform target detection on the sample image to obtain the sample semantic object and the position information of the sample semantic object in the sample image. Based on the sample semantic object and its position information in the sample image, as well as the labeled semantic object and its sample position information in the sample image, the model loss of the target detection model is determined. The model parameters of the target detection model are updated based on the model loss.

9. The method according to claim 8, characterized in that, The step of determining the model loss of the target detection model based on the sample semantic object and its position information in the sample image, as well as the labeled semantic object and its sample position information in the sample image, includes: Based on the sample location information of the labeled semantic object in the sample image and the location information of the sample semantic object in the sample image, the location detection loss is determined; Based on the labeled semantic objects annotated for the sample images and the sample semantic objects, the semantic detection loss is determined; The model loss of the target detection model is obtained by weighting the location detection loss and the semantic detection loss.

10. The method according to claim 8, characterized in that, Before extracting image features from the sample image using the feature extraction network in the target detection model to obtain the image features of the sample image, the method further includes: The sample image is divided into blocks to obtain an image block sequence; Image encoding is performed on each image block in the image block sequence to obtain an image block encoding sequence; The step of extracting image features from the sample image using the feature extraction network in the target detection model to obtain the image features of the sample image includes: The feature extraction network in the target detection model is used to extract features from the image block encoding sequence to obtain the image features of the sample image.

11. The method according to claim 8, characterized in that, The sample images include at least one of RGB images and infrared images.

12. A white balance processing device, characterized in that, include: The region determination module is used to perform target detection on the image to be processed and determine the region of interest pixels in the image to be processed. The color temperature estimation module is used to estimate the color temperature of the non-interested pixel regions in the image to be processed, excluding the interest pixel regions, and to determine the estimated light source color temperature of the image to be processed. The gain determination module is used to determine the white balance gain for the image to be processed based on the estimated light source color temperature corresponding to the image to be processed and the historical light source color temperature corresponding to the reference image; the reference image and the image to be processed are acquired by the same image acquisition device; the reference image includes at least one of the first K frames of the image to be processed, where K is a positive integer; The white balance processing module is used to perform white balance processing on at least the region of interest in the image to be processed, based on the white balance gain for the image to be processed.

13. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-11.

15. A computer program product comprising computer-readable instructions, characterized in that, When executed by a processor, the computer-readable instructions implement the steps of the method according to any one of claims 1-11.

Images