Image processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: performing light sensation parameter prediction processing on an input image to obtain a light sensation parameter of the input image; determining an expected color temperature corresponding to the light sensation parameter from a corresponding relationship between the candidate light sensation parameter and the candidate expected color temperature; determining the current color temperature of the input image, and determining the color temperature error rate of the input image based on the expected color temperature and the current color temperature; updating the light sensation parameter based on the color temperature error rate to obtain an updated light sensation parameter; and adjusting the input image through the updated light sensing parameter to obtain an adjusted input image. According to the invention, the accuracy of the image color temperature can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive field in computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.

[0003] With the rapid development of artificial intelligence technology, artificial intelligence plays an increasingly important role in image processing. The field of image processing includes the precise capture, calibration, and reproduction of image colors to ensure the accurate transmission of color information and the authenticity of visual effects. In current image processing, it is impossible to accurately estimate the color temperature of the image light source and perform color correction on the image. In high or low color temperature environments, especially under extreme color temperature conditions, challenges such as color cast problems are faced, which reduces the authenticity of image color temperature.

[0004] The related art lacks an effective solution for accurately perceiving color temperature and improving the accuracy of image color temperature in different color temperature scenes. Summary of the Invention

[0005] Embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of image color temperature.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present invention provides an image processing method, which includes:

[0008] Performing light sensitivity parameter prediction processing on the input image to obtain the light sensitivity parameters of the input image;

[0009] Determining the desired color temperature corresponding to the candidate light sensitivity parameters from the corresponding relationship between the candidate light sensitivity parameters and the candidate desired color temperatures;

[0010] determining a current color temperature of the input image, and determining a color temperature error rate of the input image based on the expected color temperature and the current color temperature;

[0011] updating the light perception parameter based on the color temperature error rate to obtain an updated light perception parameter;

[0012] The input image is adjusted using the updated light sensitivity parameters to obtain an adjusted input image.

[0013] An embodiment of the present application provides an image processing device, comprising:

[0014] A light sensitivity prediction module, configured to perform light sensitivity parameter prediction processing on an input image to obtain light sensitivity parameters of the input image;

[0015] a color temperature error rate acquisition module, configured to determine, from a correspondence between candidate light sensitivity parameters and candidate expected color temperatures, an expected color temperature corresponding to the light sensitivity parameters; determine a current color temperature of the input image, and determine a color temperature error rate of the input image based on the expected color temperature and the current color temperature;

[0016] An image updating module is configured to update the light sensitivity parameters based on the color temperature error rate to obtain updated light sensitivity parameters; and adjust the input image using the updated light sensitivity parameters to obtain an adjusted input image.

[0017] In the above technical solution, the light perception prediction module is also used to map the input image to obtain the probability distribution of the input image in the latent space; sample latent space samples from the latent space, and remap the latent space samples to obtain the prediction features corresponding to the input image, wherein the latent space samples conform to the probability distribution of the input image in the latent space; and determine the light perception parameters of the input image based on the prediction features corresponding to the input image.

[0018] In the above technical solution, the light perception prediction module is further configured to perform the following processing on each color channel in the color space corresponding to the prediction feature:

[0019] Determine a first probability distribution of the prediction feature in each color channel, where the first probability distribution is used to characterize the proportion of pixels with different pixel values ​​in each color channel in the pixels of the prediction feature; determine the pixel value with the maximum probability in the first probability distribution corresponding to each color channel as the target pixel value; and determine the inverse of the target pixel value as the light sensitivity parameter of the input image.

[0020] In the above technical solution, the light perception prediction module is also used to map the prediction features to obtain a second probability distribution of the input image, wherein the second probability distribution is used to characterize the weight of the light perception parameters of the input image corresponding to each color channel; from the second probability distribution, the target weight of the light perception parameters of the target color channel is determined; the ratio of the preset parameter to the target weight is determined as the light perception parameter coefficient; and the following processing is performed for each color channel in the color space corresponding to the input image: the product of the light perception parameter coefficient and the weight of the light perception parameter corresponding to each color channel is determined as the light perception parameter of the input image.

[0021] In the above technical solution, the color temperature error rate acquisition module is further used to convert the input image from the current color space to the target color space to obtain a converted input image; perform color histogram statistics on the converted input image to obtain a color histogram corresponding to the input image; and determine the current color temperature corresponding to the peak of the color histogram from the correspondence between the candidate peaks and the candidate color temperatures.

[0022] In the above technical solution, the color temperature error rate acquisition module is further used to obtain the difference between the expected color temperature and the current color temperature; and determine the color temperature error rate of the input image based on the difference.

[0023] In the above technical solution, the color temperature error rate acquisition module is further used to determine the color temperature error rate of the input image by one of the following methods: determining the ratio of the difference value to the current color temperature as the color temperature error rate of the input image; determining the ratio of the square of the difference value to the square of the current color temperature as the color temperature error rate of the input image.

[0024] In the above technical solution, the image update module is also used to determine the update learning rate of the input image as a first learning rate when the color temperature error rate is greater than or equal to the color temperature threshold; when the color temperature error rate is less than the color temperature threshold, determine the update learning rate of the input image as a second learning rate; determine the current light perception parameter corresponding to the current color temperature from the correspondence between the candidate color temperature and the candidate light perception parameter; based on the update learning rate of the input image and the current light perception parameter, update the light perception parameter to obtain the updated light perception parameter.

[0025] In the above technical solution, the image update module is also used to determine the ratio of the difference between the light sensitivity parameter and the current light sensitivity parameter to the preset parameter as the update step size; and the sum of the product of the update learning rate and the update step size and the current light sensitivity parameter is determined as the updated light sensitivity parameter.

[0026] In the above technical solution, the image update module is also used to obtain the adjustment factor corresponding to the color temperature error rate from the correspondence between the candidate color temperature error rate and the candidate adjustment factor, wherein the adjustment factor is proportional to the color temperature error rate; and the product of the adjustment factor and the light perception parameter is determined as the updated light perception parameter.

[0027] In the above technical solution, the image update module is further used to perform the following processing on each pixel of the input image: determining the product of the pixel and the light sensitivity parameter as the adjusted pixel; and combining each adjusted pixel into the input image.

[0028] In the above technical solution, the image update module is also used to obtain at least one neighbor image of the input image and determine the updated light sensitivity parameters of each neighbor image; perform weighted averaging on the updated light sensitivity parameters of the input image and the updated light sensitivity parameters of each neighbor image to obtain target light sensitivity parameters; and adjust the input image according to the target light sensitivity parameters to obtain an adjusted input image.

[0029] An embodiment of the present application provides an electronic device, including:

[0030] a memory for storing computer-executable instructions;

[0031] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0032] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the image processing method provided in the embodiment of the present application when executed by a processor.

[0033] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the image processing method provided in the embodiment of the present application is implemented.

[0034] The embodiments of the present application have the following beneficial effects:

[0035] The input image is subjected to light sensitivity parameter prediction processing to obtain the light sensitivity parameters of the input image, the expected color temperature corresponding to the light sensitivity parameters is determined from the correspondence between the candidate light sensitivity parameters and the candidate expected color temperature, and the color temperature error rate of the input image is determined based on the expected color temperature and the current color temperature, the light sensitivity parameters are updated based on the color temperature error rate to obtain updated light sensitivity parameters, and the input image is adjusted using the updated light sensitivity parameters to obtain an adjusted input image. In this way, by updating the light sensitivity parameters, the color temperature of input images in some extreme color temperature environments can be adjusted to improve the accuracy of the image color temperature. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the structure of the image processing system architecture provided by an embodiment of the present application;

[0037] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0038] Figure 3A This is a schematic diagram of the first flow chart of the image processing method provided in an embodiment of the present application;

[0039] Figure 3B 2 is a schematic diagram of a second flow chart of the image processing method provided in an embodiment of the present application;

[0040] Figure 3C 3 is a schematic diagram of a third flow chart of the image processing method provided in an embodiment of the present application;

[0041] Figure 3D 4 is a schematic diagram of a fourth flow chart of the image processing method provided in an embodiment of the present application;

[0042] Figure 3E 5 is a schematic diagram of a fifth flow chart of the image processing method provided in an embodiment of the present application;

[0043] Figure 3F is a sixth flow chart of the image processing method provided in an embodiment of the present application;

[0044] Figure 3G 7 is a schematic diagram of a seventh flow chart of the image processing method provided in an embodiment of the present application;

[0045] Figure 3H This is a schematic diagram of the eighth flow chart of the image processing method provided in an embodiment of the present application;

[0046] Figure 3I This is a ninth flow chart of the image processing method provided in an embodiment of the present application;

[0047] Figure 3J 10 is a schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0048] Figure 3K This is an eleventh flow chart of the image processing method provided in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the palm image correction process provided by an embodiment of the present application;

[0050] Figure 5 This is a schematic diagram of a camera collector provided in an embodiment of the present application;

[0051] Figure 6This is a schematic diagram of the palm prints to be collected provided in an embodiment of the present application;

[0052] Figure 7 This is a diagram of the data enhancement principle provided by an embodiment of the present application;

[0053] Figure 8 This is a block schematic diagram provided by an embodiment of the present application;

[0054] Figure 9 This is a schematic diagram of the model construction provided in the embodiment of the present application;

[0055] Figure 10 This is a parameter update principle diagram provided by an embodiment of the present application;

[0056] Figure 11 This is a schematic diagram of the ambient color temperature perception estimation provided by an embodiment of the present application.

[0057] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0061] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0062] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0063] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0064] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0065] 1) Light sensitivity parameters are used to adjust the color temperature of the image. By adjusting the color temperature of the image, the color differences caused by the ambient light temperature are balanced, so that white objects can be correctly presented as white regardless of the lighting conditions.

[0066] 2) Color temperature, which is used to describe the color of the light source. That is, when a black body is heated to different temperatures, it will emit light of different colors. The unit of color temperature is Kelvin (K).

[0067] 3) Color space, used to describe and represent the color of an image. Different color spaces have different color ranges and application areas.

[0068] 4) Mapping: Mapping the input data to a latent space, which usually has a lower dimension and can represent some implicit pattern of the original data.

[0069] 5) Remapping: Mapping the latent vector back to the original data space to generate an output similar to the input data. The latent vector is a sample obtained by sampling in the latent space.

[0070] In the related art, color temperature adjustment is performed based on a camera module (ISP, Image Signal Processor), which cannot accurately perceive the ambient color temperature and uses constant light sensing parameters to cover all color temperature scenes, resulting in color cast problems at some high or low color temperatures.

[0071] Based on the above analysis, the applicant found that the related art method of color correction of images through constant light parameters cannot improve the accuracy of image color temperature. To address the above problem, the embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of image color temperature.

[0072] The image processing method described in the embodiments of the present application can be applied to various fields, such as palm color temperature correction, facial color temperature correction, and professional photography. That is, the image processing method in the embodiments of the present application is not limited to a certain field.

[0073] The following describes an exemplary application of the electronic device provided in the embodiment of the present application. The device provided in the embodiment of the present application can be implemented as a terminal or a server. The following describes an exemplary application when the device is implemented as a server.

[0074] See also Figure 1 , Figure 1 This is a schematic diagram of the image processing system architecture provided in an embodiment of the present application. To support an image processing application, a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0075] Terminal 400 is used to send input data to server 200 through network 300. Server 200 is used to perform light sensitivity parameter prediction processing on the input image to obtain the light sensitivity parameters of the input image, determine the expected color temperature corresponding to the light sensitivity parameters from the correspondence between the candidate light sensitivity parameters and the candidate expected color temperature, determine the current color temperature of the input image, and determine the color temperature error rate of the input image based on the expected color temperature and the current color temperature, update the light sensitivity parameters based on the color temperature error rate, obtain updated light sensitivity parameters, adjust the input image according to the updated light sensitivity parameters, and return the adjusted input image to terminal 400. Terminal 400 displays the adjusted input image through a graphical interface 410.

[0076] Next, an example of image processing performed by terminal 400 will be described.

[0077] In some embodiments, the terminal 400 can independently complete image processing tasks. For example, the terminal 400 is used to perform light sensitivity parameter prediction processing on the input image to obtain the light sensitivity parameters of the input image, determine the expected color temperature corresponding to the light sensitivity parameters from the correspondence between the candidate light sensitivity parameters and the candidate expected color temperature, determine the current color temperature of the input image, and determine the color temperature error rate of the input image based on the expected color temperature and the current color temperature, update the light sensitivity parameters based on the color temperature error rate, obtain updated light sensitivity parameters, adjust the input image according to the updated light sensitivity parameters, and display the adjusted input image through the graphical interface 410.

[0078] In one implementation scenario, the server or terminal can perform light perception parameter prediction processing on the facial image to obtain the light perception parameters of the facial image, determine the expected color temperature corresponding to the light perception parameters from the correspondence between the candidate light perception parameters and the candidate expected color temperatures, determine the current color temperature of the facial image, and determine the color temperature error rate of the facial image based on the expected color temperature and the current color temperature, update the light perception parameters based on the color temperature error rate to obtain updated light perception parameters, and adjust the facial image using the updated light perception parameters to obtain an adjusted facial image.

[0079] In one implementation scenario, the server or terminal can perform light sensitivity parameter prediction processing on the palm image to obtain the light sensitivity parameters of the palm image, determine the expected color temperature corresponding to the light sensitivity parameters from the correspondence between the candidate light sensitivity parameters and the candidate expected color temperatures, determine the current color temperature of the palm image, and determine the color temperature error rate of the palm image based on the expected color temperature and the current color temperature, update the light sensitivity parameters based on the color temperature error rate to obtain updated light sensitivity parameters, and adjust the palm image using the updated light sensitivity parameters to obtain an adjusted palm image.

[0080] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0081] The terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, intelligent voice interaction device, smart home appliance, vehicle-mounted terminal, aircraft, etc., but is not limited thereto. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0082] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, as well as the promotion of search services, social networks, mobile commerce and open collaboration, each item may have its own hash code identification mark in the future, and all of them need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0083] The embodiments of the present application can be implemented through artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0084] See also Figure 2 , Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1 In the terminal 400 or server 200, the electronic device 500 includes: at least one processor 510, a memory 550, and at least one network interface 520. The various components in the server 200 are coupled together through a bus system 540. It can be understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 540 .

[0085] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0086] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0087] In some embodiments, when the terminal 400 independently completes the image processing task, the server 200 provided in the embodiment of the present application does not include the user interface 530 .

[0088] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard drive, an optical drive, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0089] The memory 550 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0090] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0091] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0092] A network communication module 552 for reaching other computing devices via one or more (wired or wireless) network interfaces 520 , exemplary network interfaces 520 including Bluetooth, WiFi, and USB;

[0093] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0094] In some embodiments, when the image processing task is completed independently by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0095] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions; in some embodiments, when the embodiment independently completes the image processing task by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0096] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 An image processing device 555 stored in memory 550 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a light perception prediction module 5551, a color temperature error rate acquisition module 5552, and an image update module 5553. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0097] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0098] It should be noted that in the following image processing examples, those skilled in the art may apply the image processing method provided in the embodiments of the present application to image processing based on their understanding of the following.

[0099] See also Figure 3A , Figure 3A This is a first flow chart of the image processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are used for illustration. The image processing method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will illustrate the collaborative implementation by a server and a terminal as an example.

[0100] In step 101, light sensitivity parameter prediction processing is performed on the input image to obtain the light sensitivity parameters of the input image.

[0101] In some embodiments, the light sensitivity parameter is used to adjust the color balance of the image so that white objects appear white under different lighting conditions and objects of other colors appear in their true colors. The light sensitivity parameter is used to characterize the gain of each channel of the input image.

[0102] For example, the light sensitivity parameter is expressed as a color gain of [1.2, 1, 1.5] in the red, green, and blue (RGB) color space, where 1.2 is the color gain of the input image in the red channel, 1 is the color gain of the input image in the green channel, and 1.5 is the color gain of the input image in the blue channel. The input image (for example, [R, G, B]) can be color corrected based on the light sensitivity parameter to obtain the adjusted input image [1.2*R, 1*G, 1.5*B].

[0103] In some embodiments, see Figure 3B , Figure 3B is a second flow chart of the image processing method provided in an embodiment of the present application, Figure 3A Step 101 shown can be performed by Figure 3B Steps 1011 to 1013 are implemented as described below.

[0104] In step 1011, the input image is mapped to obtain the probability distribution of the input image in the latent space.

[0105] In some embodiments, the probability distribution of the input image in the latent space may be a Gaussian distribution.

[0106] In some embodiments, the input image is encoded to obtain encoding features, and the encoding features are mapped based on the fully connected layer of the encoder to obtain the mean and variance of the input image in the latent space, thereby obtaining the Gaussian distribution of the input image in the latent space, that is, the probability distribution of the input image in the latent space.

[0107] In step 1012, latent space samples are sampled from the latent space, and the latent space samples are remapped to obtain prediction features corresponding to the input image.

[0108] Among them, the latent space samples conform to the probability distribution of the input image in the latent space.

[0109] In some embodiments, a noise vector is randomly sampled from a standard normal distribution, and then transformed using the mean and variance of the input image in the latent space to obtain a latent space sample, and the latent space sample is remapped based on the decoder to obtain a predicted feature corresponding to the input image, wherein the predicted feature has the same dimension as the input image.

[0110] In step 1013 , light sensitivity parameters of the input image are determined based on the predicted features corresponding to the input image.

[0111] In some embodiments, the light sensitivity parameter of the predicted feature is determined as the light sensitivity parameter of the input image.

[0112] In some embodiments, the following processing is performed for each color channel in the color space corresponding to the predicted feature, see Figure 3C , Figure 3C is a third flow chart of the image processing method provided in an embodiment of the present application, Figure 3B Step 1013 shown may be performed by Figure 3C Steps 10131A to 10133A are implemented as described below.

[0113] In step 10131A, a first probability distribution of the prediction feature in each color channel is determined, where the first probability distribution is used to characterize the proportion of pixels with different pixel values ​​in each color channel in the pixels of the prediction feature.

[0114] In some embodiments, the proportion of pixels with different pixel values ​​in the pixels of the predicted feature is determined in each color channel to obtain a first probability distribution corresponding to each color channel.

[0115] For example, taking the red (R) color channel of the input image as an example, the value of the red channel of the input image is [[0, 1], [1, 2]], and the number of pixels with different pixel values ​​is counted. The number of pixel values ​​with a value of 0 is 1, the number of pixel values ​​with a value of 1 is 2, and the number of pixel values ​​with a value of 2 is 1. The proportion of pixels with different pixel values ​​in the pixels of the predicted features is [0.25, 0.5, 0.25].

[0116] In step 10132A, the pixel value with the maximum probability in the first probability distribution corresponding to each color channel is determined as the target pixel value.

[0117] For example, taking the red color channel of the input image as an example, the first probability distribution of the red color channel is obtained as [0.25, 0.5, 0.25], the pixel value with the maximum probability is obtained as 1, and the value 1 is determined as the target pixel value.

[0118] In step 10133A, the inverse of the target pixel value is determined as the light sensitivity parameter of the input image.

[0119] For example, taking the red color channel of the input image as an example, the target pixel value of the red color channel is 0.8, and the reciprocal of the target pixel value 1.25 is determined as the light sensitivity parameter of the red color channel of the input image.

[0120] In some embodiments, see Figure 3D , Figure 3D is a fourth flow chart of the image processing method provided in an embodiment of the present application, Figure 3B Step 1013 shown may be performed by Figure 3D Steps 10131B to 10134B are implemented as described below.

[0121] In step 10131B, the predicted features are mapped to obtain a second probability distribution of the input image.

[0122] The second probability distribution is used to represent the weight of the light sensitivity parameter of each color channel of the input image.

[0123] In some embodiments, the second probability distribution of the input image is a proportion of the light sensitivity parameter of each color channel.

[0124] For example, the predicted features are mapped, the predicted features are encoded to obtain encoded predicted features, and the encoded predicted features are mapped based on the fully connected layer to obtain the weights of the light sensitivity parameters as [0.3, 0.2, 0.5].

[0125] In step 10132B, a target weight of the light perception parameter of the target color channel is determined from the second probability distribution.

[0126] In some embodiments, a preset target color channel (eg, a red color channel) is selected from color channels corresponding to the input image (eg, RGB channels of the input image in an RGB color space), where the target color channel is one of the color channels corresponding to the input image.

[0127] In step 10133B, the ratio of the preset parameter to the target weight is determined as the light sensitivity parameter coefficient.

[0128] In some embodiments, the ratio of the value 1 to the target weight is determined as the light sensitivity parameter coefficient.

[0129] In some embodiments, the following processing is performed for each color channel in the color space corresponding to the input image, see step 10134B, and the product of the light sensitivity parameter coefficient and the weight of the light sensitivity parameter corresponding to each color channel is determined as the light sensitivity parameter of the input image.

[0130] For example, the weight of the light sensitivity parameter corresponding to each color channel is [0.3, 0.2, 0.5]. When the target color channel is the blue color channel, the light sensitivity parameter coefficient is 2. The light sensitivity parameter coefficient is multiplied by the weight of the light sensitivity parameter corresponding to each color channel to obtain the light sensitivity parameter of the input image [0.6, 0.4, 1].

[0131] Through the embodiments of the present application, the latent space samples in the latent space are remapped, and the light perception parameters of the input image are determined based on the predicted features obtained by the remapping, so as to facilitate color temperature adjustment of the input image.

[0132] Continue to refer Figure 3A In step 102, the expected color temperature corresponding to the light perception parameter is determined from the corresponding relationship between the candidate light perception parameters and the candidate expected color temperature.

[0133] In some embodiments, the correspondence between the candidate light perception parameters and the candidate desired color temperatures may be linear or nonlinear, depending on the characteristics of the lighting device and the application scenario.

[0134] In step 103 , the current color temperature of the input image is determined, and based on the expected color temperature and the current color temperature, the color temperature error rate of the input image is determined.

[0135] In some embodiments, the color temperature error rate can be the absolute difference or relative difference between the current color temperature and the expected color temperature. The absolute difference is the absolute value of the difference between the current color temperature and the expected color temperature, and the relative difference is the percentage of the absolute error between the current color temperature and the expected color temperature or the percentage of the mean square error between the current color temperature and the expected color temperature. The embodiment of the present application does not limit the method for obtaining the color temperature error rate.

[0136] In some embodiments, see Figure 3E , Figure 3E This is a fifth flow chart of the image processing method provided in the embodiment of the present application. Figure 3A The current color temperature of the input image in step 103 can be determined by Figure 3E Steps 1031A to 1033A are implemented.

[0137] In step 1031A, the input image is converted from the current color space to the target color space to obtain a converted input image.

[0138] In some embodiments, the target color space determines different color spaces according to different application scenarios.

[0139] For example, when the application scenario focuses more on the color of the input image, the input image is converted from the current color space to the hue-saturation-value (HSV) color space. When the application scenario focuses more on the brightness of the input image, the input image is converted from the current color space to the luminance-chromaticity (Lab) color space.

[0140] In step 1032A, color histogram statistics are performed on the converted input image to obtain a color histogram corresponding to the input image.

[0141] In some embodiments, a color histogram is used to characterize the frequency of occurrence of each color or color range in the input image.

[0142] For example, taking the input image after converting it from the current color space to the HSV color space as an example, the color value H in the color space is quantized, that is, the continuous color value is divided into discrete intervals or "buckets", where each bucket represents a color range. When counting, the color value of each pixel in the image is assigned to the corresponding bucket, and a color histogram is constructed based on the number of pixels in each bucket.

[0143] Continuing with the above example, the value of the hue channel of the converted input image is [[0, 1], [1, 2]], and the number of pixels with different pixel values ​​is counted. The number of pixel values ​​with a value of 0 is 1, the number of pixel values ​​with a value of 1 is 2, and the number of pixel values ​​with a value of 2 is 1. The proportion of pixels with different pixel values ​​in the pixels of the converted input image is [0.25, 0.5, 0.25], which is also the color histogram [0.25, 0.5, 0.25].

[0144] In step 1033A, the current color temperature corresponding to the peak value of the color histogram is determined from the correspondence between the candidate peak values ​​and the candidate color temperatures.

[0145] In some embodiments, the peak of the color histogram is the hue with the largest pixel proportion in the converted input image.

[0146] For example, taking the input image after converting it from the current color space to the HSV color space as an example, the correspondence between the hue and the color temperature of the color histogram of the converted input image is obtained. For example, when the hue is 0, the corresponding color temperature is 3000K, and when the hue is 30, the corresponding color temperature is 3500K. It is determined that the peak value of the color histogram is 30, that is, the hue has a value of 30. From the correspondence between hue and color temperature, it is obtained that the current color temperature corresponding to the peak value of the color histogram is 3500K.

[0147] In some embodiments, see Figure 3F , Figure 3F This is a sixth flow chart of the image processing method provided in the embodiment of the present application. Figure 3A In step 103, the color temperature error rate of the input image is determined based on the expected color temperature and the current color temperature, which can be obtained by Figure 3F Steps 1031B to 1032B are implemented.

[0148] In step 1031B, the difference between the desired color temperature and the current color temperature is obtained.

[0149] For example, the expected color temperature is 3500K, the current color temperature is 3000K, and the difference between the expected color temperature and the current color temperature is determined to be 500K.

[0150] In step 1032B, a color temperature error rate of the input image is determined based on the difference.

[0151] In some embodiments, the color temperature error rate of the input image is determined by one of the following methods: the ratio of the difference value to the current color temperature is determined as the color temperature error rate of the input image; and the ratio of the square of the difference value to the square of the current color temperature is determined as the color temperature error rate of the input image.

[0152] For example, the current color temperature is 3000K, the difference between the expected color temperature and the current color temperature is determined to be 500K, and the ratio of the difference to the current color temperature, 0.6, is determined as the color temperature error rate of the input image, or the ratio of the square of the difference to the square of the current color temperature, 0.36, is determined as the color temperature error rate of the input image.

[0153] Through the embodiments of the present application, histogram statistics are performed on the image to obtain the current color temperature of the image, which can accurately perceive the ambient color temperature.

[0154] Continue to refer Figure 3A In step 104, the light perception parameter is updated based on the color temperature error rate to obtain an updated light perception parameter.

[0155] In some embodiments, the updated light sensitivity parameters are used to adjust the color temperature of the input image.

[0156] In some embodiments, see Figure 3G , Figure 3G is a seventh flow chart of the image processing method provided in an embodiment of the present application, Figure 3A Step 104 shown may be performed by Figure 3G Steps 1041A to 1044A are implemented as described below.

[0157] In step 1041A, when the color temperature error rate is greater than or equal to the color temperature threshold, the updated learning rate of the input image is determined to be the first learning rate.

[0158] In some embodiments, when the color temperature error rate is greater than or equal to the color temperature threshold, the color temperature of the current input image is significantly different from the expected color temperature. The light sensitivity parameters of the current input image are adjusted through the first learning rate. Using a larger learning rate can increase the amplitude of the light sensitivity parameter update of the current input image, thereby reducing the image color cast problem caused by the lower light sensitivity parameters.

[0159] For example, the color temperature error rate is 0.6 and the color temperature threshold is 0.5. Since the color temperature error rate is greater than the color temperature threshold, the update learning rate of the input image is determined to be a larger learning rate of 0.6.

[0160] In step 1042A, when the color temperature error rate is less than the color temperature threshold, the updated learning rate of the input image is determined to be the second learning rate.

[0161] In some embodiments, the second learning rate is smaller than the first learning rate. When the color temperature error rate is smaller than the color temperature threshold, the color temperature of the current input image is slightly different from the expected color temperature. The light sensitivity parameters of the current input image are adjusted by the second learning rate. Using a smaller learning rate can reduce the amplitude of the light sensitivity parameter update of the current input image.

[0162] For example, the color temperature error rate is 0.3, and the color temperature threshold is 0.5. Since the color temperature error rate is less than the color temperature threshold, the update learning rate of the input image is determined to be a smaller learning rate of 0.2.

[0163] In step 1043A, the current light perception parameter corresponding to the current color temperature is determined from the correspondence between the candidate color temperatures and the candidate light perception parameters.

[0164] For example, the correspondence between the candidate color temperature and the candidate light sensitivity parameter is obtained. For example, when the light sensitivity parameter is 0.7, the corresponding color temperature is 3000K, when the light sensitivity parameter is 0.8, the corresponding color temperature is 3500K, and the current color temperature is 3500K. It is determined that the current light sensitivity parameter corresponding to the current color temperature is 0.8.

[0165] In step 1044A, the light sensitivity parameters are updated based on the updated learning rate of the input image and the current light sensitivity parameters to obtain updated light sensitivity parameters.

[0166] In some embodiments, the update learning rate of the input image is used to control the step size of the light sensitivity parameter update. The larger the update learning rate, the faster the light sensitivity parameter is updated, and the smaller the update learning rate, the slower the light sensitivity parameter is updated.

[0167] In some embodiments, see Figure 3H , Figure 3H This is an eighth flow chart of the image processing method provided in an embodiment of the present application. Figure 3G Step 1044A shown may be performed by Figure 3HSteps 10441A to 10442A are implemented as described below.

[0168] In step 10441A, the ratio of the difference between the light sensitivity parameter and the current light sensitivity parameter to the preset parameter is determined as the update step size.

[0169] In some embodiments, the update step size is used to dynamically adjust the light sensing parameter according to the change in the light sensing parameter and a preset parameter.

[0170] For example, the light sensitivity parameter is 1.2, the current light sensitivity parameter is 0.2, the preset parameter is 100, and the ratio of the difference between the light sensitivity parameter and the current light sensitivity parameter to the preset parameter is determined to be 0.01. 0.01 is determined as the update step size to achieve more precise adjustment of the light sensitivity parameter.

[0171] In some embodiments, a preset parameter is determined as the update step size.

[0172] In step 10442A, the product of the updated learning rate and the updated step size is added to the sum of the current light sensing parameters to determine the updated light sensing parameters.

[0173] For example, the current light sensitivity parameter is 0.2, the update step size is 0.01, and the update learning rate is 0.6. The product of the update learning rate and the update step size and the sum of the current light sensitivity parameter, 0.206, are determined as the updated light sensitivity parameter.

[0174] Through the embodiments of the present application, the light sensing parameters are dynamically adjusted by adaptively updating the learning rate, so that the light sensing parameters converge to the optimal solution more quickly, thereby further improving the effect of light sensing parameter updating.

[0175] In some embodiments, see Figure 3I , Figure 3I This is a ninth flow chart of the image processing method provided in an embodiment of the present application. Figure 3A Step 104 shown may be performed by Figure 3I Steps 1041B to 1042B are implemented as described below.

[0176] In step 1041B, the adjustment factor corresponding to the color temperature error rate is obtained from the correspondence between the candidate color temperature error rates and the candidate adjustment factors.

[0177] Among them, the adjustment factor is proportional to the color temperature error rate.

[0178] In some embodiments, the correspondence between the color temperature error rate and the adjustment factor is obtained. For example, when the color temperature error rate is 0.1, the corresponding adjustment factor is 1.10, and when the color temperature error rate is 0.2, the corresponding adjustment factor is 1.2. When the color temperature error rate increases, the adjustment factor is increased to correct the light perception parameters.

[0179] In step 1042B, the product of the adjustment factor and the light perception parameter is determined as the updated light perception parameter.

[0180] In some embodiments, the adjustment factor is used to reduce the color temperature error rate of the input image obtained by adjusting the input image based on the updated light sensitivity parameter.

[0181] For example, if the current light sensitivity parameter is 0.6 and the adjustment factor is 1.12, the updated light sensitivity parameter is 0.6 multiplied by 1.12, that is, 0.672.

[0182] Through the embodiments of the present application, based on the correspondence between the candidate color temperature error rates and the candidate adjustment factors, the adjustment factor corresponding to the current color temperature error rate is determined to reduce the color temperature error rate of the image after the input image is adjusted based on the adjustment factor.

[0183] Continue to refer Figure 3A In step 105, the input image is adjusted using the updated light sensitivity parameters to obtain an adjusted input image.

[0184] In some embodiments, the following processing is performed for any domain, see Figure 3J , Figure 3J This is a tenth flow chart of the image processing method provided in the embodiment of the present application. Before step 105, execute Figure 3J Step 201 to step 202.

[0185] In step 201 , at least one neighbor image of an input image is acquired, and updated light sensitivity parameters of each neighbor image are determined.

[0186] In some embodiments, the neighbor image may be the previous and next frame images of the input image in a time series, or an image from a different perspective of the same scene as the input image. The embodiment of the present application does not limit the method for obtaining the neighbor image.

[0187] In some embodiments, the updated light sensitivity parameters of each neighbor image are determined according to a method for acquiring the updated light sensitivity parameters of the input image.

[0188] In step 202 , a weighted average is performed on the updated light sensitivity parameter of the input image and the updated light sensitivity parameter of each neighbor image to obtain a target light sensitivity parameter.

[0189] In some embodiments, the updated light sensitivity parameter of the input image is .

[0190] For example, taking the input image in the RGB color space as an example, the updated light sensitivity parameters of the input image are [1.2, 1, 1.5], and the updated light sensitivity parameters of the neighboring image are [1, 0.8, 1.3]. When the updated light sensitivity parameters of the input image and the updated light sensitivity parameters of the neighboring image have the same weight, that is, the weight of the updated light sensitivity parameters of the input image and the weight of the updated light sensitivity parameters of the neighboring image are both 0.5, the target light sensitivity parameters are [1.1, 0.9, 1.4].

[0191] Continuing with the above example, when the updated light sensitivity parameters of the input image and the updated light sensitivity parameters of the neighboring image have different weights, the weight of the updated light sensitivity parameters of the input image is 0.4, and the weight of the updated light sensitivity parameters of the neighboring image is 0.6, the target light sensitivity parameters are [1.08, 0.88, 1.38].

[0192] In some embodiments, the input image is adjusted using the target light sensitivity parameter to obtain an adjusted input image.

[0193] Following the above embodiment, the following processing is performed for each pixel of the input image, the product of the pixel and the target light sensitivity parameter is determined as the adjusted pixel, and each adjusted pixel is combined into the input image.

[0194] In some embodiments, the following processing is performed for each pixel of the input image, see Figure 3K , Figure 3K This is an eleventh flow chart of the image processing method provided in an embodiment of the present application. Figure 3A Step 105 shown may be performed by Figure 3K Steps 1051 to 1052 are implemented as described below.

[0195] In step 1051 , the product of the pixel and the light sensitivity parameter is determined as the adjusted pixel.

[0196] In some embodiments, the adjusted pixel is determined by multiplying the pixel value of each color channel by the light sensitivity parameter corresponding to each color channel.

[0197] For example, the value of the pixel in the RGB color space is [10, 10, 20], and the light sensitivity parameters corresponding to each color channel include the light sensitivity parameter 1.2 of the red color channel, the light sensitivity parameter 1 of the green color channel, and the light sensitivity parameter 1.5 of the blue color channel. The adjusted pixel values ​​of each color channel of the pixel are 12, 10, and 15 respectively, and the adjusted pixel is [12, 10, 30].

[0198] In step 1052, each adjusted pixel is combined into an input image.

[0199] In some embodiments, the order in which pixels of the input image are combined is equivalent to the order in which each pixel is adjusted.

[0200] Through the embodiments of the present application, the light sensitivity parameters are continuously adjusted to cover the current color temperature scene, thereby avoiding color cast problems in the input image at some high color temperatures or low color temperatures.

[0201] Below, an exemplary application of the image processing method provided in an embodiment of the present application in a practical application scenario will be described.

[0202] In related technologies, palm-scanning devices all adjust color temperature based on a camera module ISP, which cannot accurately perceive the ambient color temperature and uses a set of white balance parameters (i.e., light sensitivity parameters) to cover all color temperature scenes. As a result, the palm image will have color cast problems at some high or low color temperatures, making palm-scanning unusable.

[0203] In order to solve the above problems, an embodiment of the present application proposes an image processing method for improving the accuracy of the color temperature when brushing the palm. By integrating the neural convolution network and the ambient color temperature perception method on the basis of the white balance algorithm, the color temperature accuracy when brushing the palm is improved.

[0204] Take palm image correction as an example, see Figure 4 , Figure 4 This is a schematic diagram of the palm brush image correction process provided in an embodiment of the present application. The palm brush image correction process provided in an embodiment of the present application is explained below.

[0205] exist Figure 4 middle, Figure 4 The present application includes multiple modules for performing palm brush image correction, such as a data collection module 101, a data annotation module 102, a data preprocessing module 103, a model construction module 104, an initialization weight module 105, a training model module 106, a model evaluation module 107, a hyperparameter adjustment module 108, and a prediction and application module 109. Among them, the data collection module 101, the data annotation module 102, the data preprocessing module 103, the model construction module 104, the initialization weight module 105, the training model module 106, and the model evaluation module 107 are used to generate light perception parameters and generate white balance parameters through a convolutional neural network. Each module is described in detail below.

[0206] 1) About Data Collection Module 101

[0207] The data collection module 101 is used to collect data. The data collection process is described in detail below.

[0208] First, determine the target gesture (such as left hand or right hand), collect a certain number of palm images, and collect the palm prints to be collected through the camera collector to obtain the palm images, where the camera collector and the palm prints to be collected are respectively referred to as Figure 5 and Figure 6 , Figure 5 This is a schematic diagram of a camera collector provided in an embodiment of the present application. Figure 6 This is a schematic diagram of the palm prints to be collected provided in an embodiment of the present application.

[0209] Second, determine the negative sample gestures (for example, when the target gesture is the left hand, the corresponding right hand is the negative sample gesture; or when the target gesture is the right hand, the corresponding left hand is the negative sample gesture) and collect the same number of palm images as the target gesture.

[0210] Third, the collected palm images of the target gesture and the negative sample gesture are cleaned by using methods such as finding duplicate values, finding missing values, and finding outliers to remove invalid files. The embodiment of the present application does not limit the method of data cleaning.

[0211] 2) About Data Annotation Module 102

[0212] The data annotation module 102 is used to annotate the collected palm images to obtain annotated palm images.

[0213] 3) About the data preprocessing module 103

[0214] The data preprocessing module 103 is used to obtain, process and extract meaningful features and attributes from the palm images collected by the data collection module 101 using signal data processing technology, so as to facilitate their input into the model for training. The data preprocessing process is described in detail below.

[0215] First, the multi-scale retina (MSR) algorithm is used to enhance the palm image data to obtain an enhanced palm image to improve the visual effect of the image and enhance the useful information in the image. Figure 7 , Figure 7The data enhancement principle diagram provided in the embodiment of the present application obtains the original image in the red, green and blue (RGB) color space, and records the maximum pixel value in the three color channels of red, green and blue in the original image, so as to normalize the image in the MSR image enhancement algorithm, ensure that the algorithm has better stability and robustness when processing images in different brightness ranges, and replace the Gaussian convolution with the average filter to achieve the purpose of improving the operation speed of the MSR image enhancement algorithm. The original image is enhanced based on the MSR image enhancement algorithm, while considering the information of the image at different scales, so as to more comprehensively enhance the quality and details of the image. Then, a matrix containing multiple related parameters is received, and the red, green and blue channels of the RGB color space are obtained based on the relative multiple parameters in the matrix of the related parameters.

[0216] Second, the palm image in the data collection module 101 is cleaned by median filtering. The specific method is to standardize the palm image and perform binarization processing on the standardized data.

[0217] Third, the palm image in the first step is divided into blocks and segments, see Figure 8 , Figure 8 The block schematic diagram provided in the embodiment of this application is as follows: Figure 8 As shown, the palm image is divided into blocks with a certain window length 801 (adjustable, such as 5*5). The palm image contains the pixel value of each pixel, and the palm image is divided into several blocks and corresponding labels. Here, the block mechanism is trained using a network model.

[0218] To avoid the situation where a palm coordinate label affects label accuracy after extracting features from an entire palm image, for example, when a palm image contains different gestures, a voting mechanism is introduced. After training a network model, the input image is divided into weighted blocks. The network model here uses a lightweight convolutional network. The network model inputs the palm grayscale data value as a feature, and then passes it through an attention layer to finally output the palm region of interest (ROI).

[0219] With a resolution of 60x60, the Histogram of Directed Gradients (HOG) feature is extracted (this feature is more suitable for gesture detection). Here, the Sobel algorithm is used to extract the edge features of the palm image, as shown in formula (1) and formula (2).

[0220]

[0221]

[0222] Among them, I x and Iy Represents the gradient values ​​in the horizontal and vertical directions, M(x, y) represents the magnitude of the gradient, and θ(x, y) represents the direction of the gradient.

[0223] Fourth, the HOG features extracted from each block are connected end to end and combined into a large one-dimensional vector. This is the final image feature, which can be input into the regression algorithm for training.

[0224] 4) About the model building module 104

[0225] The model building module 104 is used to select two convolutional networks as the inference network (i.e., encoder) and the generation network (i.e., decoder) to construct the light perception parameter prediction model provided in the embodiment of the present application, see Figure 9 , Figure 9 This is a schematic diagram of the model construction provided in an embodiment of the present application. A generation network that matches the current application scenario is selected from multiple generation networks (automatic white balance generation network, incandescent white balance generation network, and shadow white balance generation network). The generation network takes the latent vector z as input and outputs parameters p(x|z) for observing the conditional distribution. In addition, the unit Gaussian prior p(z) is used to model the latent vector z.

[0226] The generated network includes a fully connected layer followed by four convolutional transpose layers (deconvolution layers). The network architecture of the generated network is that the first layer is the input layer; the second layer is the fully connected layer, and the activation function uses the rectified linear unit (relu, RectifiedLinear Unit); the third layer is the resampling layer, which reshapes the input features of the second layer (resharp) to obtain the updated feature dimensions (batch_size, 7, 7, 32); the fourth layer is the deconvolution layer, which uses 64 convolution kernels, the convolution kernel size is 3x3, the stride is 2, the padding is SAME, and the activation function is relu; the fifth layer is the deconvolution layer, which uses 32 convolution kernels, the convolution kernel size is 3x3, the stride is 2, the padding is SAME, and the activation function is relu; the sixth layer is the deconvolution layer, which uses 16 convolution kernels, the convolution kernel size is 3x3, the stride is 2, the padding is the appropriate padding mode (SAME), and the activation function is relu; the seventh layer is the deconvolution layer, which uses 1 convolution kernel, the convolution kernel size is 3x3, the stride is 1, the padding is SAME, and the activation function is the linear function f(x).

[0227] The inference network is used to obtain the posterior distribution q(z|x), which takes the observation value as input and outputs a set of parameters for the conditional distribution of the potential representation. The inference network includes 3 convolutions + 1 fully connected network layers. The network architecture of the inference network is as follows: the first layer is the input layer; the second layer is the convolution layer, using 16 convolution kernels, the convolution kernel size is 3x3, the stride is 2, the padding is SAME, and the activation function is relu; the third layer is the convolution layer, using 32 convolution kernels, the convolution kernel size is 3x3, the stride is 2, the padding is SAME, and the activation function is relu; the fourth layer is the convolution layer, using 64 convolution kernels, the convolution kernel size is 3x3, the stride is 2, the padding is SAME, and the activation function is relu; the fifth layer is the fully connected layer.

[0228] 5) About initializing weight module 105

[0229] The weight initialization module 105 is used to initialize the weights with a random value less than the weight threshold to avoid output saturation caused by weight values ​​greater than the weight threshold when the model starts training. The methods for initializing the weights include zero initialization (initializing all weights to zero) and random number initialization (initializing the weights to a random value less than the weight threshold, usually from a uniform distribution or a normal distribution) and other weight initialization methods. This application does not limit the selection of the method for initializing the weights.

[0230] 6) About the training model module 106

[0231] The training model module 106 is used to start training based on the model selected in the model construction module 104 and the parameters and data required to meet the network model. Batch normalization is avoided during training because the use of small batch processing will cause additional randomness, thereby exacerbating the instability of random sampling. The model training process is described in detail below.

[0232] First, define the model's loss function and optimizer.

[0233] Second, training is performed by maximizing the Evidence Lower Bound (ELBO) of the marginal log-likelihood, as shown in Equation (3).

[0234]

[0235] The model is trained starting from the iterative dataset. During each iteration, we pass the image to the inference network to obtain a set of mean and log-variance parameters of the approximate posterior q(z|x). The mean and variance are used to reduce oscillations. We apply the reparameterization technique to sample from q(z|x) and pass the reparameterized samples to the generator network to obtain the log-odds (logit) of the observed value of the generative distribution p(x|z), where logit represents the sample input to the last fully connected layer of the generator network.

[0236] 7) About Model Evaluation Module 107

[0237] The model evaluation module 107 is used to input the input image into the model for generating pictures that has been trained by the training model module 106, sample a set of latent vectors z from the unit Gaussian prior distribution p(z), and generate a network to convert the latent vectors into the logit of the observed values ​​to obtain the probability distribution p(x|z), and use the probability distribution p(x|z) as the red, green, and blue gain (RGB gain) of the white balance parameter (light sensitivity parameter) corresponding to the input picture.

[0238] 8) About Hyperparameter Adjustment Module 108

[0239] The hyperparameter adjustment module 108 is used to update the white balance parameters of the image, see Figure 10 , Figure 10 This is a parameter update principle diagram provided in an embodiment of the present application.

[0240] like Figure 10 As shown, in step 901, the automatic white balance parameters (AWB) in the image signal processor (ISP) in the camera are initialized. The white balance parameters are used to restore the white imaged under ambient light of different color temperatures to true white (usually the white observed by the human eye under natural daylight) through a certain algorithm.

[0241] In step 902, block histogram color temperature statistics are performed on the input image, the input image is passed through the model evaluation module 107 to obtain white balance parameters, and the input image is input into the ambient color temperature perception estimation module 201. In the ambient color temperature perception estimation module 201, the block histogram color temperature statistics of the input image in step 902 and the color temperature error rate of each channel in step 903 are obtained.

[0242] 9) About the Ambient Color Temperature Perception Estimation Module 201

[0243] The ambient color temperature perception estimation module 201 is used to obtain the current color temperature of the image. Figure 11 , Figure 11 This is a schematic diagram of the ambient color temperature perception estimation provided by an embodiment of the present application. Unlike the traditional grayscale world assumption and white point estimation, the ambient color temperature perception estimation module 201 performs color temperature estimation based on color constancy, by simulating the color constancy characteristics of the human visual system, that is, maintaining stable color perception under different lighting conditions. The process of the ambient color temperature perception estimation module 201 is described in detail below.

[0244] First, color space conversion.

[0245] Convert the image from its original color space (usually RGB) to a color space that is more suitable for white balance processing, such as the coordinate (XYZ) color space or the illuminance chromaticity (Lab) color space developed by the International Commission on Illumination (CIE). The method for converting an image in the HSV color space to the RGB color space is shown in the following formula (4).

[0246] f=(H / 60)-I

[0247] a=V*(1-S)

[0248] b=V*(1-S*f)

[0249] c=V*(1-S*(1-f))

[0250] R=V, G=c, B=a; if i=0

[0251] R=b, G=V, B=a; if i=1

[0252] R=a, G=V, B=c; if i=2

[0253] R=a, G=b, B=V; if i=3

[0254] R=c, G=a, B=V; if i=4

[0255] R=V, G=a, B=b; if i=5 (4)

[0256] Where i = get the integer of (H / 60), where i represents the rounded value of H / 60. In the hue-saturation-value (HSV) color space, hue (H) typically ranges from 0° to 360°, while saturation (S) and value (V) typically range from 0 to 1. Intermediate variables are calculated based on hue (H), saturation (S), and value (V), and the corresponding values ​​for each color channel in the red-green-blue (RGB) color space (red (R), green (G), and blue (B)) are obtained, enabling conversion from HSV color space to RGB color space. When the scene focuses on the image's color, the image is converted to HSV space. When the scene focuses on the image's brightness, the image is converted to Lab space.

[0257] Second, perform histogram color temperature statistics on the image.

[0258] The image is divided into 8*8 blocks, and the histogram color temperature statistics of the divided images are performed to estimate the color temperature of the light source in the current scene (current color temperature).

[0259] Third, calculate the color temperature error rate of each channel.

[0260] Based on the current color temperature and the white balance parameters of the input image obtained by the model evaluation module 107, the expected color temperature of the input image is determined according to the white balance parameters, and the color temperature error rate of each channel is counted based on the current color temperature and the expected color temperature.

[0261] Continue to see Figure 10 , determining color temperature error rates of N frames as the frame color temperature error rate, where N is an integer greater than 1, performing weighted summation of the color temperature error rates of the N frames to obtain the frame color temperature error rate, and in step 904, determining the relationship between the frame color temperature error rate and a first threshold value. When the frame color temperature error rate is less than the first threshold value, executing step 9041 to determine the white balance model as the unchanged mode and maintain the current color temperature. When the frame color temperature error rate is greater than the first threshold value, executing step 9042 to determine the relationship between the color temperature error rate of the image and a second threshold value. When the color temperature error rate of the image is greater than the second threshold value, executing step 90421 to determine the white balance model as the slow mode, adjusting the corresponding color temperature parameters, and obtaining the white balance parameter values ​​corresponding to the white balance model in the slow mode. When the color temperature error rate of the image is less than the second threshold value, executing step 90422 to determine the white balance model as the fast mode, adjusting the corresponding color temperature parameters, and obtaining the white balance parameter values ​​corresponding to the white balance model in the fast mode. That is, the greater the color temperature error rate, the faster the white balance parameter adjustment.

[0262] In step 905, the input image is updated based on the white balance parameter value in the white balance corresponding mode, and the error rate statistics are re-performed on the updated input image, and the color temperature error rates of multiple frame images are calculated to obtain the frame color temperature error rate. In step 906, the relationship between the frame color temperature error rate and the first threshold is determined. When the frame color temperature error rate is still greater than or equal to the first threshold, the N-frame statistics in step 907 are executed to determine the adjusted N frames of images, where N is a positive integer greater than 1. The color temperature of the N adjusted input images is re-adjusted until the frame color temperature error rate is determined to be less than the first threshold in the determination of the relationship between the frame color temperature error rate and the first threshold in step 906.

[0263] 10) About application module 109

[0264] The application module 109 is used to adjust the color temperature of the input data using the palm image correction method provided in the embodiment of the present application to obtain an adjusted input image, thereby further improving the accuracy of the image color temperature.

[0265] In summary, the embodiment of the present application performs histogram statistics on the input image to obtain the current color temperature corresponding to the input image, thereby improving the accuracy of estimating the current color temperature of the image, and continuously adjusting the white balance parameters. The adjusted white balance parameters can adjust the color temperature of the input data in extreme color temperature environments, covering a variety of color temperature scenarios, and improving the accuracy of the image color temperature.

[0266] The following continues to describe the exemplary structure of the image processing device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the image processing device 555 of the memory 550 may include:

[0267] The light sensitivity prediction module 5551 is used to perform light sensitivity parameter prediction processing on the input image to obtain the light sensitivity parameters of the input image.

[0268] The color temperature error rate acquisition module 5552 is used to determine the expected color temperature corresponding to the light sensitivity parameter from the correspondence between the candidate light sensitivity parameters and the candidate expected color temperature; determine the current color temperature of the input image, and determine the color temperature error rate of the input image based on the expected color temperature and the current color temperature.

[0269] The image updating module 5553 is configured to update the light sensitivity parameters based on the color temperature error rate to obtain updated light sensitivity parameters; and adjust the input image using the updated light sensitivity parameters to obtain an adjusted input image.

[0270] In some embodiments, the light perception prediction module 5551 is also used to map the input image to obtain the probability distribution of the input image in the latent space; sample latent space samples from the latent space, and remap the latent space samples to obtain prediction features corresponding to the input image, wherein the latent space samples conform to the probability distribution of the input image in the latent space; and determine the light perception parameters of the input image based on the prediction features corresponding to the input image.

[0271] In some embodiments, the light perception prediction module 5551 is also used to perform the following processing for each color channel in the color space corresponding to the predicted feature, determine the first probability distribution of the predicted feature in each color channel, the first probability distribution is used to characterize the proportion of pixels with different pixel values ​​in each color channel in the pixels of the predicted feature; determine the pixel value with the maximum probability in the first probability distribution corresponding to each color channel as the target pixel value; and determine the inverse of the target pixel value as the light perception parameter of the input image.

[0272] In some embodiments, the light perception prediction module 5551 is also used to map the predicted features to obtain a second probability distribution of the input image, wherein the second probability distribution is used to characterize the weight of the light perception parameters corresponding to each color channel of the input image; from the second probability distribution, the target weight of the light perception parameters of the target color channel is determined; the ratio of the preset parameter to the target weight is determined as the light perception parameter coefficient; the following processing is performed for each color channel in the color space corresponding to the input image, and the product of the light perception parameter coefficient and the weight of the light perception parameter corresponding to each color channel is determined as the light perception parameter of the input image.

[0273] In some embodiments, the color temperature error rate acquisition module 5552 is further used to convert the input image from the current color space to the target color space to obtain a converted input image; perform color histogram statistics on the converted input image to obtain a color histogram corresponding to the input image; and determine the current color temperature corresponding to the peak of the color histogram from the correspondence between the candidate peaks and the candidate color temperatures.

[0274] In some embodiments, the color temperature error rate acquisition module 5552 is further configured to acquire a difference between the desired color temperature and the current color temperature; and determine the color temperature error rate of the input image based on the difference.

[0275] In some embodiments, the color temperature error rate acquisition module 5552 is further used to determine the color temperature error rate of the input image by one of the following methods: determining the ratio of the difference value to the current color temperature as the color temperature error rate of the input image; and determining the ratio of the square of the difference value to the square of the current color temperature as the color temperature error rate of the input image.

[0276] In some embodiments, the image update module 5553 is further used to determine the update learning rate of the input image as a first learning rate when the color temperature error rate is greater than or equal to the color temperature threshold; when the color temperature error rate is less than the color temperature threshold, determine the update learning rate of the input image as a second learning rate; determine the current light perception parameter corresponding to the current color temperature from the correspondence between the candidate color temperature and the candidate light perception parameter; update the light perception parameter based on the update learning rate of the input image and the current light perception parameter to obtain the updated light perception parameter.

[0277] In some embodiments, the image update module 5553 is also used to determine the ratio of the difference between the light sensitivity parameter and the current light sensitivity parameter to the preset parameter as the update step size; and to determine the product of the update learning rate and the update step size and the sum of the current light sensitivity parameter as the updated light sensitivity parameter.

[0278] In some embodiments, the image update module 5553 is further used to obtain an adjustment factor corresponding to the color temperature error rate from the correspondence between the candidate color temperature error rate and the candidate adjustment factor, wherein the adjustment factor is proportional to the color temperature error rate; and the product of the adjustment factor and the light perception parameter is determined as the updated light perception parameter.

[0279] In some embodiments, the image update module 5553 is further configured to perform the following processing on each pixel of the input image: determining the product of the pixel and the light sensitivity parameter as an adjusted pixel; and combining each adjusted pixel into the input image.

[0280] In some embodiments, the image update module 5553 is also used to obtain at least one neighbor image of the input image and determine the updated light sensitivity parameters of each neighbor image; and perform weighted averaging on the updated light sensitivity parameters of the input image and the updated light sensitivity parameters of each neighbor image to obtain the target light sensitivity parameters.

[0281] In some embodiments, the image updating module 5553 is further configured to adjust the input image using the target light sensitivity parameter to obtain an adjusted input image.

[0282] An embodiment of the present application provides a computer program product, which includes computer-executable instructions. The computer-executable instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions, so that the electronic device performs the image processing method described above in the embodiment of the present application.

[0283] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the image processing method provided by the embodiment of the present application, for example, Figures 3A to 3K The image processing method is shown.

[0284] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0285] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0286] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0287] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0288] In summary, through the embodiment of the present application, the light sensitivity parameters of the input image are predicted and processed to obtain the light sensitivity parameters of the input image. From the correspondence between the candidate light sensitivity parameters and the candidate expected color temperature, the expected color temperature corresponding to the light sensitivity parameters is determined, and based on the expected color temperature and the current color temperature, the color temperature error rate of the input image is determined. The light sensitivity parameters are updated based on the color temperature error rate to obtain updated light sensitivity parameters. The input image is adjusted according to the updated light sensitivity parameters to obtain an adjusted input image. In this way, by updating the light sensitivity parameters, the color temperature of the input image in some extreme color temperature environments can be adjusted to improve the accuracy of the image color temperature.

[0289] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Performing light sensitivity parameter prediction processing on the input image to obtain the light sensitivity parameters of the input image; Determining the desired color temperature corresponding to the candidate light sensitivity parameters from the corresponding relationship between the candidate light sensitivity parameters and the candidate desired color temperatures; determining a current color temperature of the input image, and determining a color temperature error rate of the input image based on the expected color temperature and the current color temperature; updating the light perception parameter based on the color temperature error rate to obtain an updated light perception parameter; The input image is adjusted using the updated light sensitivity parameters to obtain an adjusted input image.

2. The method according to claim 1, characterized in that The performing light sensitivity parameter prediction processing on the input image to obtain the light sensitivity parameters of the input image includes: Mapping the input image to obtain a probability distribution of the input image in a latent space; Sampling latent space samples from the latent space and remapping the latent space samples to obtain prediction features corresponding to the input image, wherein the latent space samples conform to the probability distribution of the input image in the latent space; Determine light sensitivity parameters of the input image based on the predicted features corresponding to the input image.

3. The method according to claim 2, characterized in that The determining the light sensitivity parameter of the input image based on the prediction feature corresponding to the input image includes: The following processing is performed for each color channel in the color space corresponding to the predicted feature: Determine a first probability distribution of the prediction feature in each color channel, where the first probability distribution is used to represent a proportion of pixels of different pixel values ​​in each color channel among the pixels of the prediction feature; Determine the pixel value with the maximum probability in the first probability distribution corresponding to each color channel as the target pixel value; The reciprocal of the target pixel value is determined as the light sensitivity parameter of the input image.

4. The method according to claim 2, characterized in that The determining the light sensitivity parameter of the input image based on the prediction feature corresponding to the input image includes: Performing mapping processing on the predicted features to obtain a second probability distribution of the input image, wherein the second probability distribution is used to represent the weight of the light perception parameter of the input image corresponding to each color channel; determining a target weight of a light perception parameter of a target color channel from the second probability distribution; Determining the ratio of the preset parameter to the target weight as a light sensitivity parameter coefficient; The following processing is performed for each color channel in the color space corresponding to the input image: The product of the light sensitivity parameter coefficient and the weight of the light sensitivity parameter corresponding to each color channel is determined as the light sensitivity parameter of the input image.

5. The method according to claim 1, wherein The determining the current color temperature of the input image includes: Converting the input image from the current color space to the target color space to obtain a converted input image; Performing color histogram statistics on the converted input image to obtain a color histogram corresponding to the input image; The current color temperature corresponding to the peak value of the color histogram is determined from the correspondence between the candidate peak values ​​and the candidate color temperatures.

6. The method according to claim 1, characterized in that The determining, based on the expected color temperature and the current color temperature, a color temperature error rate of the input image includes: Obtaining a difference between the desired color temperature and the current color temperature; Based on the difference, a color temperature error rate of the input image is determined.

7. The method according to claim 6, characterized in that The determining, based on the difference, a color temperature error rate of the input image includes: Determine the color temperature error rate of the input image by one of the following methods: determining a ratio of the difference to the current color temperature as a color temperature error rate of the input image; A ratio of the square of the difference to the square of the current color temperature is determined as a color temperature error rate of the input image.

8. The method according to claim 1, characterized in that The updating of the light sensitivity parameter based on the color temperature error rate to obtain the updated light sensitivity parameter includes: When the color temperature error rate is greater than or equal to a color temperature threshold, determining the updated learning rate of the input image as a first learning rate; When the color temperature error rate is less than the color temperature threshold, determining the updated learning rate of the input image as a second learning rate; Determining the current light sensitivity parameter corresponding to the current color temperature from the correspondence between the candidate color temperatures and the candidate light sensitivity parameters; Based on the updated learning rate of the input image and the current light sensitivity parameter, the light sensitivity parameter is updated to obtain an updated light sensitivity parameter.

9. The method according to claim 8, characterized in that The updating of the light sensitivity parameter based on the updated learning rate of the input image and the current light sensitivity parameter to obtain the updated light sensitivity parameter includes: Determining the ratio of the difference between the light sensitivity parameter and the current light sensitivity parameter to a preset parameter as an update step length; The updated light sensing parameter is determined as the sum of the product of the updated learning rate and the updated step size and the current light sensing parameter.

10. The method according to claim 1, characterized in that The updating of the light sensitivity parameter based on the color temperature error rate to obtain the updated light sensitivity parameter includes: Obtaining an adjustment factor corresponding to the color temperature error rate from a correspondence between candidate color temperature error rates and candidate adjustment factors, wherein the adjustment factor is proportional to the color temperature error rate; The product of the adjustment factor and the light sensitivity parameter is determined as the updated light sensitivity parameter.

11. The method according to claim 1, wherein The step of adjusting the input image using the updated light sensitivity parameter to obtain an adjusted input image includes: Perform the following processing on each pixel of the input image: determining a product of the pixel and the light sensitivity parameter as an adjusted pixel; The adjusted pixels are combined into the input image.

12. The method according to claim 1, characterized in that Before adjusting the input image using the updated light sensitivity parameters to obtain the adjusted input image, the method further includes: Acquire at least one neighbor image of the input image, and determine updated light sensitivity parameters of each neighbor image; performing weighted averaging on the updated light sensitivity parameter of the input image and the updated light sensitivity parameter of each of the neighboring images to obtain a target light sensitivity parameter; The step of adjusting the input image using the updated light sensitivity parameter to obtain an adjusted input image includes: The input image is adjusted according to the target light sensitivity parameter to obtain an adjusted input image.

13. An image processing device, characterized in that: The device comprises: A light sensitivity prediction module, configured to perform light sensitivity parameter prediction processing on an input image to obtain light sensitivity parameters of the input image; a color temperature error rate acquisition module, configured to determine, from a correspondence between candidate light sensitivity parameters and candidate expected color temperatures, an expected color temperature corresponding to the light sensitivity parameters; determine a current color temperature of the input image, and determine a color temperature error rate of the input image based on the expected color temperature and the current color temperature; An image updating module is configured to update the light sensitivity parameters based on the color temperature error rate to obtain updated light sensitivity parameters; and adjust the input image using the updated light sensitivity parameters to obtain an adjusted input image.

14. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; The processor is configured to implement the image processing method according to any one of claims 1 to 12 when executing the computer-executable instructions or computer program stored in the memory.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 12 is implemented.

Images