Image processing method, device, storage medium, equipment and program product

Abstract

The application discloses an image processing method and device, a storage medium, equipment and a program product. The method comprises the following steps: acquiring color mapping relationships of each color block in a plurality of color blocks under different illumination conditions, wherein the color mapping relationship is determined according to a reference light source and color values of the plurality of color blocks under different illumination conditions; acquiring a to-be-processed image containing a virtual pixel set and a real scene pixel set; acquiring an illumination condition of a current ambient light, wherein the illumination condition at least comprises a color temperature and an illumination of the current ambient light; selecting a target color mapping relationship matched with the illumination condition of the current ambient light from the color mapping relationship, and performing pixel color adjustment processing on the to-be-processed image based on the target color mapping relationship to generate a target image, so that the illumination effect of the adjusted pixel in the target image is consistent with the illumination effect of the current ambient light, the calculation complexity is reduced, and the image quality is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image processing method, apparatus, storage medium, device, and program product. Background Technology

[0002] Illumination estimation has wide applications in computer vision, image processing, and virtual reality and augmented reality. Traditional illumination estimation methods include ray tracing, radiosity calculation, and physics-based methods. While these methods can generate relatively realistic lighting effects, they are computationally complex and difficult to meet real-time requirements, especially in mobile devices and high frame rate applications.

[0003] With the development of deep learning technology, methods based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have gradually emerged. These methods can learn lighting features from large amounts of data, achieving faster and more accurate lighting estimation. However, these methods still face challenges such as difficulties in data acquisition and annotation, and difficulty in adapting to complex and changing lighting conditions. Summary of the Invention

[0004] This application provides an image processing method, apparatus, storage medium, device, and program product that can automatically adjust the pixels of the image to be processed to generate a target image when the current ambient light illumination conditions change, so that the illumination effect of the adjusted pixels in the target image is consistent with the current ambient light illumination effect, thereby reducing computational complexity and improving image quality.

[0005] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0006] Obtain the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on the reference light source and the color values ​​of multiple color blocks under different lighting conditions.

[0007] An image to be processed, comprising a virtual pixel set and a real-scene pixel set, is acquired. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired by an image acquisition device.

[0008] Obtain the current ambient light illumination conditions, wherein the illumination conditions include at least the color temperature and illuminance of the current ambient light;

[0009] Select a target color mapping relationship from the color mapping relationships that matches the current ambient light illumination conditions, and perform pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, so that the illumination effect of the adjusted pixels in the target image is consistent with the illumination effect of the current ambient light.

[0010] On the other hand, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0011] The first acquisition unit is used to acquire the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on the reference light source and the color values ​​of the multiple color blocks under different lighting conditions.

[0012] The second acquisition unit acquires an image to be processed that includes a virtual pixel set and a real-scene pixel set. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired by an image acquisition device.

[0013] The third acquisition unit is used to acquire the current ambient light illumination conditions, wherein the illumination conditions include at least the color temperature and illuminance of the current ambient light.

[0014] The processing unit is configured to select a target color mapping relationship from the color mapping relationship that matches the current ambient light illumination conditions, and perform pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, such that the illumination effect of the adjusted pixels in the target image is consistent with the illumination effect of the current ambient light.

[0015] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the image processing method as described in any of the above embodiments.

[0016] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the image processing method as described in any of the above embodiments by calling the computer program stored in the memory.

[0017] On the other hand, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the image processing method as described in any of the above embodiments.

[0018] This application embodiment obtains the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on a reference light source and the color values ​​of multiple color blocks under different lighting conditions. It acquires an image to be processed, including a virtual pixel set and a real-scene pixel set. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired through an image acquisition device. It obtains the current ambient light lighting conditions, including at least the color temperature and illuminance of the current ambient light. It selects a target color mapping relationship from the color mapping relationships that matches the current ambient light lighting conditions, and performs pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image. This ensures that the lighting effect of the adjusted pixels in the target image is consistent with the lighting effect of the current ambient light. This application embodiment can automatically adjust the pixels of the image to be processed to generate a target image when the current ambient light lighting conditions (including color temperature and illuminance) change, ensuring that the lighting effect of the adjusted pixels in the target image is consistent with the lighting effect of the current ambient light. This avoids the color mismatch problem that may occur between the virtual pixels and real-scene pixels calculated in traditional methods. By pre-obtaining the color mapping relationships of multiple color blocks under different lighting conditions and adjusting pixel colors based on these mapping relationships, the complex calculations required for real-time calculation of the changes of each pixel under different lighting conditions are avoided, effectively reducing computational complexity. By accurately matching and applying the target color mapping relationship, pixel-level color adjustments can be made to the image, more accurately simulating changes in lighting conditions in real-world environments, thereby significantly improving the realism and visual quality of the image. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the application scenarios of traditional image processing techniques.

[0021] Figure 2 This is a schematic diagram of the structure of the image processing system provided in an embodiment of this application.

[0022] Figure 3 This is a schematic flowchart of the image processing method provided in an embodiment of this application.

[0023] Figure 4 This is a schematic diagram of a first application scenario of the image processing method provided in the embodiments of this application.

[0024] Figure 5This is a schematic diagram of a second application scenario of the image processing method provided in the embodiments of this application.

[0025] Figure 6 This is a schematic diagram of a third application scenario of the image processing method provided in the embodiments of this application.

[0026] Figure 7 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0027] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] This application provides an image processing method, apparatus, storage medium, device, and program product. Specifically, the image processing method of this application can be executed by a terminal device or a server. The computer device can be a terminal device or a server. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, smart vehicle terminal, etc. The terminal device may also include a client, which can be a client of an application capable of image processing. For example, the client includes at least one of a program client and a web client. For example, the application can be a social application, an instant messaging application, a shopping application, a virtual reality interactive application, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0030] The embodiments of this application can be applied to various application scenarios such as image processing, virtual reality, and extended reality.

[0031] Lighting estimation aims to infer lighting-related information from an image or scene. Common applications include augmented and virtual reality, computer graphics, and image analysis and processing.

[0032] Augmented Reality and Virtual Reality: In augmented reality and virtual reality applications, accurate lighting estimation is crucial for seamlessly integrating virtual objects with real-world scenes. Only by ensuring the lighting effects of virtual objects match the real environment can the user's immersion be enhanced. For example, in AR virtual try-on applications, the lighting effects of virtual clothing need to be adjusted in real-time based on the lighting conditions of the user's environment, making the virtual clothing feel as if it's actually being worn. This includes adjusting the clothing's color, shadows, and contrast.

[0033] Computer graphics: For computer-generated images and animations, lighting estimation can help create more realistic lighting effects, making scenes more believable. For example, in film special effects production and game development, accurate lighting models and lighting estimation techniques can improve the visual quality of scenes. When the color temperature and illuminance of virtual light sources change, the lighting effects of the scene also need to be adjusted accordingly.

[0034] Image Analysis and Processing: Illumination estimation is also of great significance for image analysis and processing. For example, in tasks such as image shadow removal and image enhancement, understanding the distribution and intensity of illumination can help remove shadows or enhance the contrast and brightness of an image.

[0035] Among them, the methods for implementing illumination estimation can be divided into traditional methods and deep learning-based methods.

[0036] Traditional methods include:

[0037] Ray tracing is a method for calculating lighting effects by simulating the path of light rays in a scene. It can accurately calculate effects such as light reflection, refraction, and shadows, but it has high computational complexity and long computation time, making it difficult to use in real-time applications.

[0038] Radiometrics: Primarily used to calculate the transfer of light energy between object surfaces in a scene, taking into account factors such as the reflectivity and emissivity of the object's surface. The radiometric method can generate relatively realistic lighting effects, but it also suffers from high computational complexity.

[0039] Physical model-based methods utilize physical principles and optical laws to establish lighting models, such as lighting models based on Lambert's law. These methods require an accurate understanding of the physical properties of the scene, but in practical applications, obtaining accurate physical parameters is often difficult.

[0040] Deep learning-based methods can include:

[0041] Convolutional Neural Network (CNN) based methods: CNNs have powerful image processing capabilities, automatically learning lighting features from images. Through extensive training data, CNNs can learn lighting patterns in different scenes, thus enabling lighting estimation. For example, some studies use CNNs to predict information such as the direction, intensity, and color of light in a scene.

[0042] Methods based on recurrent neural networks (RNNs): RNNs are suitable for processing data with time-series characteristics. In illumination estimation, RNNs can be used to handle illumination changes in video sequences. By analyzing consecutive frames, RNNs can predict dynamic changes in illumination, improving the accuracy of illumination estimation.

[0043] Image analysis-based methods: (1) Marker-based methods: using auxiliary markers to estimate illumination. For example, marker-based shadow analysis methods use the brightness or spatial projection relationship of shadows to infer the position and distribution of light sources; marker-based surface image methods extract the brightness and darkness information or surface reflection environmental information of markers and calculate the direction of light sources according to the laws of light reflection. (2) Marker-free methods: analyzing the illumination in the scene solely through image content. Among them, methods that require scene structure analysis perform three-dimensional reconstruction by analyzing image information, and then analyze scene illumination based on the reconstructed three-dimensional image; methods that do not require scene structure analysis directly use the features of the image to estimate the scene illumination.

[0044] Among the challenges of illumination estimation are:

[0045] Adaptability to complex scenes: In practical applications, scenes are often very complex, containing various objects, different materials, and complex lighting conditions. Existing lighting estimation methods may be inaccurate when dealing with complex scenes, and improving the adaptability of methods to complex scenes is a challenge.

[0046] Real-time performance: For some real-time applications, such as augmented reality and virtual reality, lighting estimation needs to be completed within a short time to ensure a good user experience. However, some current accurate lighting estimation methods have high computational complexity, making it difficult to meet real-time requirements.

[0047] Data acquisition and annotation: Deep learning-based illumination estimation methods require a large amount of labeled data for training, but acquiring high-quality illumination data and accurately annotating it is a difficult task, which limits the application of deep learning methods in illumination estimation.

[0048] The color and illuminance of a light source can change at any time, and illumination estimation needs to be able to adapt to changing illumination conditions.

[0049] The illumination estimation methods corresponding to the physical model-based methods can include:

[0050] Spectral analysis and modeling:

[0051] The color of light is essentially determined by its spectral distribution. At a physical level, the spectrum of a light source can be analyzed. For example, common light sources such as incandescent lamps, fluorescent lamps, and LED lamps have different spectral characteristics. The spectrum of an incandescent lamp is continuous, similar to blackbody radiation, while the spectra of fluorescent lamps and LED lamps may be discrete combinations of spectral lines. By establishing a spectral reflectance model and combining it with the spectral reflectance of objects (different objects reflect light of different wavelengths differently), the influence of light color on the color of objects can be accurately estimated based on changes in the light source spectrum.

[0052] Planck's law is used to describe the spectral energy distribution of blackbody radiation (such as simulating thermal radiation sources like the sun), which is related to the temperature of the light source. When the temperature of the light source changes (for example, the color temperature of the sun changes at different times of the day), Planck's law can be used to calculate the changes in the spectrum, thus adapting to changes in the color of the light.

[0053] Two-color reflectance model (DRM):

[0054] DRM divides reflected light into diffuse and specular reflection. Under different light source colors, the diffuse reflection component mainly depends on the object's own color attributes and the overall intensity of the light source, while the specular reflection component is more strongly correlated with the light source color. For example, on the surface of a metallic object, specular reflection accounts for a large proportion, and changes in the light source color will be directly reflected in the specular reflection area. By estimating the diffuse and specular reflection components separately and dynamically adjusting the proportions of these two components according to changes in the light source color, it is possible to better adapt to changes in the color of illumination.

[0055] Among them, the illumination estimation methods corresponding to deep learning-based methods can include:

[0056] Data-driven color adaptation model:

[0057] The neural network is trained using a large amount of data labeled with different light source colors. For example, images of the same object taken under different colored light sources (such as LEDs of different colors like red, green, and blue) are collected as training data. The neural network learns the relationship between changes in light source color and the object's color, thus automatically adjusting the illumination estimation when encountering new changes in light source color.

[0058] Generative Adversarial Networks (GANs) can be used to generate images that adapt to different light source colors. In a GAN, the generator learns to produce images with the correct colors under different light source colors, while the discriminator judges whether the generated image conforms to the true color distribution. Through adversarial training, the generator can generate images with correct colors based on the input light source color change information, thus adapting to changes in lighting color.

[0059] Multi-task learning and adaptive adjustment of deep neural networks:

[0060] Design a neural network to simultaneously perform tasks such as light source color estimation and object color correction. For example, one branch of the network estimates the color of the light source (e.g., by analyzing the colors of highlight or shadow areas in an image to infer the light source color), while another branch uses the estimated light source color to correct the object color. During training, the network learns to automatically adjust its object color correction strategy based on changes in the light source color.

[0061] An adaptive neural network architecture, such as a dynamic convolutional neural network, is employed. When a change in the color of the light source is detected, the network can dynamically change the parameters of the convolutional kernels (such as size and weights) or adjust the number of layers to better adapt to the impact of changes in light color on image color. This adaptive adjustment can improve the network's performance under different light source color environments.

[0062] However, physics-based methods rely on certain idealized assumptions. When complex and diverse real-world scenarios exceed these assumptions, these methods can become biased. Learning-based methods depend on datasets. Data collection and annotation are extremely labor-intensive. Furthermore, the size of the neural network directly impacts the accuracy of the results. Large neural networks are a heavy burden for mobile device applications, even hindering commercial viability. Therefore, more flexible, computationally efficient methods that do not rely on large datasets are needed to achieve ambient light-adaptive color adjustment on mobile devices.

[0063] If illumination estimation and the computationally generated virtual pixels are implemented within a single neural network, color conflicts will occur in the image containing both computationally generated virtual pixels and camera pixels when lighting conditions change. This is because the illumination estimation result may differ from the camera's lighting condition cues. Implementing illumination estimation and computationally generated virtual pixels in two separate neural networks increases both training complexity and computational complexity and resource consumption in practical applications.

[0064] For example, when ambient light conditions (including color temperature and illuminance) change, the actual scene light captured by the camera also changes accordingly. These changing light rays are converted into electrical signals, which are then processed through a digital image processing link and output as a digital image, i.e., a combination of pixels. For ease of description, these pixels reflecting the actual scene are called "real-world pixels." The color attributes of real-world pixels (including brightness, saturation, and hue) are adjusted in real time as ambient light conditions change. However, if the colors of the calculated virtual pixels (including brightness, saturation, and hue) are not adjusted accordingly to changes in ambient light conditions, inconsistencies will occur between the calculated pixels and the real-world pixels.

[0065] For example, such as Figure 1 As shown, suppose there is a white porcelain plate in a real-world scene containing five pale yellow lemons. A set of real-world pixels, representing the white porcelain plate with the five pale yellow lemons, is acquired through a camera and digital image processing link. Now, if a set of virtual pixels representing a virtual pale yellow lemon in the center is generated and superimposed on the real-world pixels, an image is formed containing both the real-world pixels (the five pale yellow lemons on the outside) and the calculated virtual pixels (the virtual pale yellow lemon in the center). If the porcelain plate is then moved indoors from outdoors, and the ambient lighting conditions change from sunlight (e.g., ambient lighting condition 1, color temperature approximately 6500K) to warm-toned fluorescent lighting (e.g., ambient lighting condition 2, color temperature approximately 3000K), the pale yellow lemons will appear orange under the fluorescent light, and the real-world pixels will also turn orange. However, the calculated virtual lemons will remain pale yellow, thus conflicting with the color of the real lemons represented by the real-world pixels.

[0066] Therefore, in this case, a technical means is needed to enable the computationally generated virtual pixels to adjust accordingly based on changes in ambient light conditions, so as to ensure the consistency of their colors with real-world pixels, thereby achieving a natural integration of virtual content and real-world scenes.

[0067] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application. The image processing system includes a terminal device 10 and a server 20, etc.; the terminal device 10 and the server 20 are connected via a network, such as a wired or wireless network.

[0068] The terminal device 10 can be used to display a graphical user interface (GUI). This terminal is used to interact with the user through the GUI, such as downloading and installing a client application, running a mini-program, or accessing a website. In this embodiment, the terminal device 10 can be a terminal device with a client application capable of image processing installed.

[0069] Server 20 provides background services to clients of applications (such as image processing applications) in terminal device 10. For example, server 20 can be a background server for the aforementioned application (such as an image processing application). Server 20 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0070] This application provides an image processing method, which can be executed by a terminal device or a server, or by both a terminal device and a server. This application uses the example of the image processing method being executed by a server to illustrate the method.

[0071] During image processing, server 20 acquires the color mapping relationships of various color blocks under different lighting conditions. These color mapping relationships are determined based on a reference light source and the color values ​​of the multiple color blocks under different lighting conditions. It acquires an image to be processed, containing a virtual pixel set and a real-world pixel set. The virtual pixel set is generated based on the reference light source, and the real-world pixel set is acquired through an image acquisition device. It acquires the current ambient light conditions, including at least the color temperature and illuminance of the current ambient light. It selects a target color mapping relationship from the color mapping relationships that matches the current ambient light conditions, and performs pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image. This target image ensures that the lighting effect of the adjusted pixels in the target image is consistent with the lighting effect of the current ambient light. The target image is then sent to terminal device 10 for display.

[0072] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0073] Please see Figures 3 to 6 , Figure 3 This is a schematic flowchart of the image processing method provided in the embodiments of this application. Figures 4 to 6 These are all schematic diagrams illustrating application scenarios of the image processing method provided in the embodiments of this application. The method includes the following steps:

[0074] Step 110: Obtain the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on the reference light source and the color values ​​of multiple color blocks under different lighting conditions.

[0075] For example, color swatches are typically selected from representative colors, covering a wide range within the RGB color space to ensure the comprehensiveness and accuracy of the mapping. Color swatches can be evenly distributed on standard color charts, such as the ColorChecker 24-color chart, the datacolor 24-color chart, or the ColorChecker SG 140 color chart.

[0076] For example, lighting conditions can include color temperature and illuminance. Color temperature typically ranges from 2300K to 6500K, and illuminance ranges from 0 lx to 100,000 lx, depending on the application scenario. To obtain a more refined mapping relationship, multiple color temperature and illuminance combinations can be set, and the interval step between adjacent color temperatures or illuminances can be adjusted as needed.

[0077] For example, a spectrometer or high-precision camera can be used to collect the color (RGB) values ​​of color patches under different lighting conditions to ensure data accuracy. The collected RGB values ​​need to be calibrated and denoised to improve the accuracy of the mapping relationship.

[0078] For example, a color mapping relationship can be calculated using interpolation algorithms, linear regression, or other numerical optimization methods, based on a reference light source (a pre-defined combination of color temperature and illuminance) and color values ​​under different lighting conditions. This color mapping relationship describes the conversion rules from the reference light source color to colors under various lighting conditions.

[0079] In some embodiments, obtaining the color mapping relationship of each color block in the plurality of color blocks under different lighting conditions as described in step 110 can be achieved through the following steps 111 to 113, specifically:

[0080] Step 111: Obtain the color values ​​of the target device for multiple color blocks under different lighting conditions, wherein the lighting conditions include at least color temperature and illuminance;

[0081] Step 112: Set a reference light source, which includes at least a reference color temperature and a reference illuminance;

[0082] Step 113: Determine the color mapping relationship of each color block in the plurality of color blocks under different lighting conditions based on the reference light source and the color value.

[0083] Among these methods, target devices (such as high-precision spectrometers or image acquisition devices) can be used to measure W color patches under various color temperature and illuminance combinations (CCT). x LUX yThe RGB values ​​under (R, G, B) W |(CCT) x LUX y In this model, both x and y have values ​​greater than or equal to 1, indicating the existence of multiple color temperature and illuminance combinations. For example, the color temperature (CCT) typically ranges from 2300K to 6500K, covering a range of light sources from warm white to cool white. The illuminance (LUX) typically ranges from 0 lx to 10000 lx, but in practical applications, it can be extended to a wider range as needed, such as 100000 lx under summer sunlight. For each color patch, the RGB values ​​measured under different lighting conditions are recorded to form a color database for subsequent color mapping calculations.

[0084] Among them, a specific color temperature and illuminance combination (CCT) can be selected. p LUX q (This is used as a reference light source.) The color temperature (CCT) of the reference light source... p The typical value of p is 1, indicating that the first or a specific color temperature value is usually chosen as the reference. However, in practical applications, the reference color temperature can be determined based on the white point of the color gamut used by the algorithm that generates the virtual pixels; a typical value could be 6500K, representing the daylight color temperature. The illuminance LUX of the reference light source... q q can be 1 or other values. That is, the reference light source can contain only one type of illuminance or multiple types to adapt to different application scenarios. The specific value of the illuminance can be determined according to actual needs, such as the illuminance values ​​under typical environments like summer sunlight, cloudy outdoor conditions, indoor fluorescent lighting, indoor lighting at dusk, and streetlights at night. For example, it might be 100,000 lx under summer sunlight, 10,000 lx under cloudy outdoor conditions, 100 lx under indoor fluorescent lighting, 10 lx under indoor lighting at dusk, and 0.1 lx under streetlights at night.

[0085] Among them, it can be based on the reference light source (CCT) p LUX q The RGB values ​​(R, G, B) of each color block are shown below. W |(CCT) p LUX q ), and under different lighting conditions (CCT) x LUX y The RGB values ​​(R, G, B) of each color block are shown below. W |(CCT) x LUX y The color mapping relationship is calculated. The color mapping relationship describes the variation of color values ​​of color blocks from a reference light source condition to any lighting condition.

[0086] Color mapping relationships are typically represented by an m×n matrix, i.e., T m×n |(CCT p LUX q )->(CCT x LUX y This mapping matrix contains color conversion information from a reference light source to all possible lighting conditions. In practical applications, when it is necessary to adjust image colors to match the current ambient light, the desired target color mapping relationship can be quickly obtained by looking up or interpolating this color matrix.

[0087] Color can be represented using different color spaces or systems. Specifically, the CIE RGB color space can be used, a color space based on the characteristics of human color vision, where R, G, and B represent the primary color stimuli of red, green, and blue, respectively. Alternatively, the spectral tristimulus values ​​(CIE XYZ) can be used, a color space based on colorimetry principles, where X, Y, and Z represent the relative responses of color stimuli to three hypothetical primary colors (X for red, Y for green, and Z for blue) under specific conditions. Both representation methods have their advantages and disadvantages; the choice depends on the specific application scenario and requirements.

[0088] The luminance level of ambient light can be measured using different methods. Illuminance (LUX) is a commonly used unit of measurement, representing the luminous flux received per unit area, and can be used to describe the overall luminance level of ambient light. Luminance index (LUX Index), on the other hand, is a relative value used to describe the luminance performance of ambient light under certain specific conditions, such as the luminance distribution at different color temperatures. The choice between these two methods depends on the specific application scenario and requirements, as well as the complexity of the luminance characteristics to be described.

[0089] In some embodiments, the method further includes:

[0090] Based on the sensitivity of the human eye to color temperature and illuminance, the color temperature interval and illuminance interval of the color mapping relationship are configured.

[0091] In step 111, the interval step size of the color temperature CCT anchor points can be set to a larger value, such as 1000K, to reduce the complexity and time consumption of this step. This setting helps to reduce the number of mapping matrices required, thereby simplifying the processing flow. However, a larger color temperature interval may result in less smooth color transitions. To avoid obvious color changes or jumps observed by the human eye, after calculating the mapping matrix between different color temperatures in step 113, an interpolation method (such as linear interpolation) can be used to estimate a mapping matrix with a smaller color temperature interval. This method can maintain the smoothness and continuity of color transitions without significantly increasing the computational burden, achieving a more delicate color transition effect, while maintaining the controllability of computational resources to meet application requirements.

[0092] In step 111, the interval between two adjacent illuminance values ​​(LUX) can be set to a small value, so that when one illuminance value changes to an adjacent illuminance value, the human eye cannot observe a color jump. The illuminance interval can be a fixed value, or multiple intervals of different sizes can be selected depending on the ambient illuminance level. Since the minimum luminance difference that the human eye can distinguish is approximately 1% to 5%, a typical value for the illuminance interval can be 1% to 5% of the current illuminance. However, this increases the complexity and time consumption of this step.

[0093] To reduce computational complexity and time consumption while maintaining smooth color transitions, the interval step size between two adjacent illuminance values ​​(LUX) in step 111 can be set to a larger value, thereby reducing the complexity and time consumption of this step. The interval step size can be arbitrarily chosen. To avoid obvious color changes or jumps observed by the human eye, after calculating the mapping matrix between different illuminance values ​​in step 113, an interpolation method (such as linear interpolation) can be used to calculate a mapping matrix with smaller illuminance intervals. The accuracy of linear interpolation can meet application requirements. This reduces the amount of original data collection and computation while maintaining smooth color transitions.

[0094] When setting the reference light source in step 112, its illuminance interval can also be flexibly configured according to actual needs. If a more delicate color adjustment effect is required, a smaller illuminance interval step size can be set; if more computational efficiency is emphasized, a larger illuminance interval step size can be selected, and interpolation methods can be used for compensation in the subsequent process.

[0095] For example, the multiple illuminance intervals of a reference light source can be set to small values, so that when one illuminance changes to an adjacent illuminance, the human eye cannot observe a color jump. The illuminance interval step size can be a fixed value, or multiple step sizes of different sizes can be selected according to the ambient illuminance level. Since the minimum brightness difference that the human eye can distinguish is approximately 1% to 5%, a typical value for the illuminance step size interval can be 1% to 5% of the current illuminance. However, this increases the complexity and time consumption of this step.

[0096] For example, the multiple illuminance intervals of the reference light source can be set to larger values, thereby reducing the complexity and time consumption of this step. The interval step size can be arbitrarily chosen. To avoid the human eye observing color jumps, after calculating the mapping matrix between different illuminances in step 113, an interpolation method (such as linear interpolation) can be used to calculate a mapping matrix with smaller illuminance intervals. Among them, the accuracy of linear interpolation can meet the application requirements.

[0097] For example, the method for calculating the color mapping relationship (mapping matrix) in step 113 can be various linear optimization numerical methods or nonlinear optimization numerical methods. Specifically, these include, but are not limited to, the following methods:

[0098] Gradient descent method: Iteratively adjusts parameters to minimize the error function and gradually approaches the optimal solution.

[0099] Newton's method: It uses the second derivative information to accelerate convergence and is suitable for applications requiring high precision.

[0100] Gauss-Newton method: It combines the advantages of gradient descent and Newton's method, and is particularly suitable for handling nonlinear least squares problems.

[0101] The LM (Levonburg-Marquardt) method is an improved Gauss-Newton method that can converge quickly near local optima while maintaining global search capability.

[0102] For example, the m×n matrix in step 113 can take various forms depending on the actual application requirements:

[0103] 3×3 matrix: suitable for simple linear transformations, often used for color space conversion.

[0104] 3D LUTs (3D Lookup Tables): 3D LUTs are a more flexible color mapping representation method that can store the complete mapping relationship from the input color space to the output color space. Providing more complex non-linear mappings, 3D LUTs can capture finer color transitions and variations. Common typical sizes include 17×17×17 matrices, 33×33×33 matrices, and can also be 9×9×9 or 56×56×56 matrices. Larger 3D LUTs offer richer colors, but also increase computation time, power consumption, and storage space. However, overall, the computational resource consumption of methods using 3D LUTs is still relatively small compared to other image processing or computer vision methods.

[0105] Three 1D LUTs: These are 3×N matrices, each corresponding to a mapping curve for one of the three RGB channels. They are suitable for situations where each color channel needs independent adjustment. Each 1D LUT can be a 1D tone mapping curve (TM curve) for each of the three RGB channels. These curves are used to adjust the brightness distribution of each color channel to achieve specific visual effects or color correction.

[0106] When choosing the representation of a mapping matrix, factors such as computing resources, storage space, and the complexity of the mapping relationship need to be considered.

[0107] In some embodiments, the target device is a spectrometer, and acquiring the color values ​​of multiple color patches under different lighting conditions for the target device includes:

[0108] Under different light conditions with different color temperatures and illuminance combinations, the spectral tristimulus values ​​of W color blocks in the standard color card are collected by the spectrometer device, where W is a natural number greater than 1.

[0109] Convert the spectral tristimulus values ​​into linear color values;

[0110] Under different color temperature and illuminance combinations, the average of the linear color values ​​of Q pixels in each color block is taken as the color value of each color block under different color temperature and illuminance combinations, where Q is a natural number greater than 1.

[0111] For example, when the target device is a spectrometer, at different color temperatures (CCT) x ) and illuminance (LUX) q Under combined lighting conditions, the spectral tristimulus values ​​(CIE XYZ) of W color patches from a standard color chart are collected using a spectrometer, where W is a natural number greater than 1. These spectral tristimulus values ​​accurately reflect the color characteristics of each color patch under specific lighting conditions.

[0112] The obtained spectral tristimulus values ​​(CIE XYZ) can then be converted into linear RGB color values ​​using preset formulas (such as the CIE 1931 standard observer chromaticity matching function). This conversion ensures the consistency and accuracy of the color data, facilitating subsequent color mapping calculations. Commonly used conversion formulas include standard conversion matrices from XYZ to RGB, such as sRGB conversion matrices or conversion matrices for other specific color gamuts.

[0113] For each color patch, Q pixels (where Q is a natural number greater than 1) are selected under different lighting conditions, and the average of their linear RGB color values ​​is calculated. This average value serves as the representative color value for that color patch under those lighting conditions. This step helps reduce the error that may be caused by a single pixel and improves the reliability of the overall color measurement.

[0114] By averaging the color values ​​of multiple pixels, the impact of random noise can be effectively reduced, improving the stability and reliability of color measurement. Furthermore, the average value can better represent the color characteristics of the entire color patch.

[0115] Among them, spectrometers can capture color information with high precision, making them suitable for scenarios requiring highly accurate color measurement. By analyzing the spectral distribution of light, they provide more detailed color information than traditional RGB sensors.

[0116] In some embodiments, the target device is an image acquisition device, and acquiring the color values ​​of multiple color blocks under different lighting conditions for the target device includes:

[0117] Under different lighting conditions with varying color temperatures and illuminance combinations, the standard color chart is photographed using the image acquisition device to obtain nonlinear color values ​​for W color blocks, where W is a natural number greater than 1.

[0118] The nonlinear color values ​​are degamma corrected to obtain linear color values;

[0119] Under different color temperature and illuminance combinations, the average of the linear color values ​​of Q pixels in each color block is taken as the color value of each color block under different color temperature and illuminance combinations, where Q is a natural number greater than 1.

[0120] For example, when the target device is an image acquisition device, different color temperatures (CCT) can be used. x ) and illuminance (LUX) q Under combined lighting conditions, an image acquisition device (such as a camera) is used to photograph a standard color chart, obtaining nonlinear RGB color values ​​for W color blocks, where W is a natural number greater than 1. These nonlinear color values ​​reflect the original output of the device under specific lighting conditions.

[0121] Then, gamma correction is performed on the obtained non-linear RGB color values ​​to restore linear RGB color values. Gamma correction is used to eliminate the non-linear transformations introduced by the image acquisition device when displaying or recording colors, thereby ensuring the authenticity and consistency of color data.

[0122] Image acquisition devices (such as digital cameras) typically gamma-encode colors to optimize visual effects. Therefore, the directly acquired color values ​​are non-linear and require correction for accurate color processing. Degamma correction refers to the process of restoring gamma-encoded color values ​​to linear color values. This is because in practical applications, many color processing algorithms are based on linear color spaces, not gamma-encoded spaces.

[0123] For each color patch, Q pixels (where Q is a natural number greater than 1) are selected under different lighting conditions, and the average of their linear RGB color values ​​is calculated. This average value serves as the representative color value for that color patch under those lighting conditions. This process also helps improve the accuracy and stability of color measurement.

[0124] like Figure 4 The diagram illustrates a scene for obtaining the color mapping relationships of 24 color patches under different lighting conditions. A base color temperature is set as CCT_0. Four target color temperatures are CCT_-2, CCT_-1, CCT_1, and CCT_2. The illuminance of all light sources is set to LUX_0. The 24-color chart is illuminated using these five light sources. This yields the RGB values ​​of the 24 color patches under five different ambient lighting conditions. Four mapping matrices are then calculated using linear or nonlinear optimization methods. These four mapping matrices represent the color mapping relationships from the base light source to the other four target light sources. Then, the illuminance of the light sources is changed to LUX_1 and LUX_-1, and the above steps are repeated to obtain the color mapping relationships from the base light source to the other target light sources under two different illuminance levels. Finally, interpolation methods can be used to obtain color mapping relationships with smaller color temperature and illuminance intervals.

[0125] Step 120: Obtain an image to be processed that includes a virtual pixel set and a real-scene pixel set. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired by an image acquisition device.

[0126] For example, the set of virtual pixels in the image to be processed is generated using computer graphics rendering technology, rendering a 3D scene based on a set reference light source. The color and lighting effects of the virtual pixel set are consistent with the reference light source. For example, the calculated virtual pixel set can be text, icons, or images.

[0127] For example, the set of real-world pixels in the image to be processed is acquired from a real-world scene using an image acquisition device (such as a camera). The set of real-world pixels needs to be aligned or blended with the set of virtual pixels in spatial location.

[0128] For example, the calculated virtual pixel set is superimposed on the real pixel set to form an image to be processed that includes both the real pixel set and the calculated virtual pixel set. After obtaining the image to be processed, preprocessing steps such as denoising, sharpening, and color balancing can be performed to improve the effect of subsequent color adjustments and image quality.

[0129] In some embodiments, the method further includes: setting a flag channel for each virtual pixel in the virtual pixel set, the flag channel being used to indicate whether each virtual pixel needs to undergo pixel color adjustment processing.

[0130] For example, the decision to perform pixel color adjustment processing on all virtual pixels can be controlled by setting a global variable externally in the software. Alternatively, when calculating and generating the set of virtual pixels based on a reference light source, a flag channel can be added for each calculated virtual pixel. This flag channel controls whether the virtual pixel needs to undergo pixel color adjustment processing. For instance, when the flag channel is 1, the color of the virtual pixel will change with the ambient light, meaning the pixel color adjustment processing described in step 140 will be performed. If the flag channel is 0, the color of the virtual pixel will not change with the ambient light, meaning the pixel color adjustment processing described in step 140 will not be performed.

[0131] Step 130: Obtain the current ambient light conditions, including the color temperature and illuminance of the current ambient light.

[0132] In some embodiments, the current ambient light includes the current real-world ambient light, and the step of obtaining the lighting conditions of the current ambient light includes: obtaining the lighting conditions of the current real-world ambient light, including the color temperature and illuminance of the current real-world ambient light.

[0133] In many application scenarios, the current ambient light mainly refers to the ambient light of the current real-world scene, that is, the lighting conditions in the actual shooting scene of the camera.

[0134] In some embodiments, obtaining the current ambient light conditions of the real-world scene includes:

[0135] The color temperature of the ambient light in the current real-world scene is obtained through an automatic white balance algorithm, and the illuminance of the ambient light in the current real-world scene is obtained through an automatic exposure algorithm; or

[0136] The color temperature and illuminance of the ambient light in the current real-world scene are obtained using an ambient light sensor.

[0137] For example, an automatic white balance algorithm can be used to obtain the color temperature of the ambient light in a real-world scene. Automatic white balance algorithms typically utilize image data captured by an image sensor, analyzing the color distribution within the image to automatically adjust the camera's color temperature settings, ensuring accurate and natural color reproduction.

[0138] For example, automatic exposure algorithms can be used to obtain the ambient light level of a real-world scene. These algorithms automatically adjust camera exposure parameters, such as aperture, shutter speed, and ISO, based on the image's brightness level to achieve appropriate brightness and contrast.

[0139] For example, an ambient light sensor can be used directly to obtain the color temperature and illuminance of the ambient light in the current real-world scene. An ambient light sensor can directly measure the intensity and color temperature of light, providing accurate lighting condition data.

[0140] In some embodiments, the time interval for acquiring the color temperature and illuminance of the current real-world ambient light is an important parameter, determining the system's response speed and sensitivity to changes in ambient light. The choice of this time interval needs to be determined based on actual application requirements to ensure that the system can provide sufficiently accurate and real-time ambient light information while maintaining computational efficiency.

[0141] For example, in photography applications, the color temperature and illuminance of the current ambient light can be captured the instant the user presses the shutter button. This single measurement is sufficient to provide accurate lighting conditions for still images, facilitating subsequent color correction and image processing. Therefore, the time interval can be a single instant at each time a photo is taken.

[0142] For example, in video recording or live streaming, ambient light conditions may change with the scene, necessitating more frequent acquisition of color temperature and illuminance information. In such cases, the ambient light color temperature and illuminance can be measured at the moment each frame is captured to ensure color consistency and realism in the video content. This continuous measurement approach helps adapt to dynamically changing environments, especially in scenes with rapidly changing lighting conditions.

[0143] For example, in image preview or real-time image processing applications, such as video conferencing or augmented reality (AR) applications, the color and brightness of an image can be adjusted in real time as the user previews it. In this case, short time intervals (e.g., several times per second or more) can be set to update ambient light conditions in real time and dynamically adjust image processing parameters.

[0144] For example, acquiring ambient light color temperature and illuminance may involve the use of hardware sensors, which can affect the device's power consumption and processing performance. Therefore, in applications with limitations on power consumption and performance, a reasonable time interval can be set according to the actual situation to balance real-time performance and resource consumption.

[0145] In some applications, users may want to customize the time interval for ambient light measurements based on personal preferences or specific needs. For example, users may choose to measure the ambient light every time a photo is taken, or every few frames during video recording.

[0146] In some embodiments, the current ambient light further includes the current virtual scene ambient light, and the step of obtaining the lighting conditions of the current ambient light further includes: obtaining the lighting conditions of the current virtual scene ambient light, including the color temperature and illuminance of the current virtual scene ambient light.

[0147] In applications such as Augmented Reality (AR) and Virtual Reality (VR), in addition to the ambient light of the real scene, the lighting conditions of the virtual scene's ambient light also need to be considered. The color temperature and illuminance of the virtual scene's ambient light are usually calculated by the virtual environment rendering engine based on the scene settings and lighting model. Therefore, to obtain the current lighting conditions of the virtual scene's ambient light, it is necessary to access the relevant parameters or interfaces of the rendering engine.

[0148] In virtual scenes, lighting conditions are typically defined through scene settings, including the type, position, color (i.e., color temperature), and intensity (i.e., illuminance) of the light source. Therefore, the color temperature and illuminance of the current virtual scene's ambient light can be obtained by reading these settings.

[0149] In some cases, the virtual environment rendering engine automatically calculates lighting conditions based on factors such as objects, materials, light sources, and dynamic changes in the virtual scene (e.g., virtual time, weather changes). These calculation results can be obtained by calling the rendering engine's API or functions.

[0150] Step 140: Select a target color mapping relationship from the color mapping relationships that matches the current ambient light illumination conditions, and perform pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, so that the illumination effect of the adjusted pixels in the target image is consistent with the illumination effect of the current ambient light.

[0151] In some embodiments, selecting a target color mapping relationship from the color mapping relationships that matches the current ambient light illumination conditions, and performing pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, includes:

[0152] Select a first target color mapping relationship that matches the lighting conditions of the current real-world scene environment from the color mapping relationships;

[0153] Based on the first target color mapping relationship, the virtual pixels in the virtual pixel set of the image to be processed are subjected to pixel color adjustment processing to generate a first target image, so that the lighting effect of the adjusted virtual pixel set in the first target image is consistent with the lighting effect of the current real scene ambient light.

[0154] First, consider the color temperature and illuminance combination of the current real-world ambient light (CCT). i LUX j As input, select the first target color mapping relationship, T, from the pre-calculated and stored color mapping relationships. m×n |(CCT p LUX q )->(CCT i LUX j This process typically involves lookup or interpolation algorithms to ensure that the selected mapping accurately reflects the current ambient light conditions.

[0155] Then, the first target color mapping relationship is applied to the virtual pixel set I in the image to be processed. b Virtual pixel set I b Typically generated using computer graphics rendering techniques, its color and lighting effects are initially determined based on a set reference light source. According to the first target color mapping relationship, the color values ​​of the virtual pixels are adjusted to output a set of virtual pixels I that conforms to the lighting effects of the current real-world scene's ambient light. bf1 So that the adjusted virtual pixel set I bf1 The lighting effects are consistent with the ambient lighting effects of the current real-world scene.

[0156] After color adjustment, the adjusted virtual pixel set I bf1 The lighting effects have been matched to the ambient light of the current real-world scene. Then, these adjusted virtual pixel sets I... bf1 The virtual objects in the first target image are fused or combined with the real pixel set (if present) in the image to be processed to generate a first target image. The virtual objects in the first target image will be illuminated with lighting effects consistent with the ambient light of the current real scene, thereby enhancing the realism and immersion of the image.

[0157] In step 140, the mapping matrix representing the first target color mapping relationship is applied to the virtual pixel set I. b The method for each virtual pixel in the mapping matrix is ​​related to the category of the mapping matrix; that is, different mapping matrices require different methods of application.

[0158] For example, if the mapping matrix is ​​a 3x3 matrix, then the cross product of the mapping matrix is ​​used to calculate the virtual pixel set I. bThe RGB values ​​of each virtual pixel form a 3x1 vector.

[0159] For example, if the mapping matrix is ​​a 3D LUT, such as a 17x17x17 3D LUT, the method is to calculate the generated virtual pixel set I. b The RGB value of each virtual pixel is used as input. The resulting set of virtual pixels, I, is calculated using trilinear interpolation or tetrahedral interpolation methods of a common 3D LUT. This set of virtual pixels conforms to the lighting effects of the current real-world scene. bf1 In some embodiments, trilinear or tetrahedral interpolation can be encapsulated into a software algorithm library and computed using a Graphics Processing Unit (GPU). In other embodiments, trilinear or tetrahedral interpolation can be hardened into an ASIC chip module, such as part of a Display Processing Unit (DPU) or Image Single Processor (ISP), thereby improving real-time computation. ASIC chips are Application-Specific Integrated Circuit (ASIC) chip technologies designed for specific applications.

[0160] For example, if the mapping matrix is ​​a 3xN matrix, where N is the number of nodes in the 1D LUT, and each 1D LUT is a mapping table for one of the three RGB channels, then the virtual pixel set I... b The corresponding channel value is calculated by taking one of the RGB three channels of each virtual pixel as input and using the corresponding 1DLUT.

[0161] In some embodiments, selecting a target color mapping relationship from the color mapping relationships that matches the current ambient light illumination conditions, and performing pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, includes:

[0162] Select a second target color mapping relationship that matches the lighting conditions of the current virtual scene environment from the color mapping relationship;

[0163] Based on the second target color mapping relationship, the virtual pixels in the virtual pixel set of the image to be processed are subjected to pixel color adjustment processing to generate a second target image, so that the lighting effect of the adjusted virtual pixel set in the second target image is consistent with the lighting effect of the current virtual scene ambient light.

[0164] First, based on the current virtual scene ambient light color temperature and illuminance combination (CCT)u LUX v As input, a second target color mapping relationship, T, is selected from the pre-calculated and stored color mapping relationships to match it. m×n |(CCT p LUX q )->(CCT u LUX v This process typically involves a lookup or interpolation algorithm to ensure that the mapping relationship that best matches the current virtual scene's ambient lighting conditions is found.

[0165] Then, the second target color mapping relationship is applied to the virtual pixel set I in the image to be processed. b Virtual pixel set I b Typically generated using computer graphics rendering techniques, its color and lighting effects are initially determined based on a set reference light source. According to a second target color mapping relationship, the color values ​​of the virtual pixels are adjusted to output a set of virtual pixels I that conforms to the lighting effects of the current virtual scene's ambient light. bf2 So that the adjusted virtual pixel set I bf2 The lighting effects are consistent with the ambient lighting effects of the current virtual scene.

[0166] After color adjustment, the adjusted virtual pixel set I bf2 The virtual object is then fused or combined with other elements in the image to be processed (such as possible real-world pixels, background images, or other virtual elements) to generate a second target image. In the second target image, the lighting effects of the virtual objects will perfectly match the lighting conditions of the current virtual scene environment, providing users with an immersive and realistic virtual experience.

[0167] In step 140, the mapping matrix representing the second target color mapping relationship is applied to the virtual pixel set I. b The method for each virtual pixel in the mapping matrix is ​​related to the category of the mapping matrix; that is, different mapping matrices require different methods of application.

[0168] For example, if the mapping matrix is ​​a 3x3 matrix, then the cross product of the mapping matrix is ​​used to calculate the virtual pixel set I. b The RGB values ​​of each virtual pixel form a 3x1 vector.

[0169] For example, if the mapping matrix is ​​a 3D LUT, such as a 17x17x17 3D LUT, the method is to calculate the generated virtual pixel set I. bThe RGB value of each virtual pixel is used as input. Through trilinear interpolation or tetrahedral interpolation using a common 3D LUT, the RGB values ​​of the applied virtual pixels are calculated, resulting in a set I of virtual pixels that conforms to the lighting effects of the current virtual scene's ambient light. bf2 In some embodiments, trilinear or tetrahedral interpolation can be encapsulated into a software algorithm library and computed using a GPU. In other embodiments, trilinear or tetrahedral interpolation can be hardened into an ASIC chip module, such as part of a DPU or ISP, thereby improving the real-time performance of the computation.

[0170] For example, if the mapping matrix is ​​a 3xN matrix, where N is the number of nodes in the 1D LUT, and each 1D LUT is a mapping table for one of the three RGB channels, then the virtual pixel set I... b The corresponding channel value is calculated by taking one of the RGB three channels of each virtual pixel as input and using the corresponding 1DLUT.

[0171] like Figure 5 As shown, the lighting conditions of the real scene's ambient light are used as the global lighting conditions to perform color adjustments on the virtual pixels calculated from the image to be processed. Figure 5 The image to be processed provided includes a set of real-world pixels captured by an image acquisition device (such as a camera), representing a white porcelain plate containing five pale yellow lemons, and a set of virtual pixels generated by a computer device, representing a virtual pale yellow lemon located in the center.

[0172] The color mapping relationship under different ambient light conditions can be represented using a 3D LUT. The base color temperature is determined to be 6500K based on the white point of the virtual pixel's color gamut. A series of pixels are calculated and generated in the cache.

[0173] For example, obtain the ambient light conditions of the current real-world scene, with a color temperature of 4600K and an illuminance of 100lx. Then, retrieve the corresponding 3D LUTs for 4600K and 100lx and apply them to the calculated virtual pixels. This ensures that the adjusted virtual pixels are color-coordinated with the real-world pixels captured by the image acquisition device.

[0174] For example, continuously acquire the color temperature and illuminance of the current real-world ambient light. If the ambient light changes to lighting condition 2, the color temperature becomes 3000K and the illuminance becomes 60lx. Then, the corresponding 3D LUTs for 3000K and 60lx are retrieved and applied to calculate the generated virtual pixels, ensuring that the adjusted virtual pixels maintain color consistency with the real-world pixels captured by the image acquisition device under 3000K conditions.

[0175] In some embodiments, selecting a target color mapping relationship from the color mapping relationships that matches the current ambient light illumination conditions, and performing pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, includes:

[0176] Select a third target color mapping relationship from the color mapping relationships that matches the lighting conditions of the current real scene ambient light and the lighting conditions of the current virtual scene ambient light;

[0177] Based on the third target color mapping relationship, the pixel colors of the real-scene pixels in the real-scene pixel set of the image to be processed are adjusted to generate a third target image, so that the lighting effect of the adjusted real-scene pixel set in the third target image is consistent with the lighting effect of the current virtual scene ambient light.

[0178] First, consider the color temperature and illuminance combination of the current real-world ambient light (CCT). i LUX j ) and the color temperature and illuminance combination of the current virtual scene ambient light (CCT) u LUX v As input, a third target color mapping relationship, T, is selected from the pre-calculated and stored color mapping relationships to match. m×n |(CCT i LUX j )->(CCT u LUX v This process typically involves lookup or interpolation algorithms to ensure that the found mapping satisfies the lighting requirements of both the real-world and virtual scenes.

[0179] Then, the third target color mapping relationship is applied to the set of real-world pixels I in the image to be processed. c Real-world pixel set I c These images are acquired from the real world through image acquisition devices such as cameras, and contain lighting and color information of the real scene. Based on a third target color mapping relationship, the color values ​​of each real-world pixel are finely adjusted, including brightness, saturation, and hue, to ensure that the adjusted set of real-world pixels I... cf The lighting effects are consistent with the ambient light of the current virtual scene.

[0180] After the above color adjustment process, the adjusted real-world pixel set I cfThe virtual pixel set (if present) and other image elements are fused or combined to generate a third target image. In the third target image, the real scene and the virtual scene achieve a high degree of consistency and seamless integration in lighting effects, providing users with an immersive visual experience. This consistency is reflected not only in the global lighting effects but also in the handling of details such as local shadows, reflections, and highlights.

[0181] In step 140, the mapping matrix representing the color mapping relationship of the third target is applied to the real-scene pixel set I. c The method for each real-world pixel is related to the category of the mapping matrix; that is, different mapping matrices require different methods of application.

[0182] For example, if the mapping matrix is ​​a 3x3 matrix, then the cross product of the mapping matrix is ​​used to calculate the real-scene pixel set I. c The 3x1 vector is composed of the RGB values ​​of each real-world pixel.

[0183] For example, if the mapping matrix is ​​a 3D LUT, such as a 17x17x17 3D LUT, the method of operation is to map the set of real-world pixels I... c The RGB value of each real-world pixel is used as input. Through common 3D LUT trilinear interpolation or tetrahedral interpolation, the applied RGB values ​​of the real-world pixels are calculated, resulting in a set I of real-world pixels that conforms to the lighting effects of the current virtual scene's ambient light. cf In some embodiments, trilinear or tetrahedral interpolation can be encapsulated into a software algorithm library and computed using a GPU. In other embodiments, trilinear or tetrahedral interpolation can be hardened into an ASIC chip module, such as part of a DPU or ISP, thereby improving the real-time performance of the computation.

[0184] For example, if the mapping matrix is ​​a 3xN matrix, where N is the number of nodes in the 1D LUT, and each 1D LUT is a mapping table for one of the three RGB channels, then the set of real-world pixels I... c The corresponding channel value is calculated by taking one of the RGB three channels of each real-world pixel as input and using the corresponding 1DLUT.

[0185] like Figure 6 As shown, the lighting conditions of the real scene's ambient light are used as the global lighting conditions to perform color adjustments on the virtual pixels calculated from the image to be processed. Figure 5 The image to be processed provided includes a set of real-world pixels captured by an image acquisition device (such as a camera), representing a white porcelain plate containing five pale yellow lemons, and a set of virtual pixels generated by a computer device, representing a virtual pale yellow lemon located in the center.

[0186] The color mapping relationship under different ambient light conditions can be represented using a 3D LUT. A series of real-world pixels captured by an image acquisition device (such as a camera) are obtained, forming the first frame image.

[0187] For example, obtain the current ambient light conditions of the real scene (5500K, 100lx) and the current ambient light conditions of the virtual scene (4600K, 100lx). Then, retrieve the mapping matrix T. m×n |(5500->4600,100) applies to the real-world pixels in the first frame of the image.

[0188] For example, the color temperature of the reference light source is determined to be 6500K based on the calculated white point of the virtual pixels. A series of pixels are calculated and generated in the cache. The mapping matrix T is retrieved. m×n |(6500->4600,100) is applied to the calculated virtual pixels and rendered onto the first frame image.

[0189] For example, continuously acquire the color temperature and illuminance of the current virtual scene ambient light. If the current virtual scene ambient light changes to lighting condition 2, where the color temperature changes to 3000K and the illuminance changes to 60lx, then retrieve the mapping matrix T. m×n |(5500→3000,100→60) applies to the real-world pixels of the current frame. The mapping matrix Tm×n|(6500->3000,100->60) is retrieved and applied to the virtual pixels of the current frame.

[0190] The embodiments of this application have low computation time and low real-time computation load, making them suitable for mobile devices and high frame rate applications. Several 3D LUTs or three 1D LUTs are pre-calculated and statically stored in the mobile device. During real-time application, the color value of each pixel as it changes with ambient light is quickly obtained using a lookup table, without needing to recalculate the changes of each pixel under different lighting conditions based on complex formulas or neural networks. The more nodes in the LUT, the shorter the computation time.

[0191] The mapping matrix in this embodiment is applied to pixel calculations and can be integrated into an ASIC chip module, further reducing computation time. Only specific neural network models can be computed on an NPU. Other types of neural networks can only perform calculations on a GPU, which takes longer than the computation time described in this embodiment.

[0192] The embodiments of this application require less memory space because LUT tables are more compact than neural network models. Even storing a large number of LUTs requires relatively little memory, while large deep learning models require much more memory resources. For example, storing 670 3D LUTs of size 33x33x33 double type requires approximately 60MB of memory. 1000 3D LUTs of size 17x17x17 unsigned short type occupy 30MB of memory. The size of a neural network depends on the number of its parameters, i.e., the number of weights and biases in the network. For large deep learning models with good illumination estimation performance, the large number of parameters requires 500MB or even more of memory. Neural networks that combine illumination estimation and pixel generation calculation are even larger in size.

[0193] This application's embodiment can obtain a mapping matrix using a color chart and multiple light sources, requiring minimal effort and achievable in a laboratory environment. In contrast, data-driven, learning-based methods not only require arduous dataset acquisition and annotation but also time-consuming model training. This application's embodiment reduces the workload of dataset acquisition, annotation, and model training.

[0194] This application's embodiments are applicable to high frame rate video applications, such as 90Hz video see-through (VST) videos for Mixed Reality (MR) devices and Augmented Reality (AR) videos of high-speed motion scenes. Because the computational load is low and ambient light conditions can be directly obtained using an ambient light sensor (ALS), without relying on statistical values ​​from previous frames to calculate color temperature and brightness, this method does not suffer from frame delay issues and can be applied to ultra-high frame rate video applications.

[0195] In this embodiment, the neural network for calculating virtual pixels only needs to generate pixels based on a reference color temperature. This significantly reduces the size of the neural network and the computational resource consumption in practical applications, making it more suitable for practical applications and deployment.

[0196] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0197] This application embodiment obtains the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on a reference light source and the color values ​​of multiple color blocks under different lighting conditions. It acquires an image to be processed, including a virtual pixel set and a real-scene pixel set. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired through an image acquisition device. It obtains the current ambient light lighting conditions, including at least the color temperature and illuminance of the current ambient light. It selects a target color mapping relationship from the color mapping relationships that matches the current ambient light lighting conditions, and performs pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image. This ensures that the lighting effect of the adjusted pixels in the target image is consistent with the lighting effect of the current ambient light. This application embodiment can automatically adjust the pixels of the image to be processed to generate a target image when the current ambient light lighting conditions (including color temperature and illuminance) change, ensuring that the lighting effect of the adjusted pixels in the target image is consistent with the lighting effect of the current ambient light. This avoids the color mismatch problem that may occur between the virtual pixels and real-scene pixels calculated in traditional methods. By pre-obtaining the color mapping relationships of multiple color blocks under different lighting conditions and adjusting pixel colors based on these mapping relationships, the complex calculations required for real-time calculation of the changes of each pixel under different lighting conditions are avoided, effectively reducing computational complexity. By accurately matching and applying the target color mapping relationship, pixel-level color adjustments can be made to the image, more accurately simulating changes in lighting conditions in real-world environments, thereby significantly improving the realism and visual quality of the image.

[0198] To facilitate better implementation of the image processing method of this application embodiment, this application embodiment also provides an image processing apparatus. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The image processing apparatus 200 may include:

[0199] The first acquisition unit 210 is used to acquire the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on the reference light source and the color values ​​of multiple color blocks under different lighting conditions.

[0200] The second acquisition unit 220 acquires an image to be processed that includes a virtual pixel set and a real-scene pixel set. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired by an image acquisition device.

[0201] The third acquisition unit 230 is used to acquire the current ambient light illumination conditions, including the color temperature and illuminance of the current ambient light.

[0202] The processing unit 240 is configured to select a target color mapping relationship from the color mapping relationship that matches the current ambient light illumination conditions, and perform pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, such that the illumination effect of the adjusted pixels in the target image is consistent with the illumination effect of the current ambient light.

[0203] In some embodiments, the current ambient light includes the current real-world ambient light, and the third acquisition unit 230 can be used to: acquire the lighting conditions of the current real-world ambient light, including the color temperature and illuminance of the current real-world ambient light.

[0204] In some embodiments, the processing unit 240 may be used to: select a first target color mapping relationship from the color mapping relationship that matches the lighting conditions of the current real-world scene ambient light; perform pixel color adjustment processing on the virtual pixels in the virtual pixel set of the image to be processed based on the first target color mapping relationship to generate a first target image, such that the lighting effect of the adjusted virtual pixel set in the first target image is consistent with the lighting effect of the current real-world scene ambient light.

[0205] In some embodiments, the current ambient light further includes the current virtual scene ambient light, and the third acquisition unit 230 can also be used to: acquire the lighting conditions of the current virtual scene ambient light, including the color temperature and illuminance of the current virtual scene ambient light.

[0206] In some embodiments, the processing unit 240 may be used to: select a second target color mapping relationship from the color mapping relationship that matches the lighting conditions of the current virtual scene ambient light; perform pixel color adjustment processing on the virtual pixels in the virtual pixel set of the image to be processed based on the second target color mapping relationship to generate a second target image, such that the lighting effect of the adjusted virtual pixel set in the second target image is consistent with the lighting effect of the current virtual scene ambient light.

[0207] In some embodiments, the processing unit 240 may be configured to: select a third target color mapping relationship from the color mapping relationships that matches the lighting conditions of the current real-world scene ambient light and the lighting conditions of the current virtual scene ambient light; perform pixel color adjustment processing on the real-world pixels in the real-world pixel set of the image to be processed based on the third target color mapping relationship to generate a third target image, such that the lighting effect of the adjusted real-world pixel set in the third target image is consistent with the lighting effect of the current virtual scene ambient light.

[0208] In some embodiments, when acquiring the lighting conditions of the current real-world ambient light, the third acquisition unit 230 may be used to: acquire the color temperature of the current real-world ambient light through an automatic white balance algorithm, and acquire the illuminance of the current real-world ambient light through an automatic exposure algorithm; or acquire the color temperature and illuminance of the current real-world ambient light through an ambient light sensor.

[0209] In some embodiments, the first acquisition unit 210 may be used to: acquire color values ​​of multiple color blocks for a target device under different lighting conditions, the lighting conditions including at least color temperature and illuminance; set a reference light source, the reference light source including at least reference color temperature and reference illuminance; and determine the color mapping relationship of each color block among the multiple color blocks under different lighting conditions based on the reference light source and the color values.

[0210] In some embodiments, the target device is a spectrometer device, and the first acquisition unit 210, when acquiring the color values ​​of multiple color blocks under different lighting conditions, can be used to: acquire the spectral tristimulus values ​​of W color blocks in a standard color chart through the spectrometer device under different color temperature and illuminance combinations, where W is a natural number greater than 1; convert the spectral tristimulus values ​​into linear color values; and, under different color temperature and illuminance combinations, take the average value of the linear color values ​​of Q pixels in each color block as the color value of each color block under different color temperature and illuminance combinations, where Q is a natural number greater than 1.

[0211] In some embodiments, the target device is an image acquisition device, and the first acquisition unit 210, when acquiring the color values ​​of multiple color blocks under different lighting conditions, can be used to: capture a standard color card through the image acquisition device under different color temperature and illuminance combinations to obtain nonlinear color values ​​of W color blocks, where W is a natural number greater than 1; perform gamma correction on the nonlinear color values ​​to obtain linear color values; and take the average value of the linear color values ​​of Q pixels in each color block as the color value of each color block under different color temperature and illuminance combinations, where Q is a natural number greater than 1.

[0212] In some embodiments, the second acquisition unit 220 can also be used to: set a flag bit channel for each virtual pixel in the virtual pixel set, the flag bit channel being used to indicate whether each virtual pixel needs to undergo pixel color adjustment processing.

[0213] In some embodiments, the first acquisition unit 210 may also be used to: configure the color temperature interval and illuminance interval of the color mapping relationship according to the sensitivity of the human eye to color temperature and illuminance.

[0214] The image processing device 200 can be integrated into a terminal device or server that has storage and a processor and thus computing power, or the image processing device 200 can be the terminal device or server.

[0215] In some embodiments, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0216] In some embodiments, such as Figure 8 As shown, Figure 8 This is another schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 300 further includes a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, and a computer program stored on the memory 302 and executable on the processor. The processor 301 and the memory 302 are electrically connected. Those skilled in the art will understand that the computer device structure shown in the figures does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0217] The processor 301 is the control center of the computer device 300. It connects various parts of the computer device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it performs various functions of the computer device 300 and processes data, thereby performing overall monitoring of the computer device 300.

[0218] Optionally, the computer device 300 can be a terminal device, and the processor 301 can call software programs and modules stored in the memory 302 to perform the following operations:

[0219] The process involves: acquiring the color mapping relationships of multiple color blocks under different lighting conditions, whereby the color mapping relationships are determined based on a reference light source and the color values ​​of the multiple color blocks under different lighting conditions; acquiring an image to be processed containing a virtual pixel set and a real-scene pixel set, whereby the virtual pixel set is generated based on the reference light source and the real-scene pixel set is acquired through an image acquisition device; acquiring the current ambient light lighting conditions, whereby the lighting conditions include at least the color temperature and illuminance of the current ambient light; selecting a target color mapping relationship from the color mapping relationships that matches the current ambient light lighting conditions, and performing pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, such that the lighting effect of the adjusted pixels in the target image is consistent with the lighting effect of the current ambient light.

[0220] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0221] In some embodiments, such as Figure 8 As shown, the computer device 300 also includes a display unit 303, a radio frequency circuit 304, an audio circuit 305, an input unit 306, and a power supply 307. The processor 301 is electrically connected to the display unit 303, the radio frequency circuit 304, the audio circuit 305, the input unit 306, and the power supply 307. Those skilled in the art will understand that... Figure 8 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0222] Display unit 303 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of computer devices. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. Display unit 303 may include a display panel and a touch panel.

[0223] The radio frequency circuit 304 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0224] Audio circuitry 305 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 305 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 305, converted back into audio data, and then processed by processor 301 before being transmitted via radio frequency circuitry 304 to, for example, another computer device, or output to memory for further processing. Audio circuitry 305 may also include an earphone jack to facilitate communication between peripheral headphones and computer devices.

[0225] The input unit 306 can be used to receive input numbers, characters, or object feature information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0226] Power supply 307 is used to supply power to the various components of computer device 300.

[0227] although Figure 8As not shown in the diagram, computer equipment 300 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0228] In some embodiments, this application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a terminal device or a server, and the computer program causes the terminal device or server to execute the corresponding processes in the image processing methods of the embodiments of this application; for brevity, these will not be elaborated further here.

[0229] In some embodiments, this application also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the corresponding processes in the image processing method of this application embodiment. For brevity, these details are not elaborated here.

[0230] This application also provides a computer program, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the corresponding processes in the image processing method of the embodiments of this application. For simplicity, these details will not be elaborated further here.

[0231] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0232] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0233] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0234] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0236] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0237] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0238] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0239] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on the reference light source and the color values ​​of multiple color blocks under different lighting conditions. An image to be processed, comprising a virtual pixel set and a real-scene pixel set, is acquired. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired by an image acquisition device. Obtain the current ambient light conditions, including the color temperature and illuminance of the current ambient light; Select a target color mapping relationship from the color mapping relationships that matches the current ambient light illumination conditions, and perform pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, so that the illumination effect of the adjusted pixels in the target image is consistent with the illumination effect of the current ambient light.

2. The image processing method as described in claim 1, characterized in that, The current ambient light includes the ambient light of the current real-world scene, and the lighting conditions for obtaining the current ambient light include: Obtain the lighting conditions of the current real-world ambient light, including the color temperature and illuminance of the current real-world ambient light.

3. The image processing method as described in claim 2, characterized in that, The step of selecting a target color mapping relationship from the color mapping relationships that matches the current ambient light conditions, and performing pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, includes: Select a first target color mapping relationship that matches the lighting conditions of the current real-world scene environment from the color mapping relationships; Based on the first target color mapping relationship, the virtual pixels in the virtual pixel set of the image to be processed are subjected to pixel color adjustment processing to generate a first target image, so that the lighting effect of the adjusted virtual pixel set in the first target image is consistent with the lighting effect of the current real scene ambient light.

4. The image processing method as described in claim 2, characterized in that, The current ambient light also includes the current virtual scene ambient light, and the step of obtaining the lighting conditions for the current ambient light further includes: Obtain the lighting conditions of the current virtual scene ambient light, including the color temperature and illuminance of the current virtual scene ambient light.

5. The image processing method as described in claim 4, characterized in that, The step of selecting a target color mapping relationship from the color mapping relationships that matches the current ambient light conditions, and performing pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, includes: Select a second target color mapping relationship that matches the lighting conditions of the current virtual scene environment from the color mapping relationship; Based on the second target color mapping relationship, the virtual pixels in the virtual pixel set of the image to be processed are subjected to pixel color adjustment processing to generate a second target image, so that the lighting effect of the adjusted virtual pixel set in the second target image is consistent with the lighting effect of the current virtual scene ambient light.

6. The image processing method as described in claim 4, characterized in that, The step of selecting a target color mapping relationship from the color mapping relationships that matches the current ambient light conditions, and performing pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, includes: Select a third target color mapping relationship from the color mapping relationships that matches the lighting conditions of the current real scene ambient light and the lighting conditions of the current virtual scene ambient light; Based on the third target color mapping relationship, the pixel colors of the real-scene pixels in the real-scene pixel set of the image to be processed are adjusted to generate a third target image, so that the lighting effect of the adjusted real-scene pixel set in the third target image is consistent with the lighting effect of the current virtual scene ambient light.

7. The image processing method as described in claim 2, characterized in that, The acquisition of the current ambient light conditions includes: The color temperature of the ambient light in the current real-world scene is obtained through an automatic white balance algorithm, and the illuminance of the ambient light in the current real-world scene is obtained through an automatic exposure algorithm; or The color temperature and illuminance of the ambient light in the current real-world scene are obtained using an ambient light sensor.

8. The image processing method according to any one of claims 1-7, characterized in that, The step of obtaining the color mapping relationship of each color block in multiple color blocks under different lighting conditions includes: Obtain the color values ​​of the target device for multiple color blocks under different lighting conditions, wherein the lighting conditions include at least color temperature and illuminance; A reference light source is set, which includes at least a reference color temperature and a reference illuminance; Based on the reference light source and the color value, the color mapping relationship of each color block among the plurality of color blocks under different lighting conditions is determined.

9. The image processing method as described in claim 8, characterized in that, The target device is a spectrometer, and acquiring the color values ​​of multiple color patches under different lighting conditions for the target device includes: Under different light conditions with different color temperatures and illuminance combinations, the spectral tristimulus values ​​of W color blocks in the standard color card are collected by the spectrometer device, where W is a natural number greater than 1. Convert the spectral tristimulus values ​​into linear color values; Under different color temperature and illuminance combinations, the average of the linear color values ​​of Q pixels in each color block is taken as the color value of each color block under different color temperature and illuminance combinations, where Q is a natural number greater than 1.

10. The image processing method as described in claim 8, characterized in that, The target device is an image acquisition device, and the acquisition of color values ​​of multiple color blocks under different lighting conditions by the target device includes: Under different lighting conditions with varying color temperatures and illuminance combinations, the standard color chart is photographed using the image acquisition device to obtain nonlinear color values ​​for W color blocks, where W is a natural number greater than 1. The nonlinear color values ​​are degamma corrected to obtain linear color values; Under different color temperature and illuminance combinations, the average of the linear color values ​​of Q pixels in each color block is taken as the color value of each color block under different color temperature and illuminance combinations, where Q is a natural number greater than 1.

11. The image processing method according to any one of claims 1-7, characterized in that, The method further includes: A flag channel is set for each virtual pixel in the virtual pixel set. The flag channel is used to indicate whether each virtual pixel needs to undergo pixel color adjustment processing.

12. The image processing method according to any one of claims 1-7, characterized in that, The method further includes: Based on the sensitivity of the human eye to color temperature and illuminance, the color temperature interval and illuminance interval of the color mapping relationship are configured.

13. An image processing apparatus, characterized in that, The device includes: The first acquisition unit is used to acquire the color mapping relationship of each color block in multiple color blocks under different lighting conditions. The color mapping relationship is determined based on the reference light source and the color values ​​of the multiple color blocks under different lighting conditions. The second acquisition unit acquires an image to be processed that includes a virtual pixel set and a real-scene pixel set. The virtual pixel set is generated based on the reference light source, and the real-scene pixel set is acquired by an image acquisition device. The third acquisition unit is used to acquire the current ambient light illumination conditions, wherein the illumination conditions include at least the color temperature and illuminance of the current ambient light. The processing unit is configured to select a target color mapping relationship from the color mapping relationship that matches the current ambient light illumination conditions, and perform pixel color adjustment processing on the image to be processed based on the target color mapping relationship to generate a target image, such that the illumination effect of the adjusted pixels in the target image is consistent with the illumination effect of the current ambient light.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the image processing method as described in any one of claims 1-12.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the image processing method according to any one of claims 1-12 by calling the computer program stored in the memory.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method according to any one of claims 1-12.

Images