A color correction method

Abstract

This invention proposes a color correction method, including: disabling tone mapping in the UE4 engine; disabling automatic exposure in the UE4 engine to avoid automatic exposure interfering with color correction; determining the camera's white balance and exposure, and prohibiting modification of the camera's white balance and exposure parameters during actual shooting; removing lights from the UE4 scene to allow the screen to display self-illumination, so that the colors driven by the LED are the pure standard colors on the color chart; turning off lights on the LED screen to prevent ambient light from affecting the colors of the LED screen; using a support vector machine (SVM) to calculate the color restoration network; calculating the viewing light intensity curve for each color; extracting the light intensity of the returned color; and performing color restoration using the SVM.

Description

Technical Field

[0001] This invention proposes a data processing method in the field of virtual reality, specifically a color correction method. Background Technology

[0002] XR (Extended Reality) is a technology that emerged after AR (Augmented Reality) and MR (Mixed Reality). Early XR virtual reality technology used green screen keying, which had the advantage of inexpensive equipment but the disadvantage of being unfriendly to actors' performances. The actors were not facing a real scene, but rather various green walls, which severely tested their imagination; expressing a wide range of emotions was a huge challenge for the average person. In post-production, large areas of green would also give characters and props a faint green halo, making post-processing difficult. As an improvement, today's popular virtual reality technology is MR immersive LED screen display technology. It uses several large LED screens to form several screens, using CAVE spatial projection technology to render a 3D scene onto the large screens. The presenter or actor stands inside the LED screen, and from the camera's perspective, the person in the image appears to be standing in a virtual scene. For the presenter (or actor), what they see is what they get; the director can directly compose the shot, and no further post-production processing is needed after filming.

[0003] However, unlike green screen technology, this immersive LED large-screen display technology is not XR. XR requires the camera's field of view to extend beyond the screen wall while maintaining a consistent virtual scene with uniform perspective and color. The ideal is beautiful, but the reality is harsh. Currently, there are very few companies worldwide that can perfectly achieve this technology. The main challenges are:

[0004] 1) Camera positioning devices for this type of infinite blue box XR require millimeter-level precision, making them more expensive. Furthermore, positioning devices typically don't directly provide the camera's location in virtual space; this requires further development.

[0005] 2) The camera's intrinsic parameters and distortion affect the matching of virtual and real images.

[0006] First, the virtual camera's field of view (FOV) must match the focal length of the real camera. For zoom cameras, smooth image matching is required when the person zooms in. This necessitates a time-synchronized black screen device and, more importantly, a high-precision calibration method to ensure that the mapping error between focal length and field of view is less than 0.1 degrees.

[0007] Secondly, real cameras all have radial distortion, so distortion needs to be artificially added to the images rendered by the virtual camera. For zoom cameras, the distortion coefficient is different at each focal length, so a certain degree of distortion calibration is required. Similar to the FOV calibration above, each camera must be calibrated across the entire focal length range.

[0008] 3) The most difficult part is achieving color consistency.

[0009] There is a lack of effective color correction methods in the current technology for the field of virtual reality. Summary of the Invention

[0010] To address the lack of effective color correction methods in existing XR technologies for the virtual reality field, the purpose of this invention is to provide a data processing method for the virtual reality field, specifically a color correction method that can effectively correct colors in the virtual reality field.

[0011] To achieve the above objectives, this invention proposes a color correction method applied to the UE4 engine to process captured data; the method includes:

[0012] Step 1: Disable tone mapping in the UE4 engine;

[0013] If you need to obtain the SceneColor parameter, you can do so through the Post Process Material in the UE4 engine;

[0014] If you want to obtain the SceneColorBeforeTonemap parameter, you can change the Phase of PostProcess Material in the UE4 engine to Before Toonmapping.

[0015] Step 2: Disable automatic exposure in UE4 engine to avoid automatic exposure affecting color correction;

[0016] Step 3: Determine the camera's white balance and exposure, and do not modify the camera's white balance and exposure parameters during actual shooting;

[0017] Step 4: Remove the lights from the UE4 scene and make the screen display self-illuminating so that the colors driven by the LEDs are the pure standard colors on the color chart;

[0018] Step 5: At the LED screen site, turn off the lights to prevent ambient light from affecting the color of the LED screen.

[0019] Step 6: Determine a scene consistent with the LED screen space, and configure the CAVE screen of the UE4 cluster to match the size of the actual LED screen; so that the image displayed on each LED screen is the material of the corresponding panel, regardless of the camera's position; set a self-illuminating material for each panel to construct a monochrome LED virtual environment; capture this monochrome panel with the camera to obtain a monochrome image, which contains information on the primary and secondary colors of the color chart, as well as the brightness changes of the LED pixels under different viewing angles;

[0020] Step 7: Use a Support Vector Machine (SVM) to compute the color restoration network. The SVM network used for training includes an input layer, an intermediate layer, and an output layer. The input layer includes R-image, G-image, and B-image, corresponding to the captured red, green, and blue primary color data. The n kernels of the intermediate layer can be used to obtain the output layer R-original, G-original, and B-original, corresponding to the original red, green, and blue primary color data. For each color, the primary color and return color of the 0-degree viewing angle pixel are taken. The 0-degree viewing angle data of all colors are collected together to form a sample library, and the artificial intelligence network for color restoration is trained. The return color is obtained from the LED screen projection and camera capture.

[0021] For each point in the image captured by the acquisition card, the corresponding RGB color of the UE4 engine-rendered image is calculated, and the input is the RGB of the acquisition card. The number of kernels is preset. If the training results are still unsatisfactory after a considerable number of training iterations, the number of kernels can be increased, that is, the dimension of the high-dimensional space can be increased, until it truly meets the requirements of the linear mapping of the output layer. The kernel function is the Gaussian kernel function as follows.

[0022]

[0023] Where x represents the low-dimensional space coordinates; x' represents the core coordinates of the high-dimensional space kernel in the low-dimensional space; and σ represents the kernel radius of the high-dimensional space kernel.

[0024] Step 8: Calculate the luminous intensity curve of each color: Extract the luminous intensity of the returned color, which is obtained by the LED screen projection and camera; compare the luminous intensity of the returned color with the luminous intensity of the 0-degree viewing angle to extract the contrast curve; in this process, multiple 3D points are pre-determined using the position of the CAVE screen and then converted into screen points.

[0025] The samples can be made into a table of light intensity percentages. At runtime, adjacent data interpolation can be used. This table can be passed to the GPUShader via the Texture parameter, and GUPShader has an efficient interpolation algorithm.

[0026] Step 9: Use the SVM color restoration network to restore the image from the capture card to the UE4 engine rendering image; integrate the color correction matrix and LED light intensity curve into the UE4 graphics card rendering, combine the virtual and real in the compositing stage, and use the high performance of the graphics card to obtain the real-time performance required for immersive XR color correction.

[0027] Step 7 further includes:

[0028] From the original 3D scene rendered image to the actual captured image on the acquisition card, the process involves projecting the 3D scene onto the LED screen, then capturing the LED screen with a camera and transferring the image to the acquisition card. Since the camera is subject to human settings, the color transformation is a non-linear transformation. The obtained samples are used for training to obtain a high-dimensional space, transforming the non-linear problem into a linear problem.

[0029] The above technical solution has the following beneficial effects: This invention proposes a color correction method that breaks through the nonlinear relationship between brightness and viewing angle of LED large screens. In calibration, samples for both the color correction matrix and viewing angle light intensity calculations are sampled simultaneously. An artificial intelligence SVM algorithm is used to learn the color restoration network, transforming the nonlinear mapping into a linear mapping that the GPU can process at high speed. Furthermore, this technology integrates the latest achievements from applied mathematics, 3D engines, color science, image and graphics processing, forming a technological barrier for this invention. The protection point balances the real-time performance of color correction and distortion consistency, allowing for seamless integration within GPU programming. Attached Figure Description

[0030] Figure 1 This refers to the hue diagram involved in the embodiments of the present invention;

[0031] Figure 2 A schematic diagram of saturation;

[0032] Figure 3 This is a schematic diagram of the visible light range for humans in a chromaticity diagram;

[0033] Figure 4 This is a schematic diagram of the three primary colors of an LED screen;

[0034] Figure 5 The parameter diagram is shown in red;

[0035] Figure 6 This is a schematic diagram of the SVM principle in two-dimensional space;

[0036] Figure 7 This is a comparison chart between the solution and the true value after transforming a nonlinear problem in a low-dimensional space into a linear problem in a high-dimensional space using SVM;

[0037] Figure 8The solution obtained by SVM after transforming the low-dimensional space into a high-dimensional space;

[0038] Figure 9 A schematic diagram of the network structure for training the SVM in this embodiment of the disclosure;

[0039] Figure 10 This is a schematic diagram of the light energy angle of an LED diode;

[0040] Figure 11 This is a schematic diagram of the processing procedure for an infinite blue box XR.

[0041] Figure 12 A schematic diagram of the infinite blue box process that influences color;

[0042] Figure 13 This is a schematic diagram of the light energy angle of an LED diode;

[0043] Figure 14 This is a schematic diagram for real-time color correction calculation. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0045] The most difficult aspect is actually color matching. The result is that noticeable seams appear in the rendering of each LED screen. Furthermore, there's a significant color difference between the colors on the LEDs and the 3D rendered image. This leads to a huge color difference between the image captured inside the LEDs and the virtual rendered image outside the LEDs during the blue screen compositing process. Currently, there are several methods for color correction of LED screens using an infinite blue screen (XR):

[0046] Method 1: Manual calibration: Calibration is performed by adjusting the LED screen and camera; specifically, it includes:

[0047] Step 1) Some LED screen hardware manufacturers' splicers have simple color adjustment methods, allowing technicians to visually reduce the difference between the LED color gamut and the original image color gamut. For example, if there are differences in the brightness of the blue component of the LED screen, the blue LED drive pulses of all units can be adjusted. However, if there are also differences in the chromaticity of the blue component, then the red, green, and blue color components need to be adjusted to improve the error.

[0048] Step 2) Using the color correction function of the MR immersive studio, adjust the color contrast, saturation, gamma, gain, and offset values ​​of each screen to make the colors of the screens more consistent and minimize the seams between the large screens. Generally, the goal is to make the colors of each screen match the front screen, but manual adjustment requires a very high level of skill. Currently, it has been found that in addition to a set of adjustment values ​​for normal brightness (contrast, saturation, gamma, gain, and offset), separate sets of adjustment values ​​should be used for highlights and low light. This results in 15 adjustable values ​​for one screen, and 15*N adjustable values ​​for N screens. There is no unified standard for adjustment methods; it all relies on visual perception, which is the current state of most immersive studios. Although artificial intelligence technology is developing rapidly, there are very few methods that can be applied to this field.

[0049] Step 3) In the compositing stage, the simulated virtual scene rendering is treated as an LCD screen and adjusted, as described in Step 2), with 15 adjustable gamut settings. The color difference in the composite is then weakened by visual inspection. If, as in Step 1), the LED splicer does not have a color gamut adjustment button, then this is the primary method for adjusting the color difference.

[0050] This method is extremely difficult for technical personnel; theoretically, it's impossible to adjust the colors correctly. Even if a technician dedicated their entire life to color grading, the time would be far from sufficient. Therefore, one can only hope for an approximation, and even that is highly opportunistic. It's impossible for the same technician to repeatedly adjust to the same value as before.

[0051] Method 2: Calculate color difference and perform complementary color matching. This specifically includes calculating the color difference of each pixel in the simulated camera rendering process. Basic Knowledge 1: The foundation of color is hue, represented by 0-360 degrees.

[0052] Basic knowledge 2: The purity of hue is the same as saturation. The purer the color (red, green, blue), the higher the saturation, while gray, black, and white have lower saturation.

[0053] Basic Knowledge 3: Hue and saturation together constitute chroma. (In a context like...) Figure 3 In the chromaticity diagram shown, the range of light perceptible to humans is approximately 460 to 620 nm. It can be seen that the representable color gamut of various storage formats and devices (LED, LCD, etc.) approximates a triangle. The three corner points represent the three primary colors: red, green, and blue. When rendering from UE4 and transmitting to an LED LCD, the red, green, and blue (LED) are not the same as the red, green, and blue (UE4 engine); the colors have been converted, which is the main reason for color differences in LEDs. Figure 4As shown, the color gamut triangles of the three primary colors in an LED screen, as well as those in a regular LCD screen and a standard color chart, do not completely overlap. Therefore, only by aligning the three primary colors can the correct yellow be displayed. Cyan can then fall in the middle of the left hypotenuse of the blue triangle, thus displaying the correct cyan.

[0054] Basic knowledge 4: Brightness (color level, lightness, grayscale) determines how bright the white light shining on a color is; if the brightness is allowed to exceed 100, it will produce an overexposure effect.

[0055] Basic Knowledge 5: Color = Hue + Saturation + Brightness = Chroma + Brightness. Figure 5 For example, it can generate complex color clusters.

[0056] Basic Knowledge 6: Viewing Angle Characteristics of LEDs. The smallest unit of an LED display screen, an LED diode, has the strongest light intensity at a 0-degree viewing angle, which decreases as the emission axis deviates from the viewing angle. Furthermore, to complicate matters, a single LED point is composed of three light-emitting diodes, representing the three primary colors of red, green, and blue. Their light intensity and viewing angle attenuation differ. This must be tested by specialized testing personnel from the LED manufacturer.

[0057] Basic Knowledge 7: LED brightness decreases with ambient temperature: The brightness of red, green and blue LED beads will decrease as the ambient temperature rises, and the degree of decrease is not the same for red, green and blue.

[0058] Basic Knowledge 8: Camera White Balance: The basic concept of white balance is "to restore white objects to white regardless of the light source." For color casts that occur when shooting under specific light sources, compensation is made by strengthening the corresponding complementary color. The white balance adjustment process is achieved by adjusting the color temperature. It is generally recommended to adjust the white balance to daylight (5200k) on sunny days, to shade (7000k) or overcast (6000k) on cloudy days, to tungsten lamps (3200k) when the indoor light source is tungsten, and to white fluorescent lamps (4000k) when the light source is white fluorescent.

[0059] Warm and cool tones, as well as standard color temperature values ​​under normal daylight conditions.

[0060] Color temperature > 5000K, belonging to cool color tone (the color is bluish), with a cool atmosphere effect;

[0061] With a color temperature of 3300-5000K, it belongs to the mid-tone (white) and has a refreshing and bright atmosphere.

[0062] With a color temperature of <3300K, it belongs to the warm color tone (the color is reddish), and has a warm and stable atmosphere.

[0063] Basic Knowledge 9: Camera Exposure (Aperture, Shutter Speed, ISO): The exposure command is calculated using a linear color space (grayscale index 1.0) instead of the image's current color space. It can adjust the highlights, shadows, midtones, and contrast of the image separately. In 32-bit files, it is often used to solve the problem of overexposure or underexposure of the image, and can also easily achieve special styles such as the washout effect.

[0064] The intensity of light in the real world varies greatly, and cameras cannot guarantee accuracy without loss of precision across such a wide range. As shown in the image below, the light intensity (lumens, Lux) varies at different locations. Furthermore, a camera's sensitivity range is limited to a portion of the world's light and the image's field of view. Therefore, three important parameters are introduced when taking a picture:

[0065] Aperture (f-Stop): determines the amount of light passing through at any given time (can be considered the size of the aperture);

[0066] Shutter speed (1 / s): determines the length of time light can pass through (can be considered as the time the shutter opens and closes);

[0067] ISO sensitivity: determines the film's sensitivity to light.

[0068] The larger the aperture, the more blurred the background, which is what we often call the depth of field; the larger the shutter speed, the longer the moving object can retain its movement distance, producing a motion blur effect; the larger the ISO, the easier it is to produce an overexposed image, and the more noise will also be.

[0069] Method two requires consideration of factors at each stage, including objective and manually adjustable factors, all of which require rigorous calculation. This results in extremely high difficulty in collecting data, such as ensuring consistency between camera white balance, aperture, shutter speed, ISO, and UE4 engine data. The calculation formulas are also numerous. This leads to the inability to perform real-time calculations and rendering, as it's impossible to run the calculations at high speed on the graphics card or utilize UE4 engine shaders for graphics card programming. In short, it's highly complex and lacks real-time performance.

[0070] To address the problems of the two methods mentioned above, this invention proposes a color correction method, which includes a calibration step and a real-time calculation step.

[0071] The method in this embodiment of the invention does not use the color chart provided by the UE4 engine because the colors of the color chart provided by the UE4 engine are affected by many factors, including one or more of the following: the influence of the virtual scene, the color difference caused by the color gamut of the LED screen, the influence of the ambient light of the LED, the influence of the viewing angle of each pixel of the LED relative to the camera, the influence of the camera's white balance, the influence of the camera's exposure (aperture, shutter speed, ISO), the influence of the UE4 engine's white balance, and the influence of the UE4 engine's exposure. Therefore, directly using a color chart and calculating the color correction matrix (CCM) as in OpenCV is neither feasible nor correct.

[0072] The method in this embodiment of the invention proposes 5 constraints, 1 special scenario, 2 calibration calculations, and 1 real-time calculation.

[0073] (1.1) Constraint 1 is: Disable tone mapping in the UE4 engine:

[0074] Because the color gamut of a display is smaller than that of the real world, the existing UE4 engine provides a tone mapping function, which maps a wide range of HDR colors to a smaller range of LDR colors so that the display can display them correctly. This is a method of simulating the light response of film, which includes simulating the white balance of a camera. The inventors found that when using the UE4 engine, it was impossible to ensure that the white balance of a virtual scene was always consistent with the white balance value manually adjusted by the cinematographer, so tone mapping was disabled. Specifically, it includes:

[0075] If you need to obtain the SceneColor parameter, you can do so through the Post Process Material in the UE4 engine;

[0076] If you want to obtain the SceneColorBeforeTonemap parameter, you can change the Phase of PostProcess Material in the UE4 engine to Before Toonmapping.

[0077] (1.2) Constraint 2 is: Disable automatic exposure in the UE4 engine:

[0078] Because the UE4 engine typically requires manual exposure adjustment during lighting to avoid the effects of automatic exposure, the principle of automatic exposure is to ensure that mid-gray light remains mid-gray under different lighting conditions; that is, an input of 0.18 produces an output of 0.18. For example... Figure 6 As shown, the rightmost vertical line represents the current EV100 value; the middle line represents the target EV100 value; and the leftmost vertical line represents the final EV100 value after exposure compensation.

[0079] In addition, the UE4 engine adds exposure compensation, which offsets the exposure value based on the automatically calculated exposure. If the exposure calculated by automatic exposure is not satisfactory, the brightness and contrast levels can be manually adjusted. Exposure compensation is an exponential value, specifically including:

[0080] ExposureCompensation=2^AutoExposureBias

[0081] 0: No compensation

[0082] -1:1 / 2 bright

[0083] -2:1 / 4 Bright

[0084] 1:2 times brighter

[0085] 2:4 times brighter

[0086] In this embodiment of the disclosure, automatic exposure of UE4 is not required because it introduces an uncontrollable factor that interferes with color correction.

[0087] (1.3) Constraint 3: Before calibration, determine the white balance and exposure of the camera.

[0088] Color correction in this embodiment is related to manual camera settings. Color correction needs to be performed every time the camera configuration is modified, so it is forbidden to modify the internal values ​​of the camera during actual shooting.

[0089] (1.4) Constraint 4: Remove lights from the UE4 scene.

[0090] To reproduce the true colors of the color chart, the UE4's backlight must be removed, allowing the screen to emit its own light. This way, the color driven by the LEDs will be a pure, standard color from the color chart.

[0091] (1.5) Constraint 5: On-site LED screen lighting is off.

[0092] The main purpose is to prevent ambient light from affecting the color of the LED screen. In some cases, the intensity of the lighting is relatively high, which can have a certain impact on the color calibration results.

[0093] (2.1) Special Scenarios: Scenarios Consistent with LED Screen Space: If our virtual space also has panels identical to those on a real LED screen, and the CAVE screen configuration of the UE4 cluster is guaranteed to match the size of the real LED screen, then regardless of the camera's position, the image displayed on each LED screen will be the material of the corresponding panel. Thus, by assigning each panel a self-illuminating material, with its color specified by the program, we can construct a monochrome LED virtual environment. This monochrome image is then captured by the camera. It includes information on the primary and secondary colors of the color chart, as well as the brightness changes of the LED pixels from different viewing angles.

[0094] (3.1) Calibration calculation 1: Use support vector machine (SVM) to calculate the color restoration network.

[0095] For each color, the primary and secondary colors of the pixels at a 0-degree viewing angle (directly in front) are taken (LED screen projection + camera capture). By collecting all the 0-degree viewing angle (directly in front) data of all colors together to form a sample library, an artificial intelligence network for color reproduction can be trained.

[0096] From the original 3D scene rendered image to the actual captured image on the capture card, the process involves projecting the 3D scene onto the LED screen, then capturing the image from the LED screen onto the capture card. Since the camera is subject to human settings (exposure, white balance, etc.), the color transformation must be a non-linear transformation.

[0097] To achieve real-time color correction (restoration), we need to obtain a high-dimensional space from the above samples, transforming the nonlinear problem into a linear problem that is very easy to compute.

[0098] Most manufacturers on the market use an artificially imagined high-dimensional space V, which has multiple combinations of dimensions, from the original 3-dimensional space to a 19-dimensional space with a very large computational load.

[0099] V3 = [R, G, B]

[0100] V4 = [R, G, B, R * G * B]

[0101] V6 = [R,G,B,R*G,R*B,G*B]

[0102] V7=[R,G,B,R*G,R*B,G*B,R*G*B]

[0103] V9=[R,G,B,R*G,R*B,G*B,R 2 G 2 B 2 ]

[0104] V 10=[R,G,B,R*G,R*B,G*B,R 2 G 2 B 2 [R*G*B]

[0105] V 15 =[R,G,B,R*G,R*B,G*B,G*R 2 ,R*G 2 ,R*B 2 ,B*R 2 ,B*G 2 ,G*B 2 ]

[0106] V 16 =[R,G,B,R*G,R*B,G*B,G*R 2 ,R*G 2 ,R*B 2 ,B*R 2 ,B*G 2 ,G*B 2 [R*G*B]

[0107] V 19 =[R,G,B,R*G,R*B,G*B,G*R 2 ,R*G 2 ,R*B 2 ,B*R 2 ,B*G 2 ,G*B 2 ,R 3 G 3 B 3 [R*G*B]

[0108] Where: R represents the red channel value.

[0109] G represents the green channel value.

[0110] B represents the blue channel value.

[0111] Example V 15 It represents 15-dimensional space.

[0112] For example, R*G 2 This represents the square of the red channel value multiplied by the green channel value.

[0113] Through the high-dimensional space V, it is ideally assumed that color reproduction enters into a linear transformation.

[0114] X = A T *V

[0115] Roi = a 11 v 1i +a12 v 2i +...+a 1j v ji

[0116] Goi = a 21 v 1i +a 22 v 2i +...+a 2j v ji

[0117] Boi = a 31 v 1i +a 32 v 2i +...+a 3j v ji

[0118] Parameter description:

[0119] AT represents the matrix consisting of the parameters of the linear equations to be output below.

[0120] In Roi, 'o' represents the output, 'i' represents the i-th sample, and 'R' represents the red channel value. In summary, it means the red channel value of the output color for the i-th sample.

[0121] V represents a higher-dimensional space, such as V15 above.

[0122] In vji, i represents the i-th sample, and j represents the j-th coordinate in the high-dimensional space.

[0123] This training method is also very simple. After transforming the samples into high-dimensional space samples, the parameters of the linear transformation can be obtained simply through the pseudo-inverse operation of the matrix.

[0124] A=(V×V T ) -1 (V×X T )

[0125] Parameter description:

[0126] V represents the matrix consisting of all training samples in the high-dimensional space mentioned above.

[0127] VT represents the transpose of matrix V.

[0128] X represents the matrix consisting of all training samples in the input low-dimensional space, where the low-dimensional space is the color space (3-dimensional).

[0129] XT represents the transpose of matrix X.

[0130] However, in reality, no one can prove that the high-dimensional space to the RGB three-dimensional space is a linear mapping, so the training result is only that the error is small for the sample, and the generalization effect is usually very poor.

[0131] The Support Vector Machine (SVM) algorithm we use is itself a method for linear classification of nonlinear samples.

[0132] like Figure 6 The diagram shows the Support Vector Machine (SVM) algorithm in a two-dimensional space. Figure 6 The decision surface for classifying two-dimensional spatial samples shown is a straight line (dividing line).

[0133] The equation of a straight line is: x² = ax₁ + b

[0134] Parameter description:

[0135] x1 and x2 represent two dimensions of a 2D space.

[0136] The matrix equation is:

[0137] The general decision surface equation is expressed as: ω T x+γ=0

[0138] The vector controlling the direction of the line: ω = [ω1, ω2] T

[0139] The (n-1)-dimensional parameter controlling the position of a line: (intercept) γ is the intercept, which controls the position of the line.

[0140] Distance from any sample point x in space to the decision surface

[0141] Classification interval: W = 2d

[0142] The SVM algorithm aims to maximize W.

[0143] Add category labels to the sample points (here, there are two categories);

[0144] The minimum distance d from the two classes of samples to the decision boundary can then be written as:

[0145]

[0146] The decision surface equation is: and the line ω T x + γ = 0 is actually a straight line;

[0147] New direction vector and intercept:

[0148] By using the new direction vector and intercept, sample classification can be simplified to:

[0149]

[0150] The constraints are further refined in form by multiplying the category labels by the left-hand sides of the two inequalities to form a unified representation of all samples under the decision surface equation.

[0151]

[0152] In the improved formula, these special samples can be called support vectors:

[0153]

[0154] The problem of maximizing the classification margin W = 2d then becomes a problem of minimizing the decision equation; the mathematical description of the linear SVM optimization problem is as follows:

[0155]

[0156] sty i (ω T x i +γ)≥1, i=1, 2,..., m

[0157] Here: m is the total number of sample points, and the abbreviation st stands for "Subject to", meaning "subject to a certain condition".

[0158] To solve the constrained optimization problem described above, this embodiment of the disclosure removes the constraints using Lagrange duality, thus enabling the solution of the classification problem.

[0159] This embodiment of the disclosure does not use SVM to solve the classification problem, but rather uses SVM to transform a low-dimensional nonlinear problem (color transformation) into a high-dimensional linear problem. For nonlinear problems, SVM can obtain a very close approximation solution. For example... Figure 7 and Figure 8 As shown, SVM transforms a low-dimensional space into a high-dimensional space through a kernel function, and the kernel function can be trained. Through sample error feedback, the kernel function can continuously adjust its core and radius, and finally achieve the linear mapping we expect within a certain error tolerance range. As long as the kernel parameters and the parameters of the high-dimensional space mapping are obtained, they can be used for real-time color restoration on the GPU.

[0160] In this embodiment of the disclosure, the network used for training is as follows: Figure 9As shown, it includes R-shot, G-shot, and B-shot, corresponding to the captured red, green, and blue primary color data; through n kernels, R-raw, G-raw, and B-raw data can be obtained, corresponding to the original red, green, and blue primary color data. For example... Figure 9 As shown, the intermediate layer (such as Figure 9 The n kernels shown represent a linear mapping from the output layer (Roriginal, Goriginal, Boriginal), and this linear mapping is one of our training objectives. The mapping from the input layer (Rimage, Gimage, Bimage) to the intermediate layers is the SVM kernel function, but the core of the kernel is another training objective.

[0161] For each point in the image captured by the acquisition card, the technical solution of this disclosure embodiment can calculate the RGB color of the corresponding UE4 engine rendered image, and the input is the RGB of the acquisition card. The number of kernels is set by the technical solution of this disclosure embodiment. If the training result is still unsatisfactory after a considerable number of training iterations in the future, the number of kernels can be increased, that is, the dimension of the high-dimensional space can be increased, until it truly meets the requirements of linear mapping of the output layer.

[0162] There are many types of kernel functions. The technical solution of this disclosure embodiment selects the following Gaussian kernel function;

[0163]

[0164] Where x represents the low-dimensional space coordinates; x' represents the core coordinates of the high-dimensional space kernel in the low-dimensional space; and σ represents the kernel radius of the high-dimensional space kernel.

[0165] The Gaussian function is a kernel function with strong locality and is also the most widely used; it has good performance for both large and small samples, and it has fewer parameters than the polynomial kernel function.

[0166] (3.2) Calibration Calculation 2: Calculate the viewing angle light intensity curve for each color.

[0167] Extract the light intensity of the returned color (LED screen projection + camera capture), compare it with the light intensity from a 0-degree viewing angle (directly in front), and extract the light intensity as shown in the image. Figure 10 The curve shown is an example of this. The trick here is to utilize the position of the CAVE screen to pre-determine several 3D points (with viewing angles of 0, 5, 10, 15, ..., 85, 90), and then convert them into screen points. This makes sampling directly from the image on the capture card very efficient and simple. The samples are then created as a table of light intensity percentages, and during runtime, adjacent data interpolation can be used. This table can be passed to the GPU Shader via the Texture parameter, and GUPShader has an efficient interpolation algorithm.

[0168] (4.1) Real-time calculation 1: The image from the acquisition card is restored to the image rendered by the UE4 engine through the SVM color restoration network.

[0169] By integrating the color correction matrix and LED light intensity curve into UE4's graphics rendering, the "virtual" aspect of the compositing stage is brought closer to the "real" aspect. Leveraging the high performance of the graphics card, the real-time performance required for immersive XR (Infinite Blue Box) color correction can be achieved. This also includes offsetting camera distortion. The final output image can reflect the color changes and distortions of each pixel in the virtual scene.

[0170] Existing advanced color calibration technologies for infinite blue boxes are monopolized by a few top companies, and are neither widespread nor open-source. Most companies' XR color calibration technologies rely on manual eye adjustment, which presents significant difficulties for ordinary users due to its usability and high failure rate, likely resulting in lost customers. Solutions that rely on rigorous color theory for full-color calculations are computationally complex and cannot meet the real-time rendering requirements of 3D rendering engines. They cannot be integrated into the engine's GPU programming. Therefore, they cannot be used in immersive LED infinite blue box XR, and have been phased out from broadcasting and cultural tourism applications. This invention addresses data sampling, environmental constraints, and real-time rendering requirements, reaching the technological level of developed countries. Theoretically, it boasts a 90% calibration success rate and is exceptionally simple for end users, yielding excellent returns in practical applications.

[0171] The key feature of this disclosed embodiment is its breakthrough in overcoming the nonlinear relationship between brightness and viewing angle of LED large screens. During calibration, samples for both the color calibration matrix and viewing angle light intensity calculations are simultaneously collected. An artificial intelligence SVM algorithm is employed to learn the color reproduction network, transforming the nonlinear mapping into a linear mapping that can be processed at high speed by the GPU. Furthermore, this technology integrates the latest achievements from applied mathematics, 3D engines, color science, image and graphics processing, forming a significant technological barrier for this invention. The protection point is to balance the real-time performance of color correction and distortion consistency, seamlessly integrating both within GPU programming.

[0172] like Figure 11 The diagram shown is a flowchart of the infinite blue box processing in an embodiment of the present invention. The process includes:

[0173] Based on the 3D scene package provided by the art team, the first path involves a split-screen rendering computer, rendering on each screen according to its own viewpoint and projection area, and then displaying it on each split-screen LED. The CCD camera captures the image and transmits it to the compositing computer's capture card. The second path uses a camera positioning system (OptiTrack, HTC VIVE, or Mo-Sys, etc.) to obtain the real camera's position and focal length in CAV space in real time. Then, the position and field of view (FOV) of the virtual camera in the virtual scene are determined, and the image can be rendered. Because CCD camera images are distorted, the UE4 engine also needs to simulate camera distortion after rendering. Specifically, UV offset is added, implementing pre-calculated distortion offset images for different zooms (similar to a lookup table), and then UV distortion offset can be performed in real time. Finally, the two images are composited, which utilizes... Figure 4 This section introduces a real-time color correction algorithm.

[0174] like Figure 12 As shown, its entire process and Figure 11 It's basically the same, except that the analysis focused on factors that could significantly affect color. In summary, factors related to the UE4 engine, the LED screen, and the camera's shooting capabilities cause noticeable color differences (seams) in the synthesized image.

[0175] like Figure 13 The diagram shown illustrates the light energy angle of an LED diode. The inventors discovered a crucial factor in color accuracy: a strong correlation between the light intensity of an LED screen and its viewing angle, a phenomenon rarely mentioned in other literature. Accurate camera positioning is essential for calculating the viewing angle.

[0176] like Figure 14 The diagram illustrates the steps for real-time color correction. In the pixel processing flow of the rendering pipeline, the 3D spatial position and normal of a pixel can be calculated using its UV coordinates. Combined with the input camera positioning, the camera's viewing angle to that pixel can be calculated. Then, by consulting the viewing angle light intensity table, the percentage of current light intensity can be obtained; at a 0-degree viewing angle directly in front, the light intensity is 100%. Using a CAVE mask, it can be analyzed whether the current pixel is a "virtual" or "real" image. Images captured by a CCD are directly output without processing. If rendered using the UE4 engine, color transformation is used to calculate the simulated color on the LED screen. Then, the simulated color and light intensity percentage are processed to obtain the final simulated color under the viewing angle, which closely approximates the LED. This real-time rendering flow also includes distortion UV offset.

[0177] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A color correction method, characterized in that, Applied to the UE4 engine to process captured data; the method includes: Step 1: Disable tone mapping in the UE4 engine; If you need to obtain the SceneColor parameter, you can do so through the Post Process Material in the UE4 engine; If you want to get the SceneColorBeforeTonemap parameter, you can change the Phase of the Post ProcessMaterial parameter in the UE4 engine to Before Toonmapping. Step 2: Disable automatic exposure in UE4 engine to avoid automatic exposure affecting color correction; Step 3: Determine the camera's white balance and exposure, and do not modify the camera's white balance and exposure parameters during actual shooting; Step 4: Remove the lights from the UE4 scene and make the screen display self-illuminating so that the colors driven by the LEDs are the pure standard colors on the color chart; Step 5: At the LED screen site, turn off the lights to prevent ambient light from affecting the color of the LED screen. Step 6: Determine a scene consistent with the LED screen space, and configure the CAVE screen of the UE4 cluster to match the size of the actual LED screen; so that the image displayed on each LED screen is the material of the corresponding panel, regardless of the camera's position; set a self-illuminating material for each panel to construct a monochrome LED virtual environment; capture this monochrome panel with the camera to obtain a monochrome image, which contains information on the primary and secondary colors of the color chart, as well as the brightness changes of the LED pixels under different viewing angles; Step 7: Use a Support Vector Machine (SVM) to compute the color restoration network. The SVM network used for training includes an input layer, an intermediate layer, and an output layer. The input layer includes R-image, G-image, and B-image, corresponding to the captured red, green, and blue primary color data. The n kernels of the intermediate layer can be used to obtain the output layer R-original, G-original, and B-original, corresponding to the original red, green, and blue primary color data. For each color, the primary color and return color of the 0-degree viewing angle pixel are taken. The 0-degree viewing angle data of all colors are collected together to form a sample library, and the artificial intelligence network for color restoration is trained. The return color is obtained from the LED screen projection and camera capture. For each point in the image captured by the acquisition card, the corresponding RGB color of the UE4 engine-rendered image is calculated, and the input is the RGB of the acquisition card. The number of kernels is preset. If the training results are still unsatisfactory after a considerable number of training iterations, the number of kernels can be increased, that is, the dimension of the high-dimensional space can be increased, until it truly meets the requirements of the linear mapping of the output layer. The kernel function is the Gaussian kernel function as follows. Where x represents the high-dimensional space coordinates; x' represents the core coordinates of the high-dimensional space kernel; and σ represents the kernel radius of the high-dimensional space kernel. Step 8: Calculate the luminous intensity curve of each color: Extract the luminous intensity of the returned color, which is obtained by the LED screen projection and camera; compare the luminous intensity of the returned color with the luminous intensity of the 0-degree viewing angle to extract the contrast curve; in this process, multiple 3D points are pre-determined using the position of the CAVE screen and then converted into screen points. The samples can be made into a table of light intensity percentages. At runtime, adjacent data interpolation can be used. This table can be passed to the GPUShader through the Texture parameter, and GUPShader has an efficient interpolation algorithm. Step 9: Use the SVM color restoration network to restore the image from the capture card to the UE4 engine rendering image; integrate the color correction matrix and LED light intensity curve into the UE4 graphics card rendering, combine the virtual and real in the compositing stage, and use the high performance of the graphics card to obtain the real-time performance required for immersive XR color correction.

2. The color correction method according to claim 1, characterized in that, Step 7 also includes: From the original 3D scene rendered image to the actual captured image on the acquisition card, the process involves projecting the 3D scene onto the LED screen, then capturing the LED screen with a camera and transferring the image to the acquisition card. Since the camera is subject to human settings, the color transformation is a non-linear transformation. The obtained samples are used for training to obtain a high-dimensional space, transforming the non-linear problem into a linear problem.

Images