A method, system, device, and storage medium for rendering light and shadow based on 2D images.

Abstract

This application relates to the technical field of lighting and shadow rendering, and particularly to a lighting and shadow rendering method, system, and medium based on 2D images. The method includes: predicting the light source parameters of the next frame using a Long Short-Term Memory (LSTM) model based on a locally stored sequence of historical light source parameters, generating a prediction dataset; real-time acquisition of the actual light source parameters of the current frame, calculating the deviation between the prediction dataset and the actual parameters; inputting the deviation value into a pre-trained lightweight error compensation model to generate corrected light source parameters, wherein the error compensation model is obtained by offline training of the historical deviation sequence; if the cumulative deviation of N consecutive frames exceeds a dynamic threshold, initiating a full parameter request to the server and updating the weights of the local LSTM model; and using the corrected light source parameters to drive a 2D lighting and shadow rendering engine to output the image. This application completely solves the problems of bandwidth, latency, error, and energy consumption.

Description

Technical Field

[0001] This application relates to the technical field of light and shadow rendering, and in particular to a light and shadow rendering method, system, device and storage medium based on 2D images. Background Technology

[0002] In recent years, lighting and shadow rendering typically involves steps such as light source calculation, shadow generation, and reflection processing. In real-time lighting and shadow rendering of 2D images on mobile devices (such as AR navigation and interactive educational applications), dynamic light source effects rely on continuously updated light source parameters (position, intensity, color temperature).

[0003] Traditional solutions employ two modes: one is full transmission, where each frame requests complete light source data from the server, resulting in high bandwidth consumption (single light source > 2KB / frame) and significant latency in weak network environments; the other is local differential compression, where the client only transmits the parameter differences between adjacent frames, but lacks autonomous deviation control. When the light source changes abruptly (such as suddenly turning on a spotlight), errors accumulate, eventually requiring forced full synchronization, resulting in an image quality distortion rate exceeding 15%.

[0004] Clients rely excessively on server-side validation or simple compression algorithms, especially in pure client-side modes without server feedback (such as standalone design tools). Existing solutions struggle to balance the conflict between data volume and rendering accuracy. Therefore, there is an urgent need for a client-integrated prediction-correction solution to break through the strong dependence of traditional architectures on server-side collaboration. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this application provides a method, system, electronic device, and storage medium for rendering light and shadow based on 2D images.

[0006] Firstly, this application provides a lighting and shadow rendering method based on 2D images, employing the following technical solution:

[0007] Based on the locally stored historical light source parameter sequence, the light source parameters of the next frame are predicted using a Long Short-Term Memory (LSTM) network model, generating a prediction dataset.

[0008] Real-time acquisition of the actual light source parameters of the current frame, and calculation of the deviation between the predicted dataset and the actual parameters;

[0009] The deviation value is input into a pre-trained lightweight error compensation model to generate corrected light source parameters. The error compensation model is obtained by offline training of historical deviation sequences.

[0010] If the cumulative deviation of N consecutive frames exceeds the dynamic threshold, a full parameter request is sent to the server and the weights of the local LSTM model are updated.

[0011] The modified light source parameters drive the 2D lighting and rendering engine to output the image.

[0012] The above scheme combines the LSTM prediction model with the client-side local error compensation model to form a "prediction-compensation-self-learning" closed loop, and links the dynamic threshold with the network state design to break through the rigid control of the traditional fixed threshold.

[0013] By using LSTM prediction and a local error compensation model in a closed loop, the vast majority of frames do not require server interaction, supporting offline rendering scenarios. This eliminates the round-trip latency of server verification in traditional solutions, greatly reducing end-to-end rendering latency and achieving complete client autonomy.

[0014] Meanwhile, the server is only requested when the cumulative error exceeds the threshold for N consecutive frames, effectively reducing the communication volume under normal conditions; the error compensation model parameters occupy very little memory, which can be adapted to low-spec mobile devices and achieve low bandwidth usage.

[0015] The dynamic threshold mechanism automatically expands the tolerance range when the light source changes abruptly, avoiding invalid server requests; the compensation model updates the weights in real time, stabilizing the low image distortion rate under abrupt change scenarios and improving the ability to resist changes.

[0016] In one possible implementation, the step of inputting the deviation value into a pre-trained lightweight error compensation model to generate corrected light source parameters, wherein the error compensation model is obtained by offline training of historical deviation sequences, includes:

[0017] The error compensation model is a linear regression model, whose input is the deviation sequence of the most recent M frames and whose output is the compensation coefficient of the current frame. The model weights are updated in real time through a local gradient descent algorithm.

[0018] The above scheme allows for real-time updates of model weights, improving compensation accuracy and overcoming the limitation of fixed compensation models in adapting to dynamic changes.

[0019] In one possible implementation, the step of initiating a full parameter request to the server and updating the local LSTM model weights if the cumulative deviation of N consecutive frames exceeds a dynamic threshold includes:

[0020] The dynamic threshold is adaptively adjusted according to the network status. When the network latency is detected to be higher than T milliseconds, the threshold is increased to K times the base value (K>1) to suppress the frequency of server requests.

[0021] The above solution reduces server requests during high latency, thereby reducing screen stuttering in scenarios with network fluctuations.

[0022] In one possible implementation, the adjustment strategy for the dynamic threshold includes:

[0023] When the client is charging and the battery level is above 80%, the first threshold T1 is used;

[0024] When the client's battery level is below 20% or the temperature exceeds 45°C, the second threshold T2 (T2>T1) is used.

[0025] When switching thresholds, knowledge distillation compression is performed on the LSTM model to reduce computation by 30%.

[0026] The above solutions reduce power consumption and extend battery life when the battery is low, or lower GPU temperature in high-temperature scenarios to avoid frequency throttling, thus solving the problem of high power consumption limiting the battery life of mobile devices.

[0027] In one possible implementation, the step of using the corrected light source parameters to drive the 2D lighting and shadow rendering engine to output the image includes employing differentiated compensation strategies for different types of light source parameters when driving 2D lighting and shadow rendering, such as:

[0028] The position parameters were smoothed using a Kalman filter.

[0029] Apply gamut boundary constraints (sRGB standard) to color parameters;

[0030] The strength parameters are compensated after logarithmic domain compression.

[0031] The above solutions effectively reduce color overflow rate, decrease position jitter amplitude, improve motion smoothness, and solve the problem of color or position distortion caused by uniform compensation.

[0032] In one possible implementation, the step of predicting the light source parameters of the next frame using a Long Short-Term Memory (LSTM) network model based on a locally stored sequence of historical light source parameters to generate a prediction dataset includes:

[0033] The training data for the LSTM model includes synthetic light source parameter sequences and user operation mode augmentation data, wherein,

[0034] The sequence of parameters of the synthesized light source is generated by a random walk algorithm to simulate the continuous changes in the position, intensity and color temperature of the light source;

[0035] The enhanced user operation mode data is based on historical user interaction records and injects sudden parameter mutation fragments to improve the model's anti-interference ability.

[0036] The above approach reduces the prediction error for sudden user actions, shortens the model training cycle, reduces annotation costs, and prevents models trained on synthetic data from deviating from real user behavior.

[0037] In one possible implementation, it further includes,

[0038] After detecting that the user has continuously manipulated the light source parameters more than Q times, the adversarial training mode is activated:

[0039] Generate adversarial light source mutation sequences and input them into the LSTM model;

[0040] Update the LSTM model weights based on the adversarial loss function of the model output and the true parameters;

[0041] Reset the training buffer of the error compensation model.

[0042] The above solutions reduce the model crash rate in highly interactive scenarios, improve the success rate of first-frame correction after adversarial training, and effectively reduce the scenarios where frequent operations cause the model to crash.

[0043] Secondly, this application provides a lighting and shadow rendering system based on 2D images, comprising:

[0044] Light source prediction module: Based on the locally stored historical light source parameter sequence, it predicts the light source parameters of the next frame through a Long Short-Term Memory (LSTM) network model, generating a prediction dataset;

[0045] Deviation calculation module: Real-time acquisition of real light source parameters in the current frame, and calculation of the deviation value between the predicted dataset and the real parameters;

[0046] Error compensation module: It has a built-in pre-trained lightweight error compensation model. The deviation value is input into the model to generate the corrected light source parameters. The model is obtained by offline training of historical deviation sequences.

[0047] Dynamic decision module: Monitors the cumulative deviation of N consecutive frames. If it exceeds the dynamic threshold, it sends a full parameter request to the server and updates the LSTM model weights.

[0048] Rendering Execution Module: Uses the corrected light source parameters to drive the 2D lighting and shadow rendering engine to output the image.

[0049] Thirdly, this application provides an electronic device including a memory and a processor, wherein the memory is used to store computer program code, and the processor is used to execute the computer program code stored in the memory to implement the methods in the first aspect and any one of the first aspects, or in the second aspect and any possible implementation of the second aspect.

[0050] Fourthly, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the methods described in the first aspect and any one thereof, or the second aspect and any possible implementation thereof.

[0051] In summary, this application includes at least one of the following beneficial technical effects:

[0052] By combining the LSTM prediction model with the client-side local error compensation model, a closed loop of "prediction-compensation-self-learning" is formed. Furthermore, the dynamic threshold is linked to the network status design, breaking through the rigid control of traditional fixed thresholds and completely solving the problems of bandwidth, latency, error, and energy consumption. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating a 2D image-based lighting and shadow rendering method provided in an embodiment of this application.

[0054] Figure 2 This is a schematic diagram of the structure of a 2D image-based lighting and shadow rendering system provided in an embodiment of this application.

[0055] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions in this application will now be described with reference to all the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0057] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.

[0058] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0059] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two.

[0060] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0061] This application provides a 2D image-based lighting and shadow rendering method, executed by an electronic device. This electronic device can be a standalone physical electronic device, a cluster of multiple physical electronic devices, a distributed system, or a cloud electronic device providing cloud computing services. This application does not impose limitations on this method. Figure 1 As shown, the method includes:

[0062] S1. Based on the locally stored historical light source parameter sequence, predict the light source parameters of the next frame using the Long Short-Term Memory (LSTM) network model to generate a prediction dataset.

[0063] Specifically, data preparation and storage: historical light source parameters are stored in JSON format, including location (two-dimensional coordinates x, y), intensity (normalized value 0.0~1.0), color temperature (2700K~6500K), sampling frequency of 30Hz, and the most recent 50 frames of data are retained.

[0064] Data is stored in a client-side SQLite database and is indexed by timestamps for fast retrieval.

[0065] The construction of an LSTM model includes the model structure, training parameters, and input sequence length.

[0066] Furthermore, the model structure consists of: an input layer (3D, corresponding to position, intensity, and color temperature), two hidden layers (each with 64 neurons and the activation function tanh), and an output layer (3D, with linear activation).

[0067] Training parameters: Adam optimizer (learning rate 0.001), loss function is mean squared error (MSE), training epochs are 100, and batch size is 32.

[0068] Input sequence length: dynamically adjusted to the most recent 10 frames (when memory is sufficient) or 5 frames (when memory is limited), calculated using the following formula:

[0069] L = [Available memory (MB) / 0.5]

[0070] Each frame parameter occupies 0.5MB of memory.

[0071] Specifically, the prediction process is as follows: before each frame is rendered, the historical sequence is loaded from the database and input into the LSTM, and the prediction parameters for the next frame are output; the prediction time is ≤3ms.

[0072] S2. Real-time acquisition of the actual light source parameters of the current frame, and calculation of the deviation between the predicted dataset and the actual parameters.

[0073] Specifically, the data acquisition method obtains real light source parameters in real time through the client rendering engine API (such as Android's SurfaceFlinger), with a sampling interval of 33ms (30Hz).

[0074] The data acquisition process employs a double-buffered queue to avoid conflicts between the acquisition and prediction threads.

[0075] Specifically, the deviation between the predicted dataset and the true parameters is calculated by including the Euclidean distance deviation for the position parameters, the absolute error for the intensity and color temperature, and the total deviation value as a weighted sum of the three.

[0076] Furthermore, the formula for calculating the Euclidean distance deviation for the position parameters is: ;

[0077] The absolute error formula for intensity and color temperature is as follows: , ;

[0078] The total deviation is the weighted sum of the three: .

[0079] S3. Input the deviation value into the pre-trained lightweight error compensation model to generate the corrected light source parameters.

[0080] Specifically, the error compensation model is a linear regression model (input: the deviation sequence of the last 5 frames, output: the compensation coefficient of the current frame), which is obtained by offline training of historical deviation sequences.

[0081] The specific training data consists of 10,000 bias sequences extracted from historical logs. Offline training uses stochastic gradient descent (SGD) with a learning rate of 0.01 and 500 iterations. The trained weight file (12KB in size) is embedded in the client application's resource directory.

[0082] Furthermore, when the current frame and the deviation values ​​of the previous 4 frames are input (a total of 5-dimensional features), the model outputs a compensation coefficient α (range 0.8~1.2).

[0083] The specific correction formula is as follows: The calculation time is ≤1ms.

[0084] S4. If the cumulative deviation of N consecutive frames exceeds the dynamic threshold, send a full parameter request to the server and update the weights of the local LSTM model.

[0085] Specifically, the baseline threshold T base =0.15 (equilibrium point determined experimentally); when network latency > 100ms, the dynamic threshold adjustment formula is: Among them, the cumulative deviation of N=5 consecutive frames exceeds T dynamic The request is triggered at that time.

[0086] Furthermore, the full parameter request data packet contains the complete light source parameters and device ID of the current frame (JSON format, 2KB in size).

[0087] Furthermore, the server returns the latest LSTM model weight file and replaces the local model weights with Android's dynamic class loader (DexClassLoader).

[0088] Furthermore, during the update, a downgrade mode is enabled, directly using the compensated parameters and skipping the prediction step.

[0089] S5. Use the corrected light source parameters to drive the 2D lighting and shadow rendering engine to output the image.

[0090] Specifically, the corrected parameters are converted into OpenGL ES shader Uniform variables (such as u_LightPos, u_LightIntensity) and 2D projection calculations are performed.

[0091] In summary, by combining the LSTM prediction model with the client-side local error compensation model, a closed loop of "prediction-compensation-self-learning" is formed, and the dynamic threshold is linked to the network state design, breaking through the rigid control of traditional fixed thresholds.

[0092] First, a closed-loop client-side autonomy is achieved. By using LSTM prediction and a local error compensation model, 99% of frames do not require server-side interaction. This supports rendering scenarios in offline and flight modes, eliminating the round-trip latency of server-side verification in traditional solutions. End-to-end rendering latency is greatly reduced, achieving complete client-side autonomy.

[0093] Meanwhile, resource consumption is greatly simplified, and the server is only requested when the cumulative error exceeds the threshold for N consecutive frames, thus effectively reducing the communication volume under normal conditions; the error compensation model parameters consume very little memory, which can be adapted to low-end mobile devices and achieve low bandwidth consumption.

[0094] In addition, the system quantifies the ability to resist mutations. The dynamic threshold mechanism automatically expands the tolerance range when the light source changes mutally, avoiding invalid server requests. The compensation model updates the weights in real time, stabilizing the low image distortion rate in mutation scenarios and improving the ability to resist mutations.

[0095] In some embodiments, S4 further includes:

[0096] S401. The dynamic threshold is adaptively adjusted according to the network status. When the network latency is detected to be higher than T milliseconds, the threshold is increased to K times the base value (K>1) to suppress the frequency of server requests.

[0097] Specifically, the network latency detection method uses a client to periodically send lightweight heartbeat packets (UDP protocol, data size 64B) to the server to calculate the average round-trip time (RTT). The heartbeat frequency is once every 500ms, the statistical window is the moving average of the last 5 RTTs, and the latency threshold T=100ms.

[0098] Furthermore, the Android NetworkRequest API is used to monitor the network type (4G / 5G / WiFi) in real time. When the average RTT is >100ms, it is determined to be a high latency state, triggering a threshold adjustment.

[0099] Specifically, the formula for calculating the dynamic threshold is:

[0100] Among them, T base =0.15, K max =3.0, to prevent extreme network jitter from causing the threshold to be too large.

[0101] For example, when RTT = 150ms, K = 1.75, then T dynamic =0.26; when RTT=300ms, K=3.0, and T dynamic =0.45.

[0102] Specifically, when the cumulative deviation of N=5 consecutive frames exceeds T dynamicThis triggers a suppression of server request frequency, forcing a server request interval of ≥2 seconds under high latency conditions. This reduces server requests during high latency, lowering the rate of screen stuttering in scenarios with network fluctuations.

[0103] In some embodiments, the dynamic threshold adjustment strategy includes:

[0104] S402. When the client is charging and the battery level is above 80%, the first threshold T1 is used.

[0105] Specifically, when the device is connected to a charger and the battery level is >80% (monitored in real time via the Android BatteryManager API), it switches to aggressive mode with a threshold T1=0.10.

[0106] Specifically, the aggressive mode allows for a higher tolerance for deviations, reduces the frequency of server requests to 0.05 times per second, and raises the priority of the rendering thread to the highest level to ensure smooth visuals.

[0107] S403. When the client's battery level is below 20% or the temperature exceeds 45°C, the second threshold T2 (T2>T1) is used.

[0108] Specifically, when the battery level is <20% (monitored in real time via the Android BatteryManager API) or the device temperature is >45℃ (CPU / GPU temperature obtained via the Android Thermal API), switch to conservative mode, set the threshold T2 to 0.25, and trigger LSTM model compression.

[0109] Specifically, by increasing the threshold to reduce the computational load of model prediction and compensation, the device's heat generation and power consumption are alleviated, and rendering degradation mode (such as turning off anti-aliasing and reducing shadow resolution) is forced to be enabled.

[0110] S404. When switching thresholds, knowledge distillation compression is performed on the LSTM model to reduce computation by 30%.

[0111] Specifically, knowledge distillation compression reduces the original model's 64 neurons to 32 neurons, decreasing the parameter size from 1.2MB to 0.42MB; simultaneously, it converts the model weights from FP32 to INT8, reducing the model size from 1.2MB to 0.4MB. This reduces single-frame prediction time from 3ms to 1.2ms and GPU utilization from 18% to 6%.

[0112] In summary, reducing power consumption and extending battery life when the battery is low, or lowering GPU temperature in high-temperature scenarios to avoid throttling, solves the problem of high power consumption limiting the battery life of mobile devices.

[0113] In some embodiments, S3 further includes:

[0114] The error compensation model is a linear regression model. Its input is the deviation sequence of the most recent M frames, and its output is the compensation coefficient of the current frame. The model weights are updated in real time through a local gradient descent algorithm.

[0115] Specifically, the input data is the deviation sequence (Δ1, Δ2, Δ3, Δ4, Δ5) of the most recent M=5 frames, with each frame's deviation including position, intensity, and color temperature errors (a total of 15 features). The output data is the compensation coefficient α of the current frame (a scalar value, ranging from 0.8 to 1.2).

[0116] Furthermore, the online learning process is as follows:

[0117] Loss function: Mean Squared Error (MSE)

[0118] Where, α real (Ideal values ​​of compensation coefficients per frame) were obtained from an offline labeled dataset.

[0119] Gradient descent update:

[0120]

[0121]

[0122] Where the learning rate η = 0.01, and X is the input feature vector.

[0123] Update frequency: Weight update is triggered once after each frame is rendered, taking ≤0.5ms.

[0124] Specifically, compared with the fixed-weight model, online learning improves the compensation accuracy by 22%; at the same time, the model weights are updated in real time, which improves the compensation accuracy and overcomes the inability of the fixed compensation model to adapt to dynamic changes.

[0125] In some embodiments, when driving 2D lighting and shadow rendering in S5, the differentiated compensation strategies for different types of light source parameters include: applying Kalman filtering to smooth the position parameters, performing color gamut boundary constraints (sRGB standard) on the color parameters, and compensating the intensity parameters after logarithmic domain compression.

[0126] Specifically, the smoothing effect is that the standard deviation of position jitter is reduced from 12 pixels to 4.8 pixels (a reduction of 60%), and the smoothness of motion trajectory is improved (PSNR is increased from 28dB to 35dB).

[0127] Specifically, the color spillover rate (ΔE>3) decreased from 12% to 0.5%, and sRGB compliance was verified by CalMAN (average ΔE=1.2).

[0128] Specifically, overexposed highlight areas were reduced by 75% (the percentage of pixels with intensity > 0.9 decreased from 8% to 2%), and shadow detail retention was improved (SSIM increased from 0.82 to 0.93).

[0129] In summary, positional smoothing is achieved, and Kalman filtering suppresses high-frequency jitter (such as coordinate jumps when AR objects move quickly); color gamut constraint is achieved to prevent the compensated colors from exceeding the device's display capabilities and to meet the color management needs of professional design tools; intensity compression is achieved, and logarithmic transformation expands the dynamic range of dark areas.

[0130] It effectively reduces color spillage, minimizes positional jitter, improves motion smoothness, and solves the problem of color or positional distortion caused by uniform compensation, preventing the loss of highlight details.

[0131] In some embodiments, S1 further includes:

[0132] The training data for the LSTM model includes synthetic light source parameter sequences and augmented data of user operation modes, among which,

[0133] The sequence of parameters for the synthesized light source is generated using a random walk algorithm to simulate the continuous changes in the position, intensity, and color temperature of the light source.

[0134] Enhanced user operation pattern data is based on historical user interaction records, and sudden parameter mutation fragments are injected to improve the model's anti-interference ability.

[0135] Specifically, 1 million light source operation events (anonymized) were extracted from anonymous user logs. Among them, the operation pattern features were extracted based on parameters exceeding a threshold between adjacent frames and operation frequency: the frequency of users dragging and scaling the light source per unit time was counted.

[0136] Specifically, the data augmentation strategy involves randomly inserting user-mutated fragments into the synthesized sequence with an insertion probability of 20% and a mutation duration of 1 to 5 frames; and applying user operation frequency features to the synthesized data to enhance temporal correlation.

[0137] Specifically, in the augmented dataset, the total data volume = synthetic data (80%) + user-augmented data (20%); the labeling system includes two types of labels: "synthetic" and "user mutation", which are used for adversarial training.

[0138] Specifically, the prediction error for sudden user actions is reduced, while the model training cycle is shortened, the annotation cost is reduced, and the model trained on synthetic data is prevented from deviating from real user behavior.

[0139] In some embodiments, it also includes:

[0140] S6. After detecting that the user has continuously operated the light source parameters more than Q times, start the adversarial training mode.

[0141] Specifically, continuous user operation is defined as modifying the light source parameters ≥ 3 times within 1 second via touch or a peripheral device (such as a mouse). The threshold Q = 10 times, and the number of times is counted in real time by an event counter.

[0142] Specifically, the adversarial training mode is triggered when the number of operations within 5 seconds is ≥Q and the parameter mutation rate (average difference between adjacent frames) is >0.2. Adversarial training mode may involve enabling degraded rendering, disabling unnecessary lighting effects (such as motion blur), and ensuring a frame rate ≥24 FPS during training.

[0143] S7. Generate adversarial light source mutation sequences and input them into the LSTM model.

[0144] Specifically, for example, mutation mode injection: such as position mutation, randomly set the light source coordinates to the edge of the canvas (x=0 or width, y=0 or height); such as intensity mutation: randomly jump within the range of 0.1~1.0 (jump amplitude ≥0.5); such as color temperature mutation: randomly jump between 2700K~6500K (jump amplitude ≥2000K).

[0145] Specifically, the adversarial light source mutation sequence construction involves generating a dataset containing 1000 adversarial sequences, each sequence being 15 frames long; the adversarial sample to normal sample mixing ratio is 1:3 to prevent the model from overfitting to the adversarial features.

[0146] S8. Update the LSTM model weights based on the adversarial loss function of the model output and the true parameters.

[0147] Specifically, a GRL is inserted during the adversarial training phase, and the gradient is multiplied by -1 during backpropagation, forcing the model to resist perturbations. The GRL is located between the LSTM output layer and the fully connected layer.

[0148] Specifically, a hot update mechanism is used to dynamically replace the LSTM model weight file (.h5 format) without restarting the rendering engine.

[0149] S9. Reset the training buffer of the error compensation model.

[0150] Specifically, the reset conditions are: after the adversarial training ends, or when the proportion of old data in the buffer is >40% (data timestamps exceed 30 seconds).

[0151] Specifically, the operation performed on the buffer is to clear the current buffer (capacity M=1000 frames), reinitialize it, and inject 50% new data generated by adversarial training and 50% historical normal data.

[0152] Specifically, full compensation is enabled in the first frame after reset (skipping the prediction module) to ensure image stability; the compensation error is monitored for 10 frames after reset, and if it exceeds the threshold, it is rolled back to the previous version buffer.

[0153] In summary, this approach reduces the model crash rate in highly interactive scenarios, improves the success rate of first-frame correction after adversarial training, and effectively reduces the likelihood of model crashes due to frequent operations.

[0154] The following describes the 2D image-based lighting and shadow rendering system provided in the embodiments of this application. The lighting and shadow rendering system described below can be referred to in correspondence with the lighting and shadow rendering method described above.

[0155] refer to Figure 2 The lighting and shadow rendering system includes:

[0156] Light source prediction module 1: Based on the locally stored historical light source parameter sequence, predict the light source parameters of the next frame through the Long Short-Term Memory (LSTM) network model to generate a prediction dataset.

[0157] Deviation Calculation Module 2: Real-time acquisition of real light source parameters in the current frame, and calculation of the deviation between the predicted dataset and the real parameters.

[0158] Error Compensation Module 3: It has a built-in pre-trained lightweight error compensation model. The deviation value is input into the model to generate the corrected light source parameters. The model is obtained by offline training of historical deviation sequences.

[0159] Dynamic Decision Module 4: Monitors the cumulative deviation of N consecutive frames. If it exceeds the dynamic threshold, it sends a full parameter request to the server and updates the LSTM model weights.

[0160] Rendering Execution Module 5: Uses the corrected light source parameters to drive the 2D lighting and shadow rendering engine to output the image.

[0161] This application provides an electronic device, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.

[0162] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in connection with the embodiments of this application. Processor 301 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0163] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0164] The memory 303 may be a ROM (Read-Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM (Electrically Erasable Programmable Read-Only Memory), a CD-ROM (Compact Disc Read-Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0165] The memory 303 is used to store application code that executes the scheme of the embodiments of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0166] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0167] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described light and shadow rendering method.

[0168] Since the embodiments of the computer-readable storage medium portion correspond to the embodiments of the method portion, please refer to the description of the embodiments of the method portion for the embodiments of the computer-readable storage medium portion.

[0169] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0170] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A lighting and shadow rendering method based on 2D images, characterized in that, include: Based on the locally stored historical light source parameter sequence, the light source parameters of the next frame are predicted using a Long Short-Term Memory (LSTM) network model, generating a prediction dataset. Real-time acquisition of the actual light source parameters of the current frame, and calculation of the deviation between the predicted dataset and the actual parameters; The deviation value is input into a pre-trained lightweight error compensation model to generate corrected light source parameters. The error compensation model is obtained by offline training of historical deviation sequences. If the cumulative deviation of N consecutive frames exceeds the dynamic threshold, a full parameter request is sent to the server and the weights of the local LSTM model are updated. The corrected light source parameters are used to drive the 2D lighting and shadow rendering engine to output the image; In the step of inputting the deviation value into a pre-trained lightweight error compensation model to generate corrected light source parameters, wherein the error compensation model is obtained by offline training of historical deviation sequences,... include, The error compensation model is a linear regression model, whose input is the deviation sequence of the most recent M frames and whose output is the compensation coefficient of the current frame. The model weights are updated in real time through a local gradient descent algorithm. The step of initiating a full parameter request to the server and updating the local LSTM model weights if the cumulative deviation of N consecutive frames exceeds the dynamic threshold includes: The dynamic threshold is adaptively adjusted according to the network status. When the network latency is detected to be higher than T milliseconds, the threshold is increased to K times the base value, where K>1, to suppress the frequency of server requests. In the step of using the corrected light source parameters to drive the 2D lighting and shadow rendering engine to output the image, the differentiated compensation strategy for different types of light source parameters when driving 2D lighting and shadow rendering includes... The position parameters were smoothed using a Kalman filter. Color gamut boundary constraints are applied to color parameters, and the color gamut boundary constraints conform to the sRGB standard. The strength parameters are compensated after logarithmic domain compression; In the step of generating a prediction dataset by predicting the light source parameters of the next frame using a Long Short-Term Memory (LSTM) model based on the locally stored historical light source parameter sequence, include, The training data for the LSTM model includes synthetic light source parameter sequences and user operation mode augmentation data, wherein, The sequence of parameters of the synthesized light source is generated by a random walk algorithm to simulate the continuous changes in the position, intensity and color temperature of the light source; The enhanced user operation mode data is based on historical user interaction records and injects sudden parameter mutation fragments to improve the model's anti-interference ability.

2. The method according to claim 1, characterized in that, The adjustment strategy for the dynamic threshold includes: When the client is charging and the battery level is above 80%, the first threshold T1 is used; When the client's battery level is below 20% or the temperature exceeds 45°C, a second threshold T2 is used, where T2 > T1; When switching thresholds, knowledge distillation compression is performed on the LSTM model to reduce computation by 30%.

3. The method according to claim 1, characterized in that, Further including, After detecting that the user has continuously manipulated the light source parameters more than Q times, the adversarial training mode is activated: Generate adversarial light source mutation sequences and input them into the LSTM model; Update the LSTM model weights based on the adversarial loss function of the model output and the true parameters; Reset the training buffer of the error compensation model.

4. A lighting and shadow rendering system based on 2D images, characterized in that, include: Light source prediction module: Based on locally stored historical light source parameter sequences, it predicts the light source parameters for the next frame using a Long Short-Term Memory (LSTM) network model, generating a prediction dataset. In this module, the LSTM model's training data includes synthetic light source parameter sequences and user operation mode enhancement data. The synthetic light source parameter sequences are generated using a random walk algorithm to simulate continuous changes in light source position, intensity, and color temperature. The user operation mode enhancement data is based on historical user interaction records, injecting sudden parameter mutation fragments to improve the model's anti-interference capability. Deviation calculation module: Real-time acquisition of real light source parameters in the current frame, and calculation of the deviation value between the predicted dataset and the real parameters; Error compensation module: It has a built-in pre-trained lightweight error compensation model. The deviation value is input into the model to generate the corrected light source parameters. The model is obtained by offline training of historical deviation sequences. In the error compensation module, the error compensation model is a linear regression model. Its input is the deviation sequence of the most recent M frames, and its output is the compensation coefficient of the current frame. The model weights are updated in real time through a local gradient descent algorithm. Dynamic decision module: Monitors the cumulative deviation of N consecutive frames. If it exceeds the dynamic threshold, it sends a full parameter request to the server and updates the LSTM model weights. In the dynamic decision module, the dynamic threshold is adaptively adjusted according to the network status. When the network latency is detected to be higher than T milliseconds, the threshold is increased to K times the baseline value, where K is greater than 1. Rendering execution module: Uses the corrected light source parameters to drive the 2D lighting and shadow rendering engine to output the image; In the rendering execution module, when the 2D lighting and shadow rendering engine outputs the image using the corrected light source parameters, a differentiated compensation strategy is adopted for different types of light source parameters, including applying Kalman filtering to smooth the position parameters, performing color gamut boundary constraints on the color parameters, and compensating the intensity parameters after logarithmic domain compression.

5. An electronic device, characterized in that, include: One or more processors; One or more memory units; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 3.

Images