Multi-modal light control system based on ai driving

Abstract

The present invention is applicable to the field of intelligent lighting, and provides a multi-modal light control system based on AI driving. The system comprises: a light control module used for selecting a specified position from a surveillance footage and moving a light spot to the specified position in the surveillance footage; an automatic light focusing module used for automatically performing sharpness focusing on the light spot when the light spot has moved to the specified position; and a light spot cutting module used for setting a target shape in the surveillance footage, expanding the size of the light spot until the light spot covers the target shape, and cutting the light spot until the light spot is cut into the target shape. The solution provides a method that integrates manual operation and reinforcement learning, capable of achieving automatic shape cutting on the basis of instructions of lighting engineers, so that manpower is saved, and model performance is not affected by usage environments; and hardware is simple and is easy to deploy and maintain, thereby achieving low cost.

Description

AI-driven multimodal lighting control system Technical Field

[0001] This invention belongs to the field of intelligent lighting, and in particular relates to an AI-driven multimodal lighting control system. Background Technology

[0002] Multimodal lighting control systems are advanced technologies applied in the field of smart lighting. They combine artificial intelligence, sensor technology, data analysis, and lighting control technology to achieve intelligent and personalized lighting control and adjustment. Through multimodal lighting control systems applied to smart lighting, users can enjoy intelligent and personalized lighting experiences, improve their quality of life, and save energy. This system has broad application potential in smart homes, commercial lighting, entertainment venues, urban lighting, and art installations.

[0003] Traditional lighting control systems lack data-driven optimization and analytical data on historical data and performance indicators, hindering continuous improvement. Insufficient system automation leads to low efficiency, high operator skill requirements, and limited ease of use restricts application scenarios and on-site lighting effects. Furthermore, scalability and compatibility are limited, and industry products lack upgrades that incorporate AI technology.

[0004] On the human aspect, lighting controllers cannot maintain high concentration for extended periods, leading to decreased efficiency over long hours. Furthermore, their operational efficiency varies due to limitations in expertise and experience. This makes it difficult to guarantee optimal handling of emergencies. Additionally, the design and control personnel cannot monitor the entire stage from a distance.

[0005] Summary of the Invention

[0006] The purpose of this invention is to provide an AI-driven multimodal lighting control system, which aims to solve the technical problems existing in the prior art as identified in the background section.

[0007] This invention is implemented as follows: an AI-driven multimodal lighting control system, the system comprising:

[0008] The lighting control module is used to select a specific location from the monitoring screen and move the light spot to the specified location on the monitoring screen;

[0009] The light autofocus module is used to automatically focus on the sharpness of the light spot after it moves to a designated position;

[0010] The spot cutting module is used to set the target shape in the monitoring screen, expand the spot size until the spot covers the target shape, and cut the spot until the spot is cut into the target shape.

[0011] The lighting effects module is used to automatically adjust the color and color temperature of the lights.

[0012] As a further embodiment of the present invention, the working process of the lighting control module is as follows:

[0013] S1. Randomly generate horizontal and vertical rotation (pan, tilt) commands, and adjust the horizontal rotation angle pan and vertical rotation angle tilt of the lamp according to the commands;

[0014] S2. Acquire the monitoring image, identify the light spot and calculate the center coordinates (x, y) of the light spot. If the light spot cannot be identified, repeat step S1.

[0015] S3. Using the position coordinates (x, y) in the monitoring screen as the model input and the light command (pan, tilt) value as the model output, model a neural network model with an input layer, N hidden layers and an output layer.

[0016] S4. Obtain the optimal model through the prediction results of the neural network model and deploy it. Select the location where the light needs to be shone in the video surveillance screen and obtain the location information of the selected location.

[0017] S5. Input the location information into the neural network model, calculate the predicted output (pan, tilt) command value through forward propagation of the neural network, generate a light control signal based on the command value, and illuminate the designated location with light.

[0018] As a further embodiment of the present invention, the step of moving the light spot to a designated position in the monitoring screen specifically includes:

[0019] S1. Obtain the coordinates (Xe, Ye) of the selected pixel position in the monitoring screen;

[0020] S2. Obtain the coordinates of the light spot position through the monitoring screen;

[0021] S3, Obtain the light movement step size.

[0022] The light spot coordinates are (X1, Y1). The system issues the command pan+1 to rotate the lamp horizontally by one degree. Repeat S2 to obtain the adjusted light spot coordinates (X2, Y2).

[0023] For every degree increase in pan, the corresponding movement vector of the light spot on the screen is (Xp, Yp);

[0024] Where Xp = X2 - X1, Yp = Y2 - Y1;

[0025] The light spot coordinates are (X1`, Y1`). The system issues the command tilt+1 to rotate the lamp vertically by one degree. Repeat S2 to obtain the adjusted light spot coordinates as (X2`, Y2`).

[0026] For each degree increase in tilt, the corresponding movement vector of the light spot on the screen is (Xt, Yt);

[0027] Where Xt = X2' - X1', Yt = Y2' - Y1';

[0028] S4. Calculate the required movement values ​​P and T for pan and tilt respectively. The solution formula is:

[0029] Xe=P*Xp+T*Xt

[0030] Ye = P * Yp + T * Yt

[0031] Based on the values ​​of P and T, commands are issued to adjust the horizontal and vertical rotation angles of the lamp, moving the light spot to the specified position;

[0032] S5. Repeat S2 to obtain the light spot coordinates (X1, Y1). If both |Xe-X1| and |Ye-Y1| are within the threshold, the light spot has moved to the target position; otherwise, repeat S3.

[0033] As a further embodiment of the present invention, obtaining the position coordinates of the light spot includes:

[0034] By flashing the light and comparing the grayscale images when the light is off and on, the positions of the pixel blocks with brightness differences exceeding a threshold are the locations covered by the light spot. The center of this shape is calculated to obtain the coordinates of the light spot position.

[0035] As a further embodiment of the present invention, obtaining the position coordinates of the light spot includes:

[0036] Enter the macro command to reset all blades, so that the light spot remains circular;

[0037] The monitoring footage is acquired, and the Hough circular transform is applied to identify circular light spots in the footage.

[0038] As a further embodiment of the present invention, the working process of the automatic focusing module for the light is as follows:

[0039] S1. Obtain the distance from the light source to the illuminated surface and the parameters set by the user, including: zoom ratio, aperture size, and the on / off status of the light disc (gobo1, gobo2, gobo3).

[0040] S2. Establish the Actor model, integrate all the above data as input to the Actor model, and calculate the focus value;

[0041] S3. Generate a focus adjustment command based on the focus value, and adjust the lamps to achieve the focal length specified by the calculation module to complete the focusing operation.

[0042] As a further aspect of the present invention, the step of increasing the size of the light spot until the light spot covers the target shape specifically includes:

[0043] S1. Draw the shape of the target on the monitoring screen;

[0044] S2, Output macro commands to adjust zoom and aperture; Iris will adjust the bokeh to the minimum.

[0045] S3. Obtain the coordinates of the polygon vertices of the target shape, and obtain the average gray value of the pixels in the area around each vertex;

[0046] S4. Output macro commands to adjust the zoom value and aperture iris value to gradually enlarge the bokeh.

[0047] S5. Determine whether the gray value increment of the area around each vertex exceeds the threshold. If it exceeds the threshold, the target polygon is covered by the light spot.

[0048] If the threshold is not exceeded, repeat S3 and S4 until the gray value increment exceeds the threshold or the spot can no longer be magnified.

[0049] If the grayscale value does not change, it means that the smallest spot has initially covered the target shape or that the largest spot cannot cover the target.

[0050] If it means that the smallest spot has initially covered the target shape, then repeat S1;

[0051] If the maximum spot size cannot cover the target, then the spot cutting operation is abandoned.

[0052] As a further embodiment of the present invention, the step of cutting the light spot until the light spot is cut into the target shape specifically includes:

[0053] S1. Output macro command to set the light spot to a rectangle and identify the horizontal rotation angle of the light spot;

[0054] S2. Use edge detection and contour finding algorithms to identify the rectangular contour coordinates of the light spot, and use the rotating caliper algorithm to obtain the horizontal rotation angle of the rectangular shape of the light spot.

[0055] S3. Use the rotating caliper algorithm to obtain the minimum bounding rectangle of the target shape and its horizontal rotation angle;

[0056] S4. Calculate the rotation angle difference between the light spot rectangle and the minimum bounding rectangle of the target shape, and control the rotation angle of the blade disk to match the rotation angle of the light spot with the target shape.

[0057] S5. Perform the spot cutting operation and determine whether the spot matches the target shape by detecting the change in the brightness of the pixel.

[0058] As a further embodiment of the present invention, the working process of the lighting effect module is as follows:

[0059] The color of the current light is captured by the camera and converted from RGB to CMY color space.

[0060] Error values ​​were calculated for the cyan, magenta, and yellow channels respectively:

[0061] e(t) = Target channel value - Actual channel value

[0062] The target channel value and the actual channel value are obtained through color decomposition;

[0063] The parameters for each color channel are calculated using the PID control formula: u(t) = K p ·e(t)+K i ·∫e(t)dt+K d ·dtde(t)

[0064] Where: K p K i K d These are the PID parameters for each channel, u(t) is the adjusted output value used to adjust the light color, and e(t) is the calculated error;

[0065] Based on the output u(t) of the PID controller, a lighting control command is issued to adjust the parameter values ​​of each color channel and modify the output color of the light.

[0066] The beneficial effects of this invention are:

[0067] This solution proposes a method that integrates manual operation and reinforcement learning. It can automatically complete shape cutting according to the lighting technician's instructions, thereby saving labor, and the model performance is not affected by the usage environment. Even if the camera is blocked, the shape and color of the light spot of the lamp changes, or the overall lighting on site changes, it will not affect the model operation. At the same time, the hardware is simple, easy to deploy and maintain, and has low cost. Attached Figure Description

[0068] Figure 1 is a structural block diagram of the AI-driven multimodal lighting control system provided in an embodiment of the present invention;

[0069] Figure 2 is a schematic diagram of the structure provided in an embodiment of the present invention;

[0070] Figure 3 shows an example of a neural network model;

[0071] Figure 4 is a block diagram of the transfer learning scheme;

[0072] Figure 5 is a structural block diagram of the light autofocus module provided in an embodiment of the present invention;

[0073] Figure 6 shows the relationship between Zoom and Focus with Gobo focusing;

[0074] Figure 7 shows the relationship between Zoom and Focus without Gobo focusing. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0076] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first script may be referred to as a second script, and similarly, a second script may be referred to as a first script.

[0077] As shown in Figures 1 and 2, the AI-driven multimodal lighting control system includes:

[0078] The lighting control module is used to select a specific location from the monitoring screen and move the light spot to the specified location on the monitoring screen;

[0079] The light autofocus module is used to automatically focus on the sharpness of the light spot after it moves to a designated position;

[0080] The spot cutting module is used to set the target shape in the monitoring screen, expand the spot size until the spot covers the target shape, and cut the spot until the spot is cut into the target shape.

[0081] The lighting effects module is used to automatically adjust the color and color temperature of the lights.

[0082] As shown in Figure 3, in this embodiment of the invention, the working process of the lighting control module is as follows:

[0083] S1. Randomly generate horizontal and vertical rotation (pan, tilt) commands, and adjust the horizontal rotation angle pan and vertical rotation angle tilt of the lamp according to the commands;

[0084] S2. Acquire the monitoring image, identify the light spot and calculate the center coordinates (x, y) of the light spot. If the light spot cannot be identified, repeat step S1.

[0085] S3. Using the position coordinates (x, y) in the monitoring screen as the model input and the light command (pan, tilt) value as the model output, model a neural network model with an input layer, N hidden layers and an output layer.

[0086] S4. Obtain the optimal model through the prediction results of the neural network model and deploy it. Select the location where the light needs to be shone in the video surveillance screen and obtain the location information of the selected location.

[0087] S5. Input the location information into the neural network model, calculate the predicted output (pan, tilt) command value through forward propagation of the neural network, generate a light control signal based on the command value, and illuminate the designated location with light.

[0088] In the process of building a neural network model, the model also needs to be trained.

[0089] 1. Hyperparameter selection:

[0090] Number of hidden layer neurons: The number of hidden layers and the number of neurons in each layer are determined using cross-validation. Initially, a small number is set to avoid overfitting, and the number of neurons is gradually increased to improve the model's learning ability until performance on the validation set no longer improves.

[0091] Learning rate: Adaptive learning rate adjustment strategies, such as learning rate decay or Adam optimizer, are used to achieve rapid convergence in the early stages of training and stable learning in the later stages of training.

[0092] Batch Size: Choose an appropriate batch size based on available computing resources. Smaller batches can provide more frequent model updates, while larger batches may benefit memory optimization and accelerate computation.

[0093] Activation function: By comparing the effects of functions such as ReLU, Sigmoid, and Tanh at different levels, the activation function that produces the best training results is selected.

[0094] Number of iterations (Epochs): The number of iterations is determined by early stopping. If the performance on the validation set does not improve significantly within several consecutive epochs, training is stopped to prevent overfitting.

[0095] Regularization: To prevent overfitting, L2 regularization is applied to the weights, and an appropriate regularization coefficient is selected to balance model complexity and generalization ability.

[0096] Weight initialization: Use He initialization or Xavier initialization to optimize the weights, providing a good starting point for the training process and helping the gradient descent algorithm work more effectively.

[0097] 2. Training process:

[0098] After determining the hyperparameters, training the neural network begins. Training uses forward propagation to compute the predicted output, and then updates the weights via backpropagation. In each iteration, the loss function (such as mean squared error) is computed and optimized using the gradient descent algorithm. During training, model performance is monitored and optimized by adjusting hyperparameters such as the learning rate.

[0099] 3. Model Evaluation and Saving:

[0100] The model is evaluated using a test dataset to determine its generalization ability. The parameters of the best-performing model configuration are saved for use in actual lamp placement adjustments.

[0101] Furthermore, the following methods can be used in combination or individually to identify the location of light spots in the image.

[0102] 1. Shape recognition method

[0103] A. Output a blade signal to return the lamp's light-shielding blade to its initial position, ensuring an ideal circular light spot output.

[0104] B. Image Preprocessing: Acquire camera surveillance footage. Convert the image to grayscale to reduce computational complexity. Apply filtering algorithms to remove image noise. Perform binarization to highlight shape contours.

[0105] C. Edge detection: Finding the boundaries of shapes in an image using edge detection algorithms (such as the Canny edge detector).

[0106] D. Circular Spot Detection: Identify circular light spots in the image using the Hough transform algorithm and calculate the center coordinates.

[0107] 2. Using the blinking method to identify the center point of light spots in the monitoring screen.

[0108] A. Output light brightness signal (fixture@100%). The light control unit adjusts the light to maximum brightness. The monitoring image is captured and saved as image 1.

[0109] B. Output light brightness signal (fixture@0%). The light control unit adjusts the light to its dimmest setting. The monitoring image is captured and saved as image 2.

[0110] C. Perform grayscale conversion and noise reduction on images 1 and 2.

[0111] D. Compare the processed images 1 and 2, set an appropriate threshold, and convert the difference image into a binary image, where the light spot area is white and the rest is black.

[0112] E. Locate the contiguous white region in the binary image, calculate the geometric center of the white region, and determine the center position of the spot using the average pixel coordinates.

[0113] 3. Color contrast method

[0114] Light color settings and image capture

[0115] A. Output the light color adjustment command. The light control unit sequentially sets the light to red, green, and blue. For each color of illumination, the monitoring screen is captured and saved as Image 1, Image 2, and Image 3 respectively.

[0116] B. Color Space Conversion and Image Preprocessing

[0117] C. Convert the captured RGB image to HSV or LAB color space, which helps to perform color segmentation and analysis more effectively.

[0118] D. Apply histogram equalization to each color space converted image to improve image contrast and make color differences clearer.

[0119] Difference analysis

[0120] E. Use image processing techniques (such as difference image and color histogram comparison) to analyze color variations between different images. Extract significant changes in the light spot regions, which show the greatest differences under different color lighting conditions.

[0121] Binarization and Morphological Processing

[0122] F. Based on the results of the difference analysis, automatically calculate the optimal binarization threshold (e.g., using the Otsu method). Convert the difference image into a binary image, where the bright areas are white and the rest are black.

[0123] G. Perform morphological operations on the binary image, such as opening or closing operations, to remove small noise and smooth the boundaries of the light spots, further refining the shape of the light spots.

[0124] Spot center positioning

[0125] H. Locate connected white regions in the binary image. Apply geometric analysis to each white region, calculate its geometric center, and determine the precise center location of the spot using the average pixel coordinates or least squares fitted circle.

[0126] As shown in Figure 4, in real-world applications, the accuracy of the pre-trained model can be affected if the camera position changes.

[0127] Model transfer learning method

[0128] When the spatial positions of the camera, work surface, and lighting fixtures change, the statistical characteristics of the (pan, tilt) and (x, y) data will change, leading to a decrease in the accuracy of model predictions.

[0129] This solution employs transfer learning to address this problem. The specific details are as follows:

[0130] Fine-tuning methods:

[0131] Select Layers: Decide which layers need to be frozen and which layers need to be updated.

[0132] Training parameters: Train the unfrozen layers using the new dataset, adjusting the learning rate and batch size to optimize the training process.

[0133] Adapter model method:

[0134] An Adapter model g(x', y') is capable of mapping a new input (x', y') back to the input space (x, y) of the original model.

[0135] Model building: Define the structure of the Adapter model g(x', y') and determine the required number and type of neural network layers.

[0136] Training the Adapter model: While keeping the parameters of the original model f(x, y) fixed, train the Adapter model using only the new data.

[0137] Integration: Connects the output layer of the Adapter model with the input layer of the original model to ensure that the overall model can seamlessly handle the changed input data.

[0138] The model's performance is unaffected by the environment. After training, the prediction calculations for pan and tilt values ​​do not depend on any external environment. Even if the camera is obstructed, the shape and color of the light spots change, or the overall lighting conditions change, the model's operation will not be affected.

[0139] This solution only requires the addition of a camera; the hardware is simple, easy to deploy and maintain, and the cost is low.

[0140] Compared to alternative solutions, this method can illuminate the target area with lights in one step without requiring multiple calculations of (pan, tilt) values ​​for gradual adjustments.

[0141] Alternative solutions

[0142] I. Vector-based computation methods.

[0143] Step 1: Obtain the pixel coordinates (Xe, Ye) selected by the operator in the monitoring screen.

[0144] Step 2: Obtain the position coordinates of the light spot using the light spot recognition scheme described above.

[0145] Step 3: Calculate the light movement step length according to the following method.

[0146] Let the light spot coordinates be (X1, Y1). The system issues a command (pan+1) to rotate the lamp horizontally by one degree. Repeat step two to obtain the adjusted light spot coordinates as (X2, Y2). Each degree increase in pan corresponds to the movement vector of the light spot on the screen as (Xp, Yp), where Xp = X2 - X1 and Yp = Y2 - Y1. Increased computational complexity is needed to achieve higher accuracy.

[0147] Similarly, the movement vector corresponding to the vertical rotation of the lamp (tilt+1) is (Xt, Yt).

[0148] Calculate the values ​​P and T that pan and tilt need to move. Xe = P*Xp + T*Xt, Ye = P*Yp + T*Yt. Solving these equations yields P and T.

[0149] The control unit issues pan and tilt commands to adjust the lamp's horizontal and vertical rotation angles and move the lamp's illumination position.

[0150] Step 5: The system repeats Step 2 to obtain the light spot coordinates (X1, Y1). If both |Xe-X1| and |Ye-Y1| are within the threshold, the light spot has moved to the target position. Otherwise, Step 3 is repeated.

[0151] II. Vector Calculation Method of Integrated PID Algorithm

[0152] Step 1: Obtain the target location

[0153] Obtain the pixel position coordinates (Xe, Ye) in the monitoring screen. These coordinates represent the target position where the light spot needs to be moved.

[0154] Step 2: Spot Position Recognition

[0155] The current position coordinates (X1, Y1) of the light spot are obtained through a light spot recognition scheme. This step can adopt the light spot recognition method described in the aforementioned scheme.

[0156] Step 3: PID Controller Integration and Error Calculation

[0157] Proportion (P): Calculate the current errors ex = Xe - X1 and ey = Ye - Y1, and multiply them by the proportionality coefficient Kp.

[0158] Integral (I): Accumulate historical errors, update the integral terms Ix and Iy, and multiply by the integral coefficient Ki.

[0159] Differential (D): Calculate the rate of change of error, that is, the difference from the error of the previous period, and then multiply it by the differential coefficient Kd.

[0160] The final control signal is obtained by adding the proportional, integral, and derivative outputs.

[0161] ux = Px + Ix + Dx

[0162] uy=Py+Iy+Dyuy=Py+Iy+Dy

[0163] Step 4: Execute control commands

[0164] The control unit issues corresponding pan+ux and tilt+uy commands based on the output of the PID controller to adjust the horizontal and vertical rotation angles of the lamp.

[0165] The luminaire adjusts its position according to the control signal to move the light spot toward the target position.

[0166] Step 5: Location Confirmation and Feedback Adjustment

[0167] The system repeats step two to obtain the adjusted spot coordinates (X1, Y1).

[0168] Check the error between the position of the light spot and the target position. If both |Xe-X1| and |Ye-Y1| are within the preset threshold range, it is confirmed that the light spot has been correctly positioned.

[0169] If the error exceeds the threshold, the system recalculates the control signal based on the current error and returns to step four to continue adjusting until the error meets the requirements.

[0170] In this embodiment of the invention, obtaining the position coordinates of the light spot includes:

[0171] By flashing the light and comparing the grayscale images when the light is off and on, the positions of the pixel blocks with brightness differences exceeding a threshold are the locations covered by the light spot. The center of this shape is calculated to obtain the coordinates of the light spot position.

[0172] In this embodiment of the invention, obtaining the position coordinates of the light spot includes:

[0173] Enter the macro command to reset all blades, so that the light spot remains circular;

[0174] The monitoring footage is acquired, and the Hough circular transform is applied to identify circular light spots in the footage.

[0175] As shown in Figures 5 to 7, in this embodiment of the invention, the working process of the automatic light focusing module is as follows:

[0176] S1. Obtain the distance from the light source to the illuminated surface and the parameters set by the user, including: zoom ratio, aperture size, and the on / off status of the light disc (gobo1, gobo2, gobo3).

[0177] S2. Establish the Actor model, integrate all the above data as input to the Actor model, and calculate the focus value;

[0178] S3. Generate a focus adjustment command based on the focus value, and adjust the lamps to achieve the focal length specified by the calculation module to complete the focusing operation.

[0179] The focusing process includes four parts: building an enhanced learning environment, model training, intelligent lighting control, and feedback loop.

[0180] Enhance the learning environment

[0181] Enhancing the learning environment setup includes state parameters, designing reward calculation methods, and setting action space.

[0182] State parameters

[0183] The system state is defined by the following elements: distance, zoom, aperture, and the states of three flower discs (gobo1, gobo2, gobo3). Here, distance, zoom, and aperture are continuous floating-point values, while the flower disc states are Boolean values ​​(1 represents the flower disc is on, 0 represents it is off).

[0184] Reward calculation method

[0185] The reward is based on the sharpness of the light spot's edges. First, we collect monitoring footage from different states when the lighting technician manually adjusts the focus, and use the Sobel operator to calculate the edge sharpness (i.e., grayscale gradient) of these images. We manually set an acceptable grayscale gradient range [g_min, g_max] as the sharpness standard. In real-time operation, if the grayscale gradient generated by the AI's focus adjustment falls within this range, the AI ​​receives a reward; otherwise, the reward is 0.

[0186] Action space settings

[0187] Action space: The focus value that the AI ​​agent can adjust is defined as a continuous range. For example, if the focus specification is from 10mm to 100mm, then the action space would be [10, 100]. This setting allows the AI ​​to find the optimal focus value within a given range.

[0188] Model training

[0189] This method uses the DDPG algorithm with an Actor-Critic structure.

[0190] 1. Network Architecture

[0191] Actor Network: Consists of multiple fully connected layers. The input is the system state [distance, zoom, iris, gobo1, gobo2, gobo3], and the output is a continuous action value, i.e., the focus parameter. ReLU activation function is used to add non-linearity, and the last layer uses tanh or sigmoid function to ensure the output action is within a reasonable range.

[0192] Critic networks, also composed of multiple fully connected layers, take the state as input and the action values ​​generated by the Actor as input, and output the expected reward value of this state-action pair. Critic networks help evaluate the decisions of Actor networks.

[0193] 2. Training methods

[0194] Critic Network Update

[0195] The goal of a Critic network is to approximate the true Q-function, which gives the expected reward of taking action a given state s. The parameters of the Critic network are updated by minimizing a loss function L, which measures the error between the Critic network's predicted Q-value and the target Q-value. The loss function L has the following form: L(θ) Q ) = E s,a,r,s′ [(Q(s,a|θ Q )-y) 2 ]

[0196] Where, θ Q These are the parameters of the Critic network, and y is the target Q value, calculated as: y=r+γQ′(s′,μ′(s′∣θμ′)∣θQ′);

[0197] s′ is the new state after taking action a. γ is a discount factor used to reduce the impact of future rewards. Q′ and μ′ are the predictions of the target Critic network and the target Actor network, respectively. θQ′ and θμ′ are the parameters of the target Critic network and the target Actor network, respectively.

[0198] The Actor network is responsible for directly outputting the optimal action 'a'. The parameters θμ of the Actor network are updated using the following gradient ascent rule to maximize the expected reward estimated by the Critic network:

[0199] It is the gradient of the Critic network with respect to action a, which evaluates the effect of taking the current action recommended by the Actor network in state s.

[0200] It is the gradient of the Actor network output with respect to its parameters.

[0201] Intelligent lighting control

[0202] Distance measurement module operation

[0203] Function: Automatically obtains the distance from the light source to the illuminated surface.

[0204] Data transmission: The measured distance data is transmitted to the calculation module in real time.

[0205] User interaction module settings

[0206] Functionality: Allows users to set other parameters of the light through an interactive interface, including zoom (focal length scaling ratio), iris (aperture size), and the status (on or off) of the three flower discs (gobo1, gobo2, gobo3).

[0207] Data transmission: Transmits user-defined parameter values ​​to the calculation module.

[0208] Calculation module processing

[0209] Data integration: After receiving data from the ranging module and the user interaction module, the calculation module integrates this data as input to the Actor model.

[0210] Model computation: The input data includes [distance, zoom, iris, gobo1, gobo2, gobo3], and the computation module uses the trained Actor model to calculate the corresponding focus value.

[0211] Transmit the focus command

[0212] Command transmission: The calculation module generates a focus adjustment command based on the output of the Actor model and sends this command to the lighting control module.

[0213] Lighting control module focus

[0214] Focusing: After receiving the focus command, the lighting control module adjusts the lights to achieve the focal length specified by the calculation module, thus completing the focusing operation.

[0215] Feedback loop light focus

[0216] Autofocus: The calculation module uses a trained Actor model to automatically calculate the focus value, and the lighting control module adjusts the lights to achieve initial focus.

[0217] Manual fine-tuning: If the lighting technician finds the focus unsatisfactory during actual use, they can manually input a new focus command through the interactive module for fine-tuning. This allows the lighting technician to make precise adjustments based on the actual situation on site to achieve the best focus effect.

[0218] Data collection and storage

[0219] Data storage module: All lighting settings parameters [distance, zoom, iris, gobo1, gobo2, gobo3], as well as focus values ​​manually adjusted by the lighting technician, will be recorded and stored in the data storage module. This data is labeled as a dataset for subsequent model training and optimization.

[0220] Data structure: Each data point includes complete state parameters and the corresponding adjusted focus value, forming a structured dataset.

[0221] Continuous optimization of the model

[0222] Periodic fine-tuning: Fine-tune the Actor model using these labeled datasets at regular intervals or when the accumulated data reaches a certain amount.

[0223] Fine-tuning process: Transfer learning and a small learning rate are used to adjust the model parameters.

[0224] When the distance between the lamp and the illuminated surface is 4.475m, as shown in Figure 6, the parameters of the fitted polynomial model are: f(x)=ax6+bx5+cx4+dx3+ex2+fx+g

[0225] The parameters are as follows:

[0226] a=-5.96×10-11a=-5.96×10-11

[0227] b = 2.82 × 10⁻⁸

[0228] c=-6.80×10-6c=-6.80×10-6

[0229] d = 9.64 × 10⁻⁴

[0230] e=-7.82×10-2e=-7.82×10-2

[0231] f = 3.07

[0232] g = -11.49 g = -11.49

[0233] As shown in Figure 7, analysis of the relationship between Focus and Zoom data reveals that in the range of Zoom > 50, Focus = -0.334 × Zoom + 91.63. In the range of 30 to 50, Focus = 73.7.

[0234] In this embodiment of the invention, increasing the size of the light spot until the light spot covers the target shape specifically includes:

[0235] S1. Draw the shape of the target on the monitoring screen;

[0236] S2, Output macro commands to adjust zoom and aperture; Iris will adjust the bokeh to the minimum.

[0237] S3. Obtain the coordinates of the polygon vertices of the target shape, and obtain the average gray value of the pixels in the area around each vertex;

[0238] S4. Output macro commands to adjust the zoom value and aperture iris value to gradually enlarge the bokeh.

[0239] S5. Determine whether the gray value increment of the area around each vertex exceeds the threshold. If it exceeds the threshold, the target polygon is covered by the light spot.

[0240] If the threshold is not exceeded, repeat S3 and S4 until the gray value increment exceeds the threshold or the spot can no longer be magnified.

[0241] If the grayscale value does not change, it means that the smallest spot has initially covered the target shape or that the largest spot cannot cover the target.

[0242] If it means that the smallest spot has initially covered the target shape, then repeat S1;

[0243] If the maximum spot size cannot cover the target, then the spot cutting operation is abandoned.

[0244] In this embodiment of the invention, the step of cutting the light spot until the light spot is cut into the target shape specifically includes:

[0245] S1. Output macro command to set the light spot to a rectangle and identify the horizontal rotation angle of the light spot;

[0246] S2. Use edge detection and contour finding algorithms to identify the rectangular contour coordinates of the light spot, and use the rotating caliper algorithm to obtain the horizontal rotation angle of the rectangular shape of the light spot.

[0247] S3. Use the rotating caliper algorithm to obtain the minimum bounding rectangle of the target shape and its horizontal rotation angle;

[0248] S4. Calculate the rotation angle difference between the light spot rectangle and the minimum bounding rectangle of the target shape, and control the rotation angle of the blade disk to match the rotation angle of the light spot with the target shape.

[0249] S5. Perform the spot cutting operation and determine whether the spot matches the target shape by detecting the change in the brightness of the pixel.

[0250] The beam cutting module includes three operation stages:

[0251] The first stage involves positioning the light spot at the geometric center of the target shape. The second stage involves expanding the light spot to cover the target shape. The third stage involves AI cutting the light spot to fit the target shape.

[0252] First stage: Positioning the light spot at the geometric center of the target shape:

[0253] 1. Adjust the zoom and iris settings in the lighting control unit to minimize the light spot.

[0254] 2. Calculate the coordinates of the geometric center of the drawn shape.

[0255] 3. The light is directed to the center of the shape manually or using the method described in "An Intelligent Positioning Method and System for Light Illumination Direction Based on Computer Vision".

[0256] Furthermore, the geometric center coordinates can be obtained by calculating the centroid of the shape:

[0257] 4. Obtain shape outline coordinates

[0258] 5. Calculate the moments of the profile.

[0259] 6. Calculate the centroid: Use the formula for moments to calculate the centroid of the shape. The x and y coordinates of the centroid can be calculated using moments M: Cx = M10 / M00, Cy = M01 / M00

[0260] M10 and M01 are first-order moments, while M00 is the zero-order moment, which is the area.

[0261] Second stage: Increase the size of the light spot to cover the target shape:

[0262] 1. First, obtain the position of each vertex of the target polygon, and calculate the average gray value of the 3x3 pixel area centered on each vertex. Define this value as the "initial gray value".

[0263] 2. Send macro commands to the lighting control unit, instructing it to adjust the zoom and iris parameters of the light spot in order to gradually enlarge the light spot.

[0264] 3. After magnifying the light spot, remeasure the new grayscale value of the above vertex region and calculate the difference between the new grayscale value and the initial grayscale value.

[0265] If the difference exceeds the preset threshold n, the target polygon is considered to have been successfully covered by the light spot.

[0266] If the difference does not exceed the threshold n, return to step 2 and repeat the adjustment of the spot size.

[0267] 4. When the light spot cannot be further magnified (reaching the maximum zoom and iris values ​​allowed by the device), if the change in grayscale value still does not exceed the threshold n, it is considered that the light spot has already covered the target polygon at its initial size. At this time, a macro command will be issued to the lighting control unit, instructing it to adjust the light spot to the initial or minimum setting.

[0268] Phase 3: AI cuts the light spot into the target shape.

[0269] 1. By training the AI ​​to operate 8 blades for cutting and overall rotation through reinforcement learning, the shape cutting is achieved by controlling the [1A,1B,2A,2B,3A,3B,4A,4B,rotate] command.

[0270] 2. Further, this section includes four parts: reinforcement learning environment setup, model training, intelligent lighting control, and feedback loop, implemented using the following methods:

[0271] a. Enhance the learning environment

[0272] Enhancing the learning environment setup includes state parameters, designing reward calculation methods, and setting action space.

[0273] State parameters

[0274] The system state consists of the target contour pixel coordinates Box[(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…], and the spot edge coordinates Contour:[(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…]

[0275] b. Reward calculation method

[0276] Two pixels are set as detection points on both the inner and outer sides of each edge of the target shape, and the initial brightness values ​​of these detection points are recorded. A brightness decrease threshold of x% is set to determine whether it is bright or dark. The shape cutting is considered complete when all outer detection points are dark and all inner detection points are bright. A large reward is given to the AI ​​agent when the shape cutting is complete. Otherwise, the reward is 0.

[0277] c. Action space settings

[0278] Motion space: The AI ​​agent can adjust the position values ​​of the eight blades:

[0279] [1A,1B,2A,2B,3A,3B,4A,4B,rotate], where each blade value is defined as a continuous interval. For example, 1A ∈ [0,100]. This setting allows the AI ​​to find the optimal blade position within a given range.

[0280] Specifically, the intelligent lighting control system includes:

[0281] 1. Calculation and processing

[0282] Data integration: After acquiring data from the camera feed and user interaction module, integrate this data as input to the Actor model.

[0283] Model calculation: The input data includes the target contour pixel coordinates Box[(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…], and the spot edge coordinates Contour:[(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…]

[0284] The calculation module uses the trained Actor model to calculate the corresponding values ​​of the eight blade positions [1A,1B,2A,2B,3A,3B,4A,4B,rotate].

[0285] 2. Transmit the focus command

[0286] Command transmission: The calculation module generates a focus adjustment command based on the output of the Actor model and sends this command to the lighting control module.

[0287] Lighting control module focus

[0288] Focusing: After receiving the command, the lighting control module adjusts the position of the lamp blade to complete the cutting operation.

[0289] 3. Feedback Loop

[0290] 1. Light cutting

[0291] Automatic cutting: The calculation module uses a trained Actor model to automatically calculate the blade and rotation parameters, and the lighting control module adjusts the lights to achieve shape cutting.

[0292] Manual fine-tuning: If the lighting technician finds the cutting effect unsatisfactory, they can manually input a new cutting command through the interactive module for fine-tuning. This allows the lighting technician to make precise adjustments based on the actual situation on site to achieve the best focus effect.

[0293] 2. Data collection and storage

[0294] Data storage module: All state parameters and action values ​​manually adjusted by the lighting technician will be recorded and stored in the data storage module. This data is labeled as a tag dataset for subsequent model training and optimization.

[0295] Data structure: Each data entry includes complete state parameters and the corresponding adjusted action value, forming a structured dataset.

[0296] 3. Continuous optimization of the model

[0297] Periodic fine-tuning: Fine-tune the Actor model using these labeled datasets at regular intervals or when the accumulated data reaches a certain amount.

[0298] Fine-tuning process: Transfer learning and a small learning rate are used to adjust the model parameters.

[0299] Alternative solutions can be adopted for the above approach:

[0300] Prerequisite: Users draw their desired shape, such as a trapezoid, on the monitoring screen using an interactive unit.

[0301] This method consists of three operational phases:

[0302] The first stage involves directing the light spot to several central positions of the target shape.

[0303] In the second stage, the light spot will be expanded to cover the shape of the target.

[0304] In the third stage, the moving blade cuts the light spot to the target shape.

[0305] The main difference from the aforementioned scheme lies in the third phase, and this section will only describe the differences in the third phase:

[0306] Third stage: Cutting to the target shape

[0307] Step 1: Set the shape of the light spot

[0308] The output command is sent to the control unit, which instructs to adjust the cutting light so that it expands by 25% in each of the four main directions, forming an approximately rectangular light spot.

[0309] Step 2: Identify the shape of the light spot and its rotation angle

[0310] 1. The system acquires monitoring footage of the current work area via a camera and uses edge detection and contour finding algorithms to identify the contour coordinates of the rectangular light spot.

[0311] 2. Use the rotating caliper algorithm to calculate the horizontal rotation angle of the light spot.

[0312] Step 3: Obtain target shape parameters

[0313] Output commands to the control unit, and use the rotating caliper algorithm to identify and obtain the minimum circumscribed rectangle of the target shape and its horizontal rotation angle.

[0314] Step 4: Match the rotation angle of the light spot with the target shape

[0315] Calculate the rotation angle difference (denoted as x) between the light spot and the target shape. Then output a command to the control unit to adjust the rotation angle x of the blade disk to match the rotation angle of the light spot to the target shape.

[0316] Step 5: Perform the cutting

[0317] 1. Set two pixels on the inside and outside of each edge of the target shape as detection points and record the initial brightness value of these detection points. Set a brightness decrease threshold of x% to determine whether it is bright or dark.

[0318] 2. Automatically adjust the blade position based on the real-time collected brightness of the detection points:

[0319] If both the inner and outer detection points are lit, the blade will cut further inward.

[0320] If the outer detection point is dark and the inner detection point is bright, the blade will stop.

[0321] If the inner detection point is dark and the outer detection point is also dark, the blade will retract a small step outward.

[0322] 3. Issue a control command, and the control unit repeats the above adjustment for the blade operation on each side until all outer detection points on all sides are dark and all inner detection points are bright. At this point, the shape cutting is considered complete.

[0323] 4. If it is detected that the blade has reached its mechanical limit and cannot complete any more cuts, the operation will be terminated based on the macro command value.

[0324] In this embodiment of the invention, the workflow of the lighting effect module is as follows:

[0325] The color of the current light is captured by the camera and converted from RGB to CMY color space.

[0326] Error values ​​were calculated for the cyan, magenta, and yellow channels respectively:

[0327] e(t) = Target channel value - Actual channel value

[0328] The target channel value and the actual channel value are obtained through color decomposition;

[0329] The parameters for each color channel are calculated using the PID control formula:

[0330] u(t) = K p ·e(t)+K i ·∫e(t)dt+K d ·dtde(t)

[0331] Where: K p K i K d These are the PID parameters for each channel, u(t) is the adjusted output value used to adjust the light color, and e(t) is the calculated error;

[0332] Based on the output u(t) of the PID controller, a lighting control command is issued to adjust the parameter values ​​of each color channel and modify the output color of the light.

[0333] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0334] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0335] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0336] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0337] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An AI-driven multi-modal light control system based on, characterized by, The system comprises: A light control module for selecting a specified position in a monitoring picture and moving a light spot to the specified position in the monitoring picture; A light automatic focusing module for automatically focusing the light spot on the specified position; A light spot cutting module for setting a target shape in the monitoring picture, expanding the size of the light spot until the light spot covers the target shape, and cutting the light spot until the light spot is cut into the target shape; A light effect module for automatically adjusting the color and color temperature of the lamp.

2. The system of claim 1, wherein, The working process of the light control module is as follows: S1. Randomly generate a horizontal and vertical rotation (pan, tilt) command, and adjust the horizontal rotation angle pan and the vertical rotation angle tilt of the lamp according to the command; S2. Obtain the monitoring picture, identify the light spot, and calculate the light spot center position coordinates (x, y). If the light spot cannot be identified, repeat step S1; S3. Take the position coordinates (x, y) in the monitoring picture as the model input, and take the light command (pan, tilt) value as the model output to model, and obtain an N-layer neural network model comprising an input layer, N hidden layers, and an output layer; S4. Obtain the optimal model through the prediction result of the neural network model and deploy it. Click the position in the video monitoring picture where the light needs to be irradiated, and obtain the clicked position information; S5. Input the position information into the neural network model, calculate the prediction output (pan, tilt) command value through the forward propagation of the neural network, generate a light control signal through the command value, and irradiate the light to the specified position.

3. The system of claim 1, wherein, The moving of the light spot to the specified position in the monitoring picture specifically comprises: S1. Obtain the selected pixel position coordinates (Xe, Ye) in the monitoring picture; S2. Obtain the light spot position coordinates through the monitoring picture; S3. Obtain the light moving step, Let the light spot coordinates be (X1, Y1). The system issues an instruction pan+1 to increase the horizontal rotation of the lamp by one degree. Repeat S2 to obtain the adjusted light spot coordinates (X2, Y2); The moving vector of the light spot in the screen corresponding to each increase of one degree of pan is (Xp, Yp); Where Xp=X2-X1, Yp=Y2-Y1; Let the light spot coordinates be (X1`, Y1`). The system issues an instruction tilt+1 to increase the vertical rotation of the lamp by one degree. Repeat S2 to obtain the adjusted light spot coordinates (X2`, Y2`); The moving vector of the light spot in the screen corresponding to each increase of one degree of tilt is (Xt, Yt); Where Xt=X2`-X1`, Yt=Y2`-Y1`; S4. Calculate the values P and T of pan and tilt that need to be moved, respectively, and solve the formulae: Xe=P*Xp+T*Xt Ye=P*Yp+T*Yt According to the values of P and T, issue an instruction to adjust the horizontal and vertical rotation angles of the lamp to move the light spot to the specified position; S5. Repeat S2 to obtain the light spot coordinates (X1, Y1). If |Xe-X1| and |Ye-Y1| are within the threshold, the light spot has moved to the target position. Otherwise, repeat S3.

4. The system of claim 3, wherein, The obtaining of the light spot position coordinates comprises: By flashing light, comparing the gray scale picture when the contrast light is off and on, the pixel block position whose brightness difference exceeds the threshold value is the position covered by the light spot, the center of the shape is calculated, and the light spot position coordinates are obtained.

5. The system of claim 3, wherein, The light spot position coordinates are obtained by: The input macro command resets all blades, and the light spot maintains a circular shape; The monitoring picture is obtained, and the Hough circle transformation is applied to identify the circular light spot in the picture.

6. The system of claim 1, wherein, The working process of the light automatic focusing module is: S1, the distance from the light to the light receiving surface and the parameters set by the user are obtained, including: zoom, iris, and gobo1, gobo2, gobo3; S2, an Actor model is established, and all the above data is integrated as the input of the Actor model to calculate the focus value; S3, the focus adjustment command is generated according to the focus value, and the light is adjusted to reach the focus specified by the calculation module, and the focusing operation is completed.

7. The system of claim 1, wherein, The light spot size is enlarged until the light spot covers the target shape, specifically including: S1, draw the target shape in the monitoring picture; S2, output the macro command to adjust the zoom and iris to adjust the light spot to the minimum; S3, obtain the polygon vertex position coordinates of the target shape, and obtain the average gray value of the pixels in the area around each vertex; S4, output the macro command to adjust the zoom value and iris value to gradually enlarge the light spot; S5, judge whether the gray value increment of the area around each vertex exceeds the threshold value, if it exceeds the threshold value, the target polygon is covered by the light spot; If the threshold value is not exceeded, repeat S3 and S4 until the gray value increment exceeds the threshold value or the light spot cannot be enlarged any more; If the gray value does not change, it means that the minimum light spot has initially covered the target shape or that the maximum light spot cannot cover the target; If it means that the minimum light spot has initially covered the target shape, repeat S1; If it means that the maximum light spot cannot cover the target, give up the light spot cutting operation. The light spot is cut until the light spot is cut into the target shape, specifically including:

8. The system of claim 7, wherein, S1, output the macro command to set the light spot to a rectangle, and identify the horizontal rotation angle of the light spot; S2, use edge detection and contour finding algorithm to identify the rectangular contour coordinates of the light spot, and use the rotation jamming algorithm to obtain the horizontal rotation angle of the light spot rectangle; S3, use the rotation jamming algorithm to obtain the minimum circumscribed rectangle of the target shape and its horizontal rotation angle; S4, calculate the rotation angle difference between the light spot rectangle and the minimum circumscribed rectangle of the target shape, and control the rotation angle of the blade disc to make the light spot rotation angle match the target shape; S5, execute the light spot cutting operation, and judge whether the light spot fits the target shape by detecting the brightness change of the point pixels. The working process of the light effect module is:

9. The system of claim 1, wherein, The color of the current light in the picture captured by the camera is captured, and the color RGB is converted into the color space CMY Calculate the error value for each cyan, magenta, and yellow channel: e(t) = target channel value - actual channel value The target channel value and the actual channel value are obtained by color decomposition; ​ The PID control formula is used to calculate the parameters of each color channel: u(t) = K p · e(t) + K i · ∫ e(t) dt + K d · dt de(t) where: K p , K i , K d are the PID parameters for each channel, u(t) is the adjusted output value, used to regulate Light color, e(t) is the calculated error; According to the output u(t) of the PID controller, the light control command is issued to adjust the parameter value of each color channel to modify the light color output.

Images