AI-driven multimodal lighting control system

Abstract

The present invention is applicable to the field of intelligent lighting and provides an AI-driven multimodal lighting control system, which includes: a lighting control module, which is used to select a specified position from a monitoring screen and move the light spot to the specified position in the monitoring screen; a lighting autofocus module, which is used to automatically focus the light spot for clarity after the light spot moves to the specified position; a light spot cutting module, which is used to set a target shape in the monitoring screen, expand the light spot size until the light spot covers the target shape, and cut the light spot until the light spot is cut into the target shape; this solution proposes a method that integrates manual operation and reinforcement learning, which can automatically complete shape cutting according to the lighting engineer's instructions, thereby saving labor, and the model performance is not affected by the use environment, the hardware is simple, easy to deploy and maintain, and the cost is low.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent lighting, and in particular to a multimodal lighting control system driven by AI. Background Art

[0002] A multimodal lighting control system is an advanced technology system used in the smart lighting field. It combines artificial intelligence, sensor technology, data analysis, and lighting control technologies to achieve intelligent and personalized lighting control and adjustment. Through the multimodal lighting control system applied in smart lighting, users can enjoy an intelligent and personalized lighting experience, improve their quality of life, and save energy. This system has broad application potential in smart homes, commercial lighting, entertainment venues, urban lighting, and art installations.

[0003] Traditional lighting control systems lack industry data-driven optimization and analytical data on extensive historical data and performance indicators, hindering continuous optimization. Inadequate system automation leads to low efficiency, high operator expertise requirements, and the inconvenience of complex operations limit the application scenarios and on-site lighting effects of lighting control systems. Furthermore, scalability and compatibility are limited, and industry products lack upgrades adapted to AI technology.

[0004] On the manual side, lighting control personnel struggle to maintain effective concentration for extended periods, leading to decreased efficiency. Furthermore, limited by their expertise and experience, the efficiency of operating control equipment varies. This makes it impossible to guarantee optimal response methods when responding to emergencies. Furthermore, design and control personnel lack visibility to monitor the overall stage situation. Summary of the Invention

[0005] The purpose of the present invention is to provide an AI-driven multimodal lighting control system, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0006] The present invention is implemented as follows: a multimodal lighting control system based on AI drive, the system comprising:

[0007] The lighting control module is used to select a designated position from the monitoring screen and move the light spot to the designated position in the monitoring screen;

[0008] Light auto-focus module, used to automatically focus the light spot after it moves to the specified position;

[0009] The light spot cutting module is used to set the target shape in the monitoring screen, expand the light spot size until the light spot covers the target shape, and cut the light spot until the light spot is cut into the target shape;

[0010] The lighting effect module is used to automatically adjust the color and color temperature of lamps.

[0011] As a further solution of the present invention, the working process of the lighting control module is as follows:

[0012] S1. Randomly generate horizontal and vertical rotation (pan, tilt) commands, and adjust the horizontal rotation angle pan and vertical rotation angle tilt of the lamp according to the commands;

[0013] S2. Obtain the monitoring image, identify the light spot and calculate the coordinates (x, y) of the center position of the light spot. If the light spot cannot be identified, repeat step S1;

[0014] S3. Using the position coordinates (x, y) in the monitoring image as the model input and the light command (pan, tilt) values ​​as the model output, a model is built to obtain an N-layer neural network model including an input layer, N hidden layers, and an output layer;

[0015] S4. Obtain the optimal model based on the prediction results of the neural network model and deploy it. Click on the location where the light needs to be illuminated in the video surveillance image and obtain the selected location information.

[0016] S5. Input the position information into the neural network model, calculate and predict the output (pan, tilt) command value through the neural network forward propagation, generate the light control signal through the command value, and illuminate the light to the specified position.

[0017] As a further solution of the present invention, the step of moving the light spot to a specified position in the monitoring image specifically includes:

[0018] S1. Obtain the pixel position coordinates (Xe, Ye) selected in the monitoring image;

[0019] S2. Obtain the coordinates of the light spot position through the monitoring screen;

[0020] S3. Get the light moving step length,

[0021] The coordinates of the light spot are recorded as (X1, Y1). The system issues the command pan+1 to increase the horizontal rotation of the lamp by one degree. Repeat S2 to obtain the adjusted coordinates of the light spot as (X2, Y2).

[0022] The movement vector of the corresponding light spot on the screen for each degree of pan increase is (Xp, Yp);

[0023] Where Xp = X2 - X1, Yp = Y2 - Y1;

[0024] The coordinates of the light spot are recorded as (X1ˋ, Y1ˋ). The system issues the command tilt+1 to increase the vertical rotation of the lamp by one degree. Repeat S2 to obtain the adjusted coordinates of the light spot as (X2ˋ, Y2ˋ).

[0025] The movement vector of the light spot on the screen for each degree of tilt increase is (Xt, Yt);

[0026] Where Xt = X2ˋ-X1ˋ, Yt = Y2ˋ-Y1ˋ;

[0027] S4. Calculate the values ​​P and T that need to be moved for pan and tilt respectively. The solution formula is:

[0028] Xe=P*Xp+T*Xt

[0029] Ye=P*Yp+T*Yt

[0030] According to the values ​​of P and T, commands are issued to adjust the horizontal and vertical rotation angles of the lamp to move the light spot to the specified position;

[0031] S5. Repeat S2 to obtain the light spot coordinates (X1, Y1). If |Xe-X1| and |Ye-Y1| are both within the threshold, the light spot has moved to the target position. Otherwise, repeat S3.

[0032] As a further solution of the present invention, obtaining the light spot position coordinates includes:

[0033] By flashing the light and comparing the grayscale images when the light is turned off and on, the pixel blocks where the brightness difference exceeds the threshold are located at the positions covered by the light spot. The center of the shape is calculated to obtain the coordinates of the light spot position.

[0034] As a further solution of the present invention, obtaining the light spot position coordinates includes:

[0035] Enter the macro command to reset all blades to keep the light spot circular;

[0036] Obtain surveillance footage and apply Hough circle transform to identify circular light spots in the footage.

[0037] As a further solution of the present invention, the workflow of the light auto-focus module is as follows:

[0038] S1. Obtain the distance from the light to the light receiving surface (distance) and the parameters set by the user, including: focal length scaling (zoom), aperture size (iris) and the on / off status of the disk (gobo1, gobo2, gobo3);

[0039] S2. Build an Actor model, integrate all the above data as input to the Actor model, and calculate the focus value;

[0040] S3. Generate a focus adjustment command based on the focus value, and adjust the lamp to achieve the focal length specified by the calculation module to complete the focus operation.

[0041] As a further solution of the present invention, the step of expanding the light spot size until the light spot covers the target shape specifically includes:

[0042] S1. Draw the target shape on the monitoring screen;

[0043] S2. Output macro commands to adjust the zoom and iris to minimize the light spot.

[0044] S3, obtaining the polygon vertex position coordinates of the target shape, and obtaining the average grayscale value of the pixels in the area around each vertex;

[0045] S4, output macro commands to adjust the zoom value and iris value to gradually enlarge the light spot;

[0046] S5, judging whether the grayscale value increment of the area around each vertex exceeds a threshold value, if it exceeds the threshold value, the target polygon is covered by the light spot;

[0047] If the threshold is not exceeded, S3 and S4 are repeated until the grayscale value increment exceeds the threshold or the light spot cannot be enlarged any further;

[0048] If the grayscale value does not change, it means that the smallest spot has initially covered the target shape or that the largest spot cannot cover the target;

[0049] If it indicates that the minimum spot has initially covered the target shape, then S1 is repeated;

[0050] If the maximum spot cannot cover the target, the spot cutting operation is abandoned.

[0051] As a further solution of the present invention, the step of cutting the light spot until the light spot is cut into a target shape specifically includes:

[0052] S1. Output macro command to set the light spot to a rectangle and identify the horizontal rotation angle of the light spot;

[0053] S2, using edge detection and contour finding algorithms to identify the coordinates of the rectangular outline of the light spot, and using a rotating caliper algorithm to obtain the horizontal rotation angle of the rectangular shape of the light spot;

[0054] S3, using the rotating caliper algorithm to obtain the minimum circumscribed rectangle of the target shape and its horizontal rotation angle;

[0055] S4, calculating the rotation angle difference between the light spot rectangle and the minimum circumscribed rectangle of the target shape, and controlling the rotation angle of the blade disk so that the light spot rotation angle matches the target shape;

[0056] S5. Execute the spot cutting operation and determine whether the spot fits the target shape by detecting the change in pixel brightness.

[0057] As a further solution of the present invention, the workflow of the lighting effect module is as follows:

[0058] The color of the current light in the picture captured by the camera is converted into the color space CMY by RGB

[0059] Calculate the error values ​​separately for the cyan, magenta, and yellow channels:

[0060] e(t) = target channel value - actual channel value

[0061] The target channel value and the actual channel value are obtained by color decomposition;

[0062] Use the PID control formula to calculate the parameters of each color channel:

[0063] u(t)=K p ·e(t)+K i ·∫e(t)dt+K d ·dtde(t)

[0064] Among them: K p , K i , K d are the PID parameters of each channel, u(t) is the adjusted output value used to adjust the light color, and e(t) is the calculated error;

[0065] According to the output u(t) of the PID controller, a lighting control command is issued to adjust the parameter value of each color channel and modify the light color output.

[0066] The beneficial effects of the present invention are:

[0067] This solution proposes a method that combines manual operation with reinforcement learning. It automatically completes shape cutting according to the lighting engineer's instructions, saving labor. The model's performance is not affected by the operating environment. The model operates even if the camera is blocked, the shape and color of the light spot changes, and the overall lighting on the scene changes. The hardware is simple, easy to deploy and maintain, and the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a block diagram of the AI-driven multi-modal lighting control system provided by an embodiment of the present invention;

[0069] Figure 2 A schematic diagram of the structure provided by an embodiment of the present invention;

[0070] Figure 3 This is an example diagram of a neural network model;

[0071] Figure 4 This is the structural diagram of the transfer learning solution;

[0072] Figure 5 This is a structural block diagram of the light auto-focus module provided by an embodiment of the present invention;

[0073] Figure 6 This is the relationship diagram between Zoom and Focus with Gobo focus;

[0074] Figure 7 This is the relationship diagram between Zoom and Focus without Gobo focus. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0076] It is understood that the terms "first," "second," etc., used herein may be used to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script without departing from the scope of this application.

[0077] like Figure 1 and Figure 2 As shown, an AI-driven multimodal lighting control system includes:

[0078] The lighting control module is used to select a designated position from the monitoring screen and move the light spot to the designated position in the monitoring screen;

[0079] Light auto-focus module, used to automatically focus the light spot after it moves to the specified position;

[0080] The light spot cutting module is used to set the target shape in the monitoring screen, expand the light spot size until the light spot covers the target shape, and cut the light spot until the light spot is cut into the target shape;

[0081] The lighting effect module is used to automatically adjust the color and color temperature of lamps.

[0082] like Figure 3 As shown, in this embodiment of the present invention, the working process of the lighting control module is as follows:

[0083] S1. Randomly generate horizontal and vertical rotation (pan, tilt) commands, and adjust the horizontal rotation angle pan and vertical rotation angle tilt of the lamp according to the commands;

[0084] S2. Obtain the monitoring image, identify the light spot and calculate the coordinates (x, y) of the center position of the light spot. If the light spot cannot be identified, repeat step S1;

[0085] S3. Using the position coordinates (x, y) in the monitoring image as the model input and the light command (pan, tilt) values ​​as the model output, a model is built to obtain an N-layer neural network model including an input layer, N hidden layers, and an output layer;

[0086] S4. Obtain the optimal model based on the prediction results of the neural network model and deploy it. Click on the location where the light needs to be illuminated in the video surveillance image and obtain the selected location information.

[0087] S5. Input the position information into the neural network model, calculate and predict the output (pan, tilt) command value through the neural network forward propagation, generate the light control signal through the command value, and illuminate the light to the specified position.

[0088] In the process of building a neural network model, the model needs to be trained.

[0089] 1. Hyperparameter selection:

[0090] Number of hidden layer neurons: Use cross-validation to determine the number of hidden layers and the number of neurons in each layer. Initially, set a small number to avoid overfitting, and gradually increase the number of neurons to improve the model's learning ability until performance on the validation set no longer improves.

[0091] Learning rate: Use an adaptive learning rate adjustment strategy, such as learning rate decay or the Adam optimizer, to achieve fast convergence in the early stages of training and stable learning in the later stages of training.

[0092] Batch Size: Choose an appropriate batch size based on available computing resources. Smaller batches can provide more frequent model updates, while larger batches may be beneficial for memory optimization and faster computation.

[0093] Activation function: By comparing the effects of functions such as ReLU, Sigmoid, and Tanh at different levels, select the activation function that can produce the best training results.

[0094] Epochs: The number of iterations is determined by early stopping. If the performance of the validation set does not improve significantly over several consecutive epochs, training is stopped to prevent overfitting.

[0095] Regularization: To prevent overfitting, L2 regularization is applied to the weights, and an appropriate regularization coefficient is selected to balance model complexity and generalization ability.

[0096] Weight initialization: Use He initialization or Xavier initialization to optimize the weights, providing a good starting point for the training process and helping the gradient descent algorithm work more efficiently.

[0097] 2. Training process:

[0098] After determining the hyperparameters, neural network training begins. This training uses forward propagation to calculate the predicted output, followed by backward propagation to update the weights. At each iteration, a loss function (such as mean squared error) is calculated and optimized using the gradient descent algorithm. During training, hyperparameters such as the learning rate are adjusted to monitor and optimize model performance.

[0099] 3. Model evaluation and saving:

[0100] The model is evaluated using a test dataset to determine its generalization ability. The parameters of the best-performing model configuration are saved for use in actual luminaire position adjustment.

[0101] Furthermore, the following methods may be used in combination or individually to identify the light spot position in the image.

[0102] 1. Shape recognition method

[0103] A. Output blade signal to return the light shielding blade to its initial position to ensure the ideal circular light spot output.

[0104] B. Image preprocessing: Obtain camera surveillance footage. Convert the image to grayscale to reduce computational complexity. Apply a filtering algorithm to remove image noise. Perform binarization to highlight shape outlines.

[0105] C. Edge detection: Find the boundaries of shapes in the image through edge detection algorithms (such as canny edge detector).

[0106] D. Circular detection: Use the Hough transform algorithm to identify circular spots in the image and calculate the center coordinates.

[0107] 2. Flicker method to identify the center point of the light spot in the monitoring image

[0108] A. Output light brightness signal (fixture@100%). The light control unit adjusts the light to the brightest.

[0109] Get the monitoring screen and save it as image 1.

[0110] B. Output the light brightness signal (fixture@0%). The light control unit adjusts the light to the lowest setting. Capture the monitoring screen and save it as image 2.

[0111] C. Perform grayscale conversion and noise reduction on images 1 and 2.

[0112] D. Compare the processed images 1 and 2, set an appropriate threshold, and convert the difference image into a binary image, where the spot area is white and the rest is black.

[0113] E. Find the white area in the binary image, calculate the geometric center of the white area, and use the average value of the pixel coordinates to determine the center position of the light spot.

[0114] 3. Color contrast

[0115] Light color setting and image capture

[0116] A. Output light color adjustment commands. The light control unit sequentially sets the light to red, green, and blue. For each color lighting state, capture the monitoring screen and save it as Image 1, Image 2, and Image 3, respectively.

[0117] B. Color space conversion and image preprocessing

[0118] C. Convert the captured RGB image to HSV or LAB color space, which helps in more efficient color segmentation and analysis.

[0119] D. Apply histogram equalization to each color space converted image to increase image contrast so that color differences can be more clearly identified.

[0120] Gap Analysis

[0121] E. Use image processing techniques (such as difference images and color histogram comparison) to analyze color variations between different images. Extract the areas with significant changes in the spot, which show the greatest differences under different color illumination.

[0122] Binarization and morphological processing

[0123] F. Based on the results of the difference analysis, automatically calculate the optimal binarization threshold (e.g., using the Otsu method) and convert the difference image into a binary image, where the spot area is white and the rest is black.

[0124] G. Perform morphological operations on the binary image, such as opening or closing operations, to remove small noise points and smooth the spot boundary to further refine the spot shape.

[0125] Light spot center positioning

[0126] H. Find connected white regions in the binary image. Apply geometric analysis to each white region, calculate its geometric center, and use the average value of pixel coordinates or the least squares method to fit a circle to determine the exact center position of the light spot.

[0127] like Figure 4 As shown in the figure, in actual usage scenarios, if the camera position changes, the accuracy of the pre-trained model will be affected.

[0128] Model transfer learning method

[0129] When the spatial positions of the camera, work surface, and lamp change, the statistical characteristics of the (pan, tilt) and (x, y) data will change, resulting in a decrease in the accuracy of the model prediction.

[0130] This solution uses transfer learning to solve this problem. The details are as follows:

[0131] Fine-tuning method:

[0132] Select layers: Decide which layers need to be frozen and which layers need to be updated.

[0133] Training parameters: Train the unfrozen layers using the new dataset, adjusting the learning rate and batch size to optimize the training process.

[0134] Adapter model method:

[0135] Adapter model g(x', y'), which is able to map new input (x', y') back to the input space (x, y) of the original model.

[0136] Model construction: Define the structure of the Adapter model g(x', y') and determine the number and type of neural network layers required.

[0137] Train the Adapter model: While keeping the original model f(x, y) parameters fixed, only use the new data to train the Adapter model.

[0138] Integration: Connect the output layer of the adapter model with the input layer of the original model to ensure that the overall model can seamlessly handle the changed input data.

[0139] Model performance is unaffected by the operating environment. After model training, the predicted pan and tilt values ​​are independent of any external environment. This is unaffected by camera obstructions, changes in the shape and color of light spots, and changes in the overall lighting conditions.

[0140] This solution only requires adding cameras, and the hardware is simple, easy to deploy and maintain, with low cost.

[0141] Compared with alternative solutions, this method can illuminate the target area with the lamp in one step without the need to calculate the (pan, tilt) values ​​multiple times for gradual adjustment.

[0142] Alternatives

[0143] 1. Method based on vector calculation.

[0144] Step 1: Obtain the pixel position coordinates (Xe, Ye) selected by the operator in the monitoring image.

[0145] Step 2: Obtain the light spot position coordinates through the light spot recognition solution in the above solution.

[0146] Step 3: Calculate the light movement step size according to the following method.

[0147] Let the light spot coordinates be (X1, Y1). The system issues the command (pan+1) to increase the horizontal rotation of the fixture by one degree. Repeat step 2 to obtain the adjusted light spot coordinates (X2, Y2). Each degree of pan increase corresponds to the light spot's movement vector on the screen being (Xp, Yp). Where Xp = X2 - X1, and Yp = Y2 - Y1. Increased computational effort is required to achieve higher accuracy.

[0148] Similarly, the vertical rotation (tilt+1) of the lamp corresponds to the moving vector (Xt, Yt)

[0149] Calculate the required pan and tilt values ​​P and T. Xe = P*Xp + T*Xt, Ye = P*Yp + T*Yt. Solve to get P and T.

[0150] When the pan and tilt commands are issued, the control unit adjusts the horizontal and vertical rotation angles of the lamp to move the lamp's illumination position.

[0151] Step 5: The system repeats step 2 to obtain the light spot coordinates (X1, Y1). If |Xe-X1| and |Ye-Y1| are both within the threshold, the light spot has moved to the target position. Otherwise, repeat step 3.

[0152] 2. Vector Calculation Method of Integrated PID Algorithm

[0153] Step 1: Get the target location

[0154] Get the pixel position coordinates (Xe, Ye) in the monitoring image. These coordinates represent the target position where the light spot needs to be moved.

[0155] Step 2: Identify the light spot position

[0156] The position coordinates (X1, Y1) of the current light spot are obtained through the light spot recognition scheme. This step can adopt the light spot recognition method described in the above scheme.

[0157] Step 3: PID controller integration and error calculation

[0158] Proportional (P): Calculate the current error ex=Xe-X1ex=Xe-X1 and ey=Ye-Y1ey=Ye-Y1, and multiply it by the proportional coefficient KpKp.

[0159] Integration (I): Accumulate historical errors, update the integral terms IxIx and IyIy, and multiply them by the integral coefficient KiKi.

[0160] Differential (D): Calculate the error change rate, that is, the difference from the error in the previous cycle, and then multiply it by the differential coefficient KdKd.

[0161] Add the proportional, integral, and differential outputs to get the final control signal

[0162] ux=Px+Ix+Dxux=Px+Ix+Dx

[0163] uy=Py+Iy+Dyuy=Py+Iy+Dy

[0164] Step 4: Execute control instructions

[0165] The control unit sends corresponding pan+ux and tilt+uy commands to adjust the horizontal and vertical rotation angles of the lamp according to the output of the PID controller.

[0166] The lamp adjusts its position according to the control signal to move the light spot to the target position.

[0167] Step 5: Position confirmation and feedback adjustment

[0168] The system repeats step 2 to obtain the adjusted light spot coordinates (X1, Y1).

[0169] Check the error between the light spot position and the target position. If both |Xe-X1|||Xe-X1|| and |Ye-Y1|||Ye-Y1|| are within the preset threshold range, it is confirmed that the light spot has been correctly positioned.

[0170] If the error exceeds the threshold, the system recalculates the control signal based on the current error and returns to step 4 to continue adjustment until the error meets the requirements.

[0171] In an embodiment of the present invention, obtaining the light spot position coordinates includes:

[0172] By flashing the light and comparing the grayscale images when the light is turned off and on, the pixel blocks where the brightness difference exceeds the threshold are located at the positions covered by the light spot. The center of the shape is calculated to obtain the coordinates of the light spot position.

[0173] In an embodiment of the present invention, obtaining the light spot position coordinates includes:

[0174] Enter the macro command to reset all blades to keep the light spot circular;

[0175] Obtain surveillance footage and apply Hough circle transform to identify circular light spots in the footage.

[0176] like Figures 5 to 7 As shown, in the embodiment of the present invention, the working process of the light auto-focus module is as follows:

[0177] S1. Obtain the distance from the light to the light receiving surface (distance) and the parameters set by the user, including: focal length scaling (zoom), aperture size (iris) and the on / off status of the disk (gobo1, gobo2, gobo3);

[0178] S2. Build an Actor model, integrate all the above data as input to the Actor model, and calculate the focus value;

[0179] S3. Generate a focus adjustment command based on the focus value, and adjust the lamp to achieve the focal length specified by the calculation module to complete the focus operation.

[0180] The focusing process includes four parts: building an enhanced learning environment, model training, intelligent lighting control, and feedback loop.

[0181] Enhanced learning environment construction

[0182] The construction of the enhanced learning environment includes state parameters, reward calculation method design, and action space setting.

[0183] State parameters

[0184] The state of the system is defined by the following elements: distance, zoom, iris, and the state of the three gobos (gobo1, gobo2, gobo3). Here, distance, zoom, and iris are continuous floating point values, while the state of the gobo is a Boolean value (1 for gobo on, 0 for gobo off).

[0185] Reward calculation method

[0186] The reward is based on the clarity of the light spot edge. First, we collect monitoring images from different states of manual focus adjustment by the lighting operator and use the Sobel operator to calculate the edge clarity (i.e., grayscale gradient) of these images. We manually set an acceptable grayscale gradient range [g_min, g_max] as the clarity standard. In real-time operation, if the grayscale gradient produced by the AI-adjusted focus falls within this range, the AI ​​will receive a reward; otherwise, the reward is 0.

[0187] Action space setting

[0188] Action Space: The focus values ​​that the AI ​​agent can adjust are defined as a continuous range. For example, if the focus specification is from 10mm to 100mm, then the action space would be [10, 100]. This setting allows the AI ​​to find the optimal focus value within a given range.

[0189] Model training

[0190] This method model adopts the DDPG algorithm with Actor-Critic structure.

[0191] 1. Network Architecture

[0192] Actor network: Consists of multiple fully connected layers. The input is the system state [distance, zoom, iris, gobo1, gobo2, gobo3], and the output is a continuous action value, i.e., the focus parameter. A ReLU activation function is used to increase nonlinearity, and the final layer uses a tanh or sigmoid function to ensure that the output action is within a reasonable range.

[0193] Critic network: Also composed of multiple fully connected layers, the Critic network takes as input the state and the action value generated by the actor, and outputs the expected reward value for this state-action pair. The Critic network helps evaluate the decisions of the Actor network.

[0194] 2. Training Methods

[0195] Critic Network Update

[0196] The goal of the critic network is to approximate the true Q function, which gives the expected reward of taking action a in a given state s. The parameters of the critic network are updated by minimizing the loss function L, which measures the error between the Q value predicted by the critic network and the target Q value. The loss function L has the following form:

[0197] L(θ Q )=E s,a,r,s′ [(Q(s,a|θ Q )-y) 2 ]

[0198] Among them, θ Q is the parameter of the Critic network, y is the target Q value, and is calculated as follows:

[0199] y=r+γQ′(s′,μ′(s′|θμ′)|θQ′);

[0200] s′ is the new state after taking action a. γ is a discount factor used to reduce the impact of future rewards. Q′ and μ′ are the predictions of the target critic network and the target actor network, respectively. θQ′ and θμ′ are the parameters of the target critic network and the target actor network, respectively.

[0201] Actor Network Update The Actor Network is responsible for directly outputting the optimal action a. The parameters θμ of the Actor Network are updated by the following gradient ascent rule to maximize the expected reward estimated by the Critic Network:

[0202]

[0203] It is the gradient of the Critic network about action a, which evaluates the effect of taking the action recommended by the current Actor network in state s.

[0204] is the gradient of the Actor network output with respect to its parameters.

[0205] Smart lighting control

[0206] Ranging module operation

[0207] Function: Automatically obtain the distance from the light to the light-receiving surface.

[0208] Data transmission: The measured distance data is transmitted to the calculation module in real time.

[0209] User interaction module settings

[0210] Function: Allows users to set other parameters of the light through an interactive interface, including zoom (focal length scaling ratio), iris (aperture size), and the status (on or off) of the three disks (gobo1, gobo2, gobo3).

[0211] Data transmission: Transmit the parameter values ​​set by the user to the calculation module.

[0212] Computing module processing

[0213] Data integration: After the calculation module receives data from the ranging module and the user interaction module, it integrates this data as the input of the Actor model.

[0214] Model operation: The input data includes [distance, zoom, iris, gobo1, gobo2, gobo3]. The calculation module uses the trained Actor model to calculate the corresponding focus value.

[0215] Transmit focus command

[0216] Command sending: The calculation module generates a focus adjustment command based on the output of the Actor model and sends this command to the lighting control module.

[0217] Lighting control module focus

[0218] Execute focus: After receiving the focus command, the lighting control module adjusts the lamp to achieve the focal length specified by the calculation module to complete the focus operation.

[0219] Feedback loop lighting focus

[0220] Autofocus: The calculation module uses the trained Actor model to automatically calculate the focus value, and adjusts the lighting through the lighting control module to achieve preliminary focus.

[0221] Manual fine-tuning: If the lighting engineer finds that the focus effect is not ideal during actual use, he can manually enter a new focus command through the interactive module for fine-tuning. This allows the lighting engineer to make fine adjustments based on the actual situation on site to achieve the best focus effect.

[0222] Data collection and storage

[0223] Data storage module: All lighting setting parameters [distance, zoom, iris, gobo1, gobo2, gobo3] and the focus value manually adjusted by the lighting engineer will be recorded and stored in the data storage module. This data is marked as a labeled dataset for subsequent model training and optimization.

[0224] Data structure: Each piece of data includes complete state parameters and the corresponding adjusted focus value, forming a structured data set.

[0225] Continuous optimization of the model

[0226] Periodic fine-tuning: Use these labeled datasets to fine-tune the Actor model at certain time intervals or when the accumulated data reaches a certain amount.

[0227] Fine-tuning process: Use transfer learning and subtle learning rates to adjust model parameters.

[0228] When the distance between the light and the illuminated surface is 4.475m, Figure 6 As shown, the parameters of the fitted polynomial model are: f(x)=ax6+bx5+cx4+dx3+ex2+fx+g

[0229] The parameters are as follows:

[0230] a=-5.96×10-11a=-5.96×10-11

[0231] b=2.82×10-8b=2.82×10-8

[0232] c=-6.80×10-6c=-6.80×10-6

[0233] d=9.64×10-4d=9.64×10-4

[0234] e=-7.82×10-2e=-7.82×10-2

[0235] f=3.07f=3.07

[0236] g = -11.49 g = -11.49

[0237] like Figure 7 As shown in the figure, by analyzing the data relationship between Focus and Zoom, it is found that in the interval of Zoom>50, Focus=-0.334×Zoom+91.63. In the interval of 30 to 50, Focus=73.7

[0238] In the embodiment of the present invention, the step of expanding the light spot size until the light spot covers the target shape specifically includes:

[0239] S1. Draw the target shape on the monitoring screen;

[0240] S2. Output macro commands to adjust the zoom and iris to minimize the light spot.

[0241] S3, obtaining the polygon vertex position coordinates of the target shape, and obtaining the average grayscale value of the pixels in the area around each vertex;

[0242] S4, output macro commands to adjust the zoom value and iris value to gradually enlarge the light spot;

[0243] S5, judging whether the grayscale value increment of the area around each vertex exceeds a threshold value, if it exceeds the threshold value, the target polygon is covered by the light spot;

[0244] If the threshold is not exceeded, S3 and S4 are repeated until the grayscale value increment exceeds the threshold or the light spot cannot be enlarged any further;

[0245] If the grayscale value does not change, it means that the smallest spot has initially covered the target shape or that the largest spot cannot cover the target;

[0246] If it indicates that the minimum spot has initially covered the target shape, then S1 is repeated;

[0247] If the maximum spot cannot cover the target, the spot cutting operation is abandoned.

[0248] In the embodiment of the present invention, the step of cutting the light spot until the light spot is cut into a target shape specifically includes:

[0249] S1. Output macro command to set the light spot to a rectangle and identify the horizontal rotation angle of the light spot;

[0250] S2, using edge detection and contour finding algorithms to identify the coordinates of the rectangular outline of the light spot, and using a rotating caliper algorithm to obtain the horizontal rotation angle of the rectangular shape of the light spot;

[0251] S3, using the rotating caliper algorithm to obtain the minimum circumscribed rectangle of the target shape and its horizontal rotation angle;

[0252] S4, calculating the rotation angle difference between the light spot rectangle and the minimum circumscribed rectangle of the target shape, and controlling the rotation angle of the blade disk so that the light spot rotation angle matches the target shape;

[0253] S5. Execute the spot cutting operation and determine whether the spot fits the target shape by detecting the change in pixel brightness.

[0254] The spot cutting module includes three operating stages:

[0255] In the first stage, the light spot is positioned at the geometric center of the target shape. In the second stage, the light spot is expanded to cover the target shape. In the third stage, the AI ​​cuts the light spot to the target shape.

[0256] Stage 1: Move the light spot to the geometric center of the target shape:

[0257] 1. Adjust zoom and iris on the lighting control unit to minimize the light spot.

[0258] 2. Calculate the geometric center coordinates of the drawn shape.

[0259] 3. Direct the light to the center of the shape manually or using the method described in “A method and system for intelligently positioning the direction of light illumination based on computer vision”.

[0260] Furthermore, the geometric center coordinates can be achieved by calculating the centroid of the shape:

[0261] 4. Get the shape outline coordinates

[0262] 5. Calculate the moments of the contour

[0263] 6. Calculate the center of mass: Use the moment formula to calculate the center of mass of the shape. The x and y coordinates of the center of mass can be calculated using the moment M: Cx = M10 / M00, Cy = M01 / M00

[0264] Among them, M10 and M01 are first-order moments, and M00 is the zero-order moment, that is, the area.

[0265] Phase 2: Expand the spot size to cover the target shape:

[0266] 1. First, obtain the positions of each vertex of the target polygon and calculate the average grayscale value of the 3x3 pixel area centered on each vertex. This value is defined as the "initial grayscale value."

[0267] 2. Send a macro command to the light control unit, instructing it to adjust the zoom and iris parameters of the light spot so as to gradually enlarge the light spot.

[0268] 3. After enlarging the light spot, re-measure the new grayscale value of the above vertex area and calculate the difference between the new grayscale value and the initial grayscale value.

[0269] If the difference exceeds a preset threshold n, it is considered that the target polygon has been successfully covered by the light spot.

[0270] If the difference does not exceed the threshold n, the process returns to step 2 and repeats the adjustment of the spot size.

[0271] 4. When the light spot cannot be further enlarged (reaching the maximum zoom and iris values ​​allowed by the device), if the change in grayscale value still does not exceed the threshold n, the light spot is considered to have covered the target polygon at its initial size. At this time, a macro command will be issued to the lighting control unit, instructing it to adjust the light spot to its initial or minimum setting.

[0272] Phase 3: AI cuts the light spot to the target shape

[0273] 1. Through reinforcement learning, AI is trained to operate 8 blades for cutting and overall rotation, and control the [1A, 1B, 2A, 2B, 3A, 3B, 4A, 4B, rotate] command to achieve shape cutting.

[0274] 2. This part further includes four parts: building an enhanced learning environment, model training, intelligent lighting control, and feedback loop, which are implemented using the following methods:

[0275] a. Enhanced learning environment construction

[0276] The construction of the enhanced learning environment includes state parameters, reward calculation method design, and action space setting.

[0277] State parameters

[0278] The state of the system is the target contour pixel coordinates Box[(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…], the spot edge coordinates Contour: [(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…]

[0279] b.Reward calculation method

[0280] Set two pixels inside and outside each edge of the target shape as detection points. Record the initial brightness of these detection points and set a brightness drop threshold of x% to determine whether they are bright or dark. The shape is considered cut until all outer detection points on the edge are dark and all inner detection points are bright. When the shape is cut, the AI ​​agent is rewarded with a large amount. Otherwise, the reward is 0.

[0281] c. Action space setting

[0282] Action space: The eight blade position values ​​that the AI ​​agent can adjust:

[0283] [1A,1B,2A,2B,3A,3B,4A,4B,rotate], where each blade value is defined as a continuous interval. For example, 1A∈[0,100]. This setting allows the AI ​​to find the optimal blade position within a given range.

[0284] Specifically speaking, the intelligent lighting control part includes:

[0285] 1. Computational processing

[0286] Data integration: After obtaining the data from the camera image and the user interaction module, integrate this data as the input of the Actor model.

[0287] Model operation: The input data includes the target contour pixel coordinates Box[(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…], the spot edge coordinates Contour: [(x,y),(x,y),(x,y),(x,y),(x,y),(x,y)…]

[0288] The calculation module uses the trained Actor model to calculate the corresponding eight blade position values ​​[1A, 1B, 2A, 2B, 3A, 3B, 4A, 4B, rotate].

[0289] 2. Transmit the focus command

[0290] Command sending: The calculation module generates a focus adjustment command based on the output of the Actor model and sends this command to the lighting control module.

[0291] Lighting control module focus

[0292] Execute focus: After receiving the command, the lighting control module adjusts the position of the lamp blade to complete the cutting operation.

[0293] 3. Feedback Loop

[0294] 1. Light cutting

[0295] Automatic cutting: The calculation module uses the trained Actor model to automatically calculate the blade and rotation parameters, and adjusts the lighting through the lighting control module to achieve shape cutting.

[0296] Manual fine-tuning: If the lighting engineer finds that the cutting effect is not ideal, he can manually enter a new cutting command through the interactive module for fine-tuning. This allows the lighting engineer to make fine adjustments based on the actual situation on site to achieve the best focus effect.

[0297] 2. Data Collection and Storage

[0298] Data storage module: All state parameters and action values ​​manually adjusted by the lighting engineer will be recorded and stored in the data storage module. These data are marked as labeled datasets for subsequent model training and optimization.

[0299] Data structure: Each piece of data includes complete state parameters and the corresponding adjusted action value, forming a structured data set.

[0300] 3. Continuous optimization of the model

[0301] Periodic fine-tuning: Use these labeled datasets to fine-tune the Actor model at certain time intervals or when the accumulated data reaches a certain amount.

[0302] Fine-tuning process: Use transfer learning and subtle learning rates to adjust model parameters.

[0303] There are also alternatives to the above solutions:

[0304] Prerequisite: The user draws an ideal shape on the monitoring screen through the interactive unit, such as a trapezoid.

[0305] This method consists of three operational phases:

[0306] In the first stage, the light spot is moved to several center positions of the target shape.

[0307] In the second stage, the light spot is expanded to cover the target shape.

[0308] In the third stage, the moving blade cuts the light spot to the target shape.

[0309] The main difference from the previous solution is in the third stage. Here we only introduce the differences in the third stage:

[0310] Stage 3: Cutting to target shape

[0311] Step 1: Set the spot shape

[0312] Output commands to the control unit, instructing it to adjust the cutting light so that it expands to 25% in each of the four main directions to form an approximately rectangular light spot.

[0313] Step 2: Identify the light spot shape and its rotation angle

[0314] 1. Use the camera to obtain the monitoring image of the current working area, and use the edge detection and contour finding algorithm to identify the outline coordinates of the rectangular light spot.

[0315] 2. Use the rotating caliper algorithm to calculate the horizontal rotation angle of the light spot.

[0316] Step 3: Get the target shape parameters

[0317] Output commands to the control unit, use the rotating caliper algorithm to identify and obtain the minimum circumscribed rectangle of the target shape and its horizontal rotation angle.

[0318] Step 4: Match the rotation angle of the light spot and the target shape

[0319] The rotation angle difference between the light spot and the target shape is calculated (denoted as x). A command is then output to the control unit to adjust the blade disk rotation angle x to match the rotation angle of the light spot to the target shape.

[0320] Step 5: Perform the Cut

[0321] 1. Set two pixels inside and outside each edge of the target shape as detection points and record the initial brightness values ​​of these detection points. Set a brightness drop threshold x% to determine light and dark.

[0322] 2. Automatically adjust the blade position based on the real-time collected detection point brightness:

[0323] If both the inner and outer detection points are lit, the blade cuts one step further inward.

[0324] If the outer detection point is dark and the inner detection point is bright, the blade stops.

[0325] If the inner detection point is dark, the outer detection point is also dark, and the blade moves back a small step toward the outside.

[0326] 3. Issue a control command, and the control unit repeats the above adjustment on the blade operation of each edge until the outer detection points of all edges are dark and the inner detection points are bright. At this time, the shape cutting is considered completed.

[0327] 4. If it is detected that the blade has reached its mechanical limit and cannot complete further cutting, the operation will be terminated according to the macro command value.

[0328] In an embodiment of the present invention, the workflow of the lighting effect module is as follows:

[0329] The color of the current light in the picture captured by the camera is converted into the color space CMY by RGB

[0330] Calculate the error values ​​separately for the cyan, magenta, and yellow channels:

[0331] e(t) = target channel value - actual channel value

[0332] The target channel value and the actual channel value are obtained by color decomposition;

[0333] Use the PID control formula to calculate the parameters of each color channel:

[0334] u(t)=K p ·e(t)+K i ·∫e(t)dt+K d ·dtde(t)

[0335] Among them: K p , K i , K d are the PID parameters of each channel, u(t) is the adjusted output value used to adjust the light color, and e(t) is the calculated error;

[0336] According to the output u(t) of the PID controller, a lighting control command is issued to adjust the parameter value of each color channel and modify the light color output.

[0337] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0338] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0339] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0340] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0341] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. AI-driven multi-modal lighting control system, characterized by: The system comprises: The lighting control module is used to select a designated position from the monitoring screen and move the light spot to the designated position in the monitoring screen; Light auto-focus module, used to automatically focus the light spot after it moves to the specified position; The light spot cutting module is used to set the target shape in the monitoring screen, expand the light spot size until the light spot covers the target shape, and cut the light spot until the light spot is cut into the target shape; Lighting effect module, used to automatically adjust the color and color temperature of lamps; The step of expanding the light spot size until the light spot covers the target shape specifically includes: S1. Draw the target shape on the monitoring screen; S2. Output macro commands to adjust the zoom and iris to minimize the light spot. S3, obtaining the polygon vertex position coordinates of the target shape, and obtaining the average grayscale value of the pixels in the area around each vertex; S4, output macro commands to adjust the zoom value and iris value to gradually enlarge the light spot; S5, judging whether the grayscale value increment of the area around each vertex exceeds a threshold value, if it exceeds the threshold value, the target polygon is covered by the light spot; If the threshold is not exceeded, S3 and S4 are repeated until the grayscale value increment exceeds the threshold or the light spot cannot be enlarged any further; If the grayscale value does not change, it means that the smallest spot has initially covered the target shape or that the largest spot cannot cover the target; If it indicates that the minimum spot has initially covered the target shape, then S1 is repeated; If it indicates that the maximum spot cannot cover the target, the spot cutting operation is abandoned; The working process of the lighting control module is as follows: S1. Randomly generate horizontal and vertical rotation (pan, tilt) commands, and adjust the horizontal rotation angle pan and vertical rotation angle tilt of the lamp according to the commands; S2. Obtain the monitoring image, identify the light spot and calculate the coordinates (x, y) of the center position of the light spot. If the light spot cannot be identified, repeat step S1; S3. Using the position coordinates (x, y) in the monitoring image as the model input and the light command (pan, tilt) values ​​as the model output, a model is built to obtain an N-layer neural network model including an input layer, N hidden layers, and an output layer; S4. Obtain the optimal model based on the prediction results of the neural network model and deploy it. Click on the location where the light needs to be illuminated in the video surveillance image and obtain the selected location information. S5. Input the position information into the neural network model, calculate and predict the output (pan, tilt) command value through the neural network forward propagation, generate the light control signal based on the command value, and illuminate the light to the specified position; The step of moving the light spot to a designated position in the monitoring screen specifically includes: S1. Obtain the pixel position coordinates (Xe, Ye) selected in the monitoring image; S2. Obtain the coordinates of the light spot position through the monitoring screen; S3. Get the light moving step length, The coordinates of the light spot are recorded as (X1, Y1). The system issues the command pan+1 to increase the horizontal rotation of the lamp by one degree. Repeat S2 to obtain the adjusted coordinates of the light spot as (X2, Y2). The movement vector of the corresponding light spot on the screen for each degree of pan increase is (Xp, Yp); Where Xp = X2 - X1, Yp = Y2 - Y1; The coordinates of the light spot are recorded as (X1ˋ, Y1ˋ). The system issues the command tilt+1 to increase the vertical rotation of the lamp by one degree. Repeat S2 to obtain the adjusted coordinates of the light spot as (X2ˋ, Y2ˋ). The movement vector of the light spot on the screen for each degree of tilt increase is (Xt, Yt); Where Xt = X2ˋ-X1ˋ, Yt = Y2ˋ-Y1ˋ; S4. Calculate the values ​​P and T that need to be moved for pan and tilt respectively. The solution formula is: Xe=P*Xp+T*Xt Ye=P*Yp+T*Yt According to the values ​​of P and T, commands are issued to adjust the horizontal and vertical rotation angles of the lamp to move the light spot to the specified position; S5. Repeat S2 to obtain the light spot coordinates (X1, Y1). If |Xe-X1| and |Ye-Y1| are both within the threshold, the light spot has moved to the target position. Otherwise, repeat S3. The obtaining of the light spot position coordinates comprises: By flashing the light and comparing the grayscale images when the light is turned off and on, the pixel blocks where the brightness difference exceeds the threshold are located at the positions covered by the light spot. The center of the shape is calculated to obtain the coordinates of the light spot position. The step of cutting the light spot until the light spot is cut into a target shape specifically includes: S1. Output macro command to set the light spot to a rectangle and identify the horizontal rotation angle of the light spot; S2, using edge detection and contour finding algorithms to identify the coordinates of the rectangular outline of the light spot, and using a rotating caliper algorithm to obtain the horizontal rotation angle of the rectangular shape of the light spot; S3, using the rotating caliper algorithm to obtain the minimum circumscribed rectangle of the target shape and its horizontal rotation angle; S4, calculating the rotation angle difference between the light spot rectangle and the minimum circumscribed rectangle of the target shape, and controlling the rotation angle of the blade disk so that the light spot rotation angle matches the target shape; S5, performing a spot cutting operation and determining whether the spot fits the target shape by detecting a change in pixel brightness; The workflow of the lighting effects module is as follows: The color of the current light in the picture captured by the camera is converted from RGB to CMY color space; Calculate the error values ​​separately for the cyan, magenta, and yellow channels: e(t) = target channel value - actual channel value; The target channel value and the actual channel value are obtained by color decomposition; Use the PID control formula to calculate the parameters of each color channel: u(t)=K p ·e(t)+K i ·∫e(t)dt+K d ·dtde(t); Among them: K p , K i , K d are the PID parameters of each channel, u(t) is the adjusted output value used to adjust the light color, and e(t) is the calculated error; According to the output u(t) of the PID controller, a lighting control command is issued to adjust the parameter value of each color channel and modify the light color output.

2. The system according to claim 1, wherein: The obtaining of the light spot position coordinates comprises: Enter the macro command to reset all blades to keep the light spot circular; Obtain surveillance footage and apply Hough circle transform to identify circular light spots in the footage.

3. The system according to claim 1, wherein: The workflow of the light autofocus module is as follows: S1. Obtain the distance from the light to the light receiving surface (distance) and the parameters set by the user, including: focal length scaling (zoom), aperture size (iris) and the on / off status of the disk (gobo1, gobo2, gobo3); S2. Build an Actor model, integrate all the above data as input to the Actor model, and calculate the focus value; S3. Generate a focus adjustment command based on the focus value, and adjust the lamp to achieve the focal length specified by the calculation module to complete the focus operation.

Images