An adaptive light adjustment system and method based on multi-modal non-inductive perception

Abstract

The application discloses a kind of adaptive light adjustment system and method based on multi-modal non-inductive perception, it is related to intelligent lighting, sensor fusion and visual technology field.System is constituted by four big modules of multi-modal sensor fusion, image processing and behavior identification, main control, light execution.Multi-modal sensor fusion module adopts millimeter wave radar and visual camera, radar exports distance, speed, direction, human micro-dynamic point cloud data, camera acquires scene color image;Behavior identification module fuses two kinds of data to identify human behavior.Main control module is built-in concentration evaluation model and light decision subunit, the former relies on identification result statistics effective concentration duration, the latter combines behavior information and concentration duration to call preset light strategy library, generates partition dimming instruction.Light execution module receives instruction and controls lighting device.The scheme fuses radar and visual advantage, relies on concentration dynamic partition dimming, realizes that lighting is adaptively regulated with user state.

Description

Technical Field

[0001] This invention relates to the fields of intelligent lighting, sensor fusion and vision technology, and in particular to an adaptive lighting adjustment system and method based on multimodal non-sensory perception. Background Technology

[0002] Currently, smart lighting is widely used in homes, offices, and schools. However, existing lighting control technologies still suffer from low levels of intelligence, poor adaptability, and a lack of health intervention capabilities, making it difficult to meet users' refined and personalized lighting needs. The current mainstream smart lighting control methods are mainly divided into three categories, all of which have significant technical shortcomings: manual and program-preset control modes require users to manually operate switches or apps, or rely on fixed programs, failing to dynamically adjust based on the user's real-time eye usage, resulting in insufficient proactive lighting control; traditional environmental sensing control only detects ambient light intensity through light sensors, adjusting brightness solely based on external environmental parameters, completely ignoring the user's human behavior, visual activity, and physical and mental state, thus failing to adapt to the user's actual eye usage needs; and simple human body sensing control based on infrared sensors can only determine whether someone is in the space, merely controlling the on / off state of the lights, making the control method extremely crude.

[0003] The existing technology has significant technical defects. First, it cannot seamlessly and accurately identify users' real-time behavior and status, resulting in a serious disconnect between lighting adjustment and users' actual needs. Second, in multi-user shared scenarios, it only supports uniform dimming across the entire area and cannot provide customized lighting for different users' differentiated visual tasks, which can easily lead to problems such as insufficient illuminance and glare. Third, the lighting function is limited, only achieving basic lighting without incorporating human eye health and biorhythm design, lacking proactive health intervention functions such as relieving eye fatigue and regulating rhythms. Its intelligence and practicality urgently need to be improved. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: an adaptive lighting adjustment system based on multimodal non-sensory perception, comprising an adaptive lighting adjustment system based on multimodal non-sensory perception, characterized in that it includes: Multimodal sensor fusion module: including millimeter-wave radar and vision camera. The millimeter-wave radar generates point cloud data containing the distance between the user and the lighting device, the user's moving speed, the user's location and the user's micro-motion information by transmitting and receiving electromagnetic waves. The vision camera is responsible for collecting color image information of the target area. Image processing and behavior recognition module: electrically connected to the multimodal sensor fusion module, and recognizes user behavior by processing the point cloud data and color image information; Main control module: electrically connected to the image processing and behavior recognition unit, receiving the recognition results of the image processing and behavior recognition unit and generating lighting adjustment instructions based on the recognition results. It has its own attention assessment model and lighting strategy decision subunit. The attention assessment model receives the recognition results, accumulates the effective attention time based on the recognition results, and sends it to the lighting strategy decision subunit. The lighting strategy decision subunit queries the preset lighting strategy database based on the recognition results and the effective attention time, and generates specific zone lighting adjustment instructions. The lighting control module is electrically connected to the main control module and adjusts the lighting devices according to the received lighting adjustment commands.

[0005] Preferably, the image processing and behavior recognition module includes: Human body detection and tracking subunit: Real-time detection and tracking of each user's position and contour from image sequences acquired from visual cameras, and assignment of a unique ID to each user; Key point detection subunit: For each detected human body, identify its skeletal key points, namely the head, shoulders, wrists and elbows; Behavior recognition subunit: Based on the sequence data formed by the identified skeletal key points, the detection and interaction relationship of the target object, and using a pre-trained behavior recognition model, it determines the specific behavior state of each user.

[0006] Preferably, the pre-trained behavior recognition model is a spatiotemporal graph convolutional network, and the workflow of the behavior recognition subunit includes the following steps: (1) Spatiotemporal graph data structure construction: The sequence data is constructed into a spatiotemporal graph G = (V, E), where: Node set V: Each skeletal keypoint in each frame is a node, and each node v i The eigenvectors are their two-dimensional coordinates (x, y), and confidence scores can be attached; the spatial boundary set E s Based on the natural anatomical structure of the human body, adjacent joints are connected within the same frame to form spatial edges; the temporal edge set E t Connect the same key point in adjacent frames to form a time edge; (2) Spatiotemporal graph convolutional network processing flow: The constructed spatiotemporal graph G is input into a pre-trained spatiotemporal graph convolutional network. The network extracts features by alternating between the following two operations: Spatial graph convolution: Convolution operation is performed on the skeleton graph of each frame to learn the spatial configuration features of human posture; Temporal convolution: One-dimensional temporal convolution is performed on the trajectory sequence of each joint along the time edge to capture the motion pattern of the joint over time; Through the stacking of multiple layers of spatial-temporal convolution, discriminative spatiotemporal features are gradually abstracted from the original coordinate sequence. (3) Classification and output: The last layer of the network is connected to a fully connected layer and a Softmax classifier, which maps the learned discriminative spatiotemporal features to the preset behavior categories. Finally, a probability distribution vector is output, which takes the behavior category with the highest probability as the behavior recognition result of the user in that time period, and outputs it together with the confidence score for subsequent attention evaluation model.

[0007] Preferably, the attention assessment model is a quantitative assessment system based on multi-feature weighted fusion and finite state machine, which is used to convert the raw data acquired by multimodal sensors into an attention index, and manage the user's state transition and health intervention trigger accordingly.

[0008] Preferably, the construction of the attention assessment model includes the following steps: The model input features and quantization formulas are constructed. The model input features include: behavioral persistence feature A, posture stability feature B, micro-motion feature C, and visual attention feature D. The quantization formula for persistence feature A is as follows: ,in To focus on action frame rate, The total number of frames; the quantization formula for the attitude stability feature B is: The quantification formula for the micro-motion feature C is as follows: ,in The normalized spectral entropy value; the quantization formula for the visual attention feature D is: To obtain the overall focus index .

[0009] Preferably, the decision-making method of the lighting strategy decision-making subunit specifically includes the following steps: (1) Construct a layered strategy library: The first layer is the basic scene strategy, which determines the scene's color temperature, illuminance, color rendering index, and whether the anti-glare mode is enabled as the basic light parameters based on the user's status. The second layer is a dynamic intervention strategy, which determines the duration of scene color temperature and illuminance based on the duration of the user's state and the frequency of state switching. (2) Query user status: Query the first-level policy library based on the user's current status to obtain the basic optical parameters. ; (3) Optimize user state and make decisions: Receive the state signal S1 output by the attention assessment model. If S1 does not contain intervention signals, then the final instruction is given. When S1 contains an intervention signal, the second-layer policy library is queried to obtain the dynamic light sequence. Final Instructions The signal is sent to the drive circuit; when S1 contains multiple intervention signals, the strategy priority weight and parameter mixing weight are defined to resolve conflicts, and the final parameters are generated by the weighted mixing of each active strategy.

[0010] Preferably, the discriminative spatiotemporal features refer to data representations that can distinguish different behaviors from the original joint coordinate sequence after multi-layer processing by a spatiotemporal graph convolutional network.

[0011] An adaptive lighting adjustment method based on multimodal non-sensory perception includes the following steps: (1) Perform multimodal sensor fusion: The multimodal sensor includes millimeter-wave radar and vision camera. The millimeter-wave radar generates point cloud data containing the distance between the user and the lighting device, the user's moving speed, the user's location and the user's micro-motion information by transmitting and receiving electromagnetic waves. The vision camera is responsible for collecting color image information of the target area. (2) Perform image processing and behavior recognition: Electrically connected to the multimodal sensor fusion module and recognize the user's behavior by processing the point cloud data and color image information; (3) Generate lighting adjustment instructions based on recognition results: It is electrically connected to the image processing and behavior recognition unit, receives the recognition results of the image processing and behavior recognition unit, and generates lighting adjustment instructions based on the recognition results. It has its own attention assessment model and lighting strategy decision subunit. The attention assessment model receives the recognition results, accumulates the effective attention time based on the recognition results, and sends it to the lighting strategy decision subunit. The lighting strategy decision subunit queries the preset lighting strategy database based on the recognition results and the effective attention time, and generates specific zone lighting adjustment instructions. (4) The lighting device executes the light adjustment command; it is electrically connected to the main control module and adjusts the lighting device according to the received light adjustment command.

[0012] By adopting the above solution, this invention has the following advantages and beneficial effects: Under the premise of strictly protecting user privacy, this invention accurately identifies user behavior and physiological states (such as reading, screen use, sitting still, and micro-movements) through a multimodal, non-intrusive method; based on the behavior state recognition results, it provides adaptive and personalized lighting parameter adjustments for different users or different behavior zones; it introduces a focus assessment model based on behavior duration, proactively initiating health interventions through lighting changes at appropriate times, guiding users to use their eyes scientifically, and improving the proactive health management capabilities of the lighting system. Attached Figure Description

[0013] Figure 1 This is a diagram illustrating the overall architecture of the system of the present invention; Figure 2 This is a flowchart of the system operation of the present invention.

[0014] Figure 3 This is a schematic diagram illustrating the execution of lighting adjustment commands by the present invention.

[0015] Figure 4 This is a schematic diagram illustrating the state transition in the focus assessment of this invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Reference manual attached Figure 1 The present invention provides an adaptive lighting adjustment system based on multimodal non-sensory perception, which mainly includes: Multimodal sensor fusion module: This module consists of a millimeter-wave radar and a vision camera working together. The millimeter-wave radar emits electromagnetic waves and receives the echoes, generating point cloud data containing target distance, velocity, orientation, and micro-motion information (such as breathing and heartbeat). The vision camera acquires color image information of the target area. Both are synchronized and calibrated through hardware to achieve spatiotemporally aligned data output. This module is deployed above or to the side of the illuminated area to ensure coverage of the entire user activity range. Its main function is to acquire continuous image sequences containing user posture, movements, and interactions with objects.

[0018] Image Processing and Behavior Recognition Unit: Electrically connected to the sensor module, it processes sensor data and identifies user behavior. Primary Perception and Fusion: Utilizing millimeter-wave radar data, it quickly and stably detects and tracks human targets within a region, unaffected by changes in lighting, and makes preliminary judgments about macroscopic behaviors (such as movement or sitting still). Simultaneously, guided by radar data, it performs targeted analysis on corresponding areas in the camera image. Fine-Grade Behavior Recognition: Combining spatial location and micro-motion information provided by radar with fine visual features provided by the camera, it accurately identifies the user's current behavior category through a fusion algorithm model, such as "focused reading," "using an electronic screen," or "looking around." The radar's micro-motion information can also be used to infer physiological states such as "whether asleep." Its core components include: Human body detection and tracking subunit: Detects and tracks the location and contour of each user in real time from the image sequence, and assigns a unique ID to each user.

[0019] Key point detection subunit: For each detected human body, identify its skeletal key points (such as head, shoulder, elbow, wrist, etc.).

[0020] Behavior recognition subunit: Based on key point sequences, the detection and interaction relationship of target objects (such as books, tablets), and using pre-trained behavior recognition models (such as spatiotemporal graph convolutional networks), it determines the specific behavioral state of each user, such as: "focused on reading", "using tablet / computer", "looking around", "resting on the table", "leaving the seat", etc.

[0021] The main control unit, connected to both the image processing and behavior recognition unit and the lighting execution module, serves as the system's decision-making center. It includes: a focus assessment model that receives behavior recognition results from each user. For states such as "focused reading" or "focused screen use," it begins accumulating effective focus time. This model can define rules (e.g., triggering a break prompt if sustained focus exceeds 40 minutes) and manage timers.

[0022] The lighting strategy decision-making subunit queries the preset lighting strategy database based on inputs (individual behavior status, attention duration, number of users) and generates specific zone lighting control instructions. For example, it generates instructions for "high color rendering index, 4000K color temperature, 500lx illuminance" for "readers"; and instructions for "low glare, 3000K color temperature, 300lx illuminance" for "tablet users"; when attention exceeds the limit, it generates a gradual change instruction to "slowly switch to warm light (2200K)".

[0023] Lighting Execution Module: Receives instructions from the main control unit and executes specific lighting adjustments. Its core is a luminaire with independent zone-controlled lighting capabilities, for example: Multi-zone LED light panel: The light panel is physically divided into multiple independently controllable LED unit groups, and each zone can illuminate a user seating area.

[0024] Driver circuit: Provides independent current drive for each LED zone to adjust its brightness, color temperature, and even color separately.

[0025] The specific steps for the collaboration between millimeter-wave radar and visual cameras are as follows: Radar Initialization and Global Detection: After system startup, the millimeter-wave radar operates continuously at full power, outputting a point cloud covering the entire field of view to establish a background model of the empty classroom. Upon detecting a new target, a trajectory IDU is immediately created for it. new .

[0026] Visual module standby and wake-up: The visual processing unit (camera and analysis algorithm) is initially in low-power standby or low-frame-rate inspection mode.

[0027] Radar-triggered visual focusing: When the radar tracks target U new If the position remains stable for more than T_stable (e.g., 2 seconds), it is determined that the user has taken a seat, and a "focus command" containing the predicted ROI coordinates of the target is sent to the vision processing unit.

[0028] High-frame-rate visual fine-grained recognition: The visual unit is activated, the camera is adjusted to high-frame-rate mode, and continuous skeletal keypoint extraction and ST-GCN behavior recognition are performed on the specified ROI. Recognition results (behavior label, confidence score) are compared with radar trajectory ID. new Bind.

[0029] Data fusion and continuous tracking: The fusion center integrates visual recognition results with radar position and velocity information to form a complete target description. The radar continuously provides accurate position updates to correct the ROI and determine whether the target has left the field.

[0030] Visual module hibernation: When the radar detects target U new If the location moves out of the detection area, the vision unit is notified. The vision unit stops analyzing the ROI and can decide whether to reduce the frame rate or enter standby mode depending on whether there are other targets.

[0031] A specific embodiment of the multimodal sensor fusion method is as follows: The specific algorithm flow is as follows: Step 1: Spatiotemporal synchronization and coordinate unification Hardware synchronization: The millimeter-wave radar and camera synchronize their timestamps through hardware trigger signals (such as GPIO) or based on the Network Time Protocol (NTP) to ensure that the time alignment error of their data frames is less than 10 milliseconds.

[0032] Coordinate calibration: During system installation, a joint calibration algorithm is used to establish a mapping relationship between the 3D point cloud coordinate system of the millimeter-wave radar and the 2D image pixel coordinate system of the camera. Specifically, this can be achieved by placing a specific calibration object within the common field of view and solving for a 3x4 projection matrix P, such that for a spatial point [X, Y, Z] detected by the radar... T It can be accessed via s [u,v, 1] T = P [X, Y, Z, 1] T The pixel position (u, v) in the image is approximately calculated, where s is the scale factor.

[0033] Step 2: Radar-led primary perception and target nomination Clustering and Tracking: The radar outputs point cloud data in real time. The algorithm first clusters the point cloud (e.g., the DBSCAN algorithm), aggregating point clouds belonging to the same human body into a single target. Subsequently, a multi-target tracking algorithm (e.g., the Joint Probabilistic Data Association Filter - JPDAF) is used to assign a unique anonymous trajectory ID (e.g., U1, U2) to each target, and continuously outputs its 3D spatial position (x, y), velocity, and target bounding box.

[0034] Generating Region of Interest (ROI): For each target (x, y) tracked by the radar, a corresponding ROI prediction box is generated by projecting the target onto the camera image plane using the projection matrix P from step 1. This ROI is slightly larger than the actual human body to guide visual analysis.

[0035] Step 3: Fine-grained visual recognition (within the ROI) ROI Cropping and Feature Extraction: The visual processing unit performs high-intensity computation only on the ROI image patches provided by the radar. First, a lightweight human pose estimation algorithm (such as OpenPose or MoveNet) is run within the ROI to extract the 2D coordinate sequence of the user's skeletal keypoints (usually 17 joints, such as nose, neck, left and right shoulders, left and right elbows, etc.). Second, behavior classification is performed, and the extracted skeletal keypoint sequence of multiple consecutive frames (e.g., 30 frames, about 1 second) is organized into a spatiotemporal graph structure. Nodes are joints, spatial edges connect joints within the human skeleton, and temporal edges connect the same joint in adjacent frames. This graph data is input into a pre-trained spatiotemporal graph convolutional network for behavior classification, outputting the user's behavior probability distribution, such as [Reading: 0.85, Using a tablet: 0.10, Writing: 0.05].

[0036] Specific Implementation Examples of Behavior Classification Steps After extracting the 2D coordinate sequence of skeletal keypoints from the target user across N consecutive frames (e.g., 30 frames), the system performs the following steps for behavior classification: (1) Construction of spatiotemporal graph data structure The sequence data is constructed into a spatiotemporal graph G = (V, E), where: Node set V: Each skeletal keypoint in each frame is a node. For a sequence of 17 keypoints and 30 frames, there are a total of 17 nodes. 30 = 510 nodes. The feature vector of each node v_i is its two-dimensional coordinates (x, y), and a confidence score can be attached.

[0037] Space boundary set E sBased on the natural anatomical structure of the human body, adjacent joints are connected within the same frame to form spatial edges. For example, the "left wrist" node is connected to the "left elbow" node, and the "left elbow" node is connected to the "left shoulder". This constitutes the human skeleton diagram in each frame.

[0038] Time edge set E t Connecting the same keypoint in adjacent frames creates a time edge. For example, the "nose" node in frame t is connected to the "nose" node in frame t+1. This represents the movement trajectory of that keypoint over time.

[0039] (2) Spatiotemporal graph convolutional network processing flow The constructed spatiotemporal graph G is input into a pre-trained spatiotemporal graph convolutional network. This network extracts features by alternately performing the following two operations: Spatial graph convolution: A convolution operation is performed on the skeleton graph of each frame. This operation aggregates features from the neighboring nodes of each joint (e.g., "left elbow" aggregates information from "left shoulder" and "left wrist") to learn the spatial configuration features of human pose.

[0040] Temporal convolution: A one-dimensional temporal convolution is performed on the trajectory sequence of each joint along the time edge. This operation captures the motion pattern of the joint over time (e.g., periodic hand movements may represent writing, while relative stillness may represent reading).

[0041] By stacking multiple layers of "space-time" convolutions, the network can gradually abstract high-level, discriminative spatiotemporal features from the original coordinate sequence.

[0042] (3) Classification and output The final layer of the network connects to a fully connected layer and a softmax classifier. It maps the learned high-level features to predefined behavior categories. Ultimately, the network outputs a probability distribution vector, for example: P = [p_reading, p_using tablet, p_writing, p_talking, p_leaving] Here, each p_i represents the confidence probability that the model determines the user belongs to the i-th type of behavior, and the sum of all probabilities is 1. The system takes the behavior category with the highest probability (e.g., argmax(P) = "reading") as the behavior identification result for the user in that time period, and outputs it together with the confidence score (e.g., 0.85) for use by the subsequent attention assessment model.

[0043] Step 4: Decision-level integration Finally, the system outputs a fused list of targets. Each target includes: ID: Anonymous trajectory ID from radar. Position / Velocity: High-precision data from radar. Behavior Label: Fine-grained recognition results from vision (e.g., "reading"). Confidence: Probabilistic confidence score of the visual classification model.

[0044] Reference manual attached Figure 2-3 The diagram below illustrates the working principle and flowchart of the adaptive lighting adjustment system based on multimodal sensorless perception provided by this invention. Step S1: Non-invasive information acquisition. The sensor module continuously acquires the data stream of the monitored area.

[0045] Step S2: Multi-target behavior recognition. The fusion processing unit processes the synchronous data stream from the multimodal sensor fusion module in real time. First, it prioritizes the use of point cloud and micro-motion information provided by millimeter-wave radar to quickly and stably detect and track the spatial position and macroscopic motion state of all users within the field of view, generating anonymous trajectory identifiers (U1, U2…) for each user. This process is unaffected by changes in lighting and strictly protects privacy. Next, guided by the spatial characteristics of the radar data, the vision processing unit performs refined analysis of specific target areas, determining the precise behavioral state of each user through a behavior recognition model (e.g., B1: reading a book, B2: operating a tablet). The entire recognition process combines "radar-led perception" with "precise visual recognition," achieving both seamless recognition and privacy protection without requiring users to wear any devices.

[0046] Step S3: Attention Assessment and Incentive Judgment. The master control unit's attention assessment model starts / continues timing for users identified as being in a focused state. For example... Figure 2 As shown, when user U1's "effective reading time" accumulates to a preset threshold T, the model determines that a "light stimulus" (rest reminder) is needed.

[0047] The construction of the attention assessment model includes the following steps: The model input features and quantization formulas are constructed. The model input features include: behavioral persistence feature A, posture stability feature B, micro-motion feature C, and visual attention feature D. The quantization formula for persistence feature A is as follows: ,in To focus on action frame rate, The total number of frames; the quantization formula for the attitude stability feature B is: The quantification formula for the micro-motion feature C is as follows: ,in The normalized spectral entropy value; the quantization formula for the visual attention feature D is: To obtain the overall focus index .

[0048] Reference manual attached Figure 2 As shown in the focus assessment section, and in the instruction manual. Figure 4 The model described above maintains a state machine for each user, containing the following states: S0: Non-focused: Initial state or recognition of non-focused behavior (such as "walking around" or "talking").

[0049] S1: Entering Focus: Focused behavior is identified, and the overall index F exceeds the threshold Th. enter (e.g., 0.6), and continue for T stable Seconds (e.g., 5 seconds).

[0050] S2: Maintain Focus: While in a focused state, continuously time the process. The total effective focus time T during this state is recorded. effective .

[0051] S3: Distraction / Fatigue: In S1 or S2, if F is below the threshold Th_distract (e.g., 0.4) for a period of time, or if a specific fatigue micro-movement pattern (e.g., frequent eye rubbing, chin rest) is detected, then this state is entered. Effective focus timer pauses.

[0052] S4: Intervention Required: When the accumulated effective focus time T_effective reaches the health threshold T_health (e.g., 2700 seconds, or 45 minutes), the system will force the user to enter this state regardless of whether the user is currently in S2 or S3 and will immediately trigger a light stimulus.

[0053] Step S4: Personalized Lighting Strategy Decision. The lighting strategy decision subunit integrates current scene information (such as...) Figure 3 As shown, there are two people at the long table, one reading a book and the other looking at a tablet, making a decision: For reader U1: The decision instruction is to "provide focused illumination with high color rendering and medium-high color temperature" to protect eyesight and improve reading clarity.

[0054] For tablet users like U2: The decision instruction is to "provide anti-glare, low color temperature soft lighting" to reduce the contrast between the screen and ambient light and alleviate eye fatigue.

[0055] Meanwhile, because the U1's focus time has reached the threshold, while providing it with reading light, a gradual light stimulus signal is superimposed: in the next minute, the color temperature of the light in its zone slowly changes from 4000K to a warm color light of 2200K, simulating sunset to remind it to rest.

[0056] The decision-making method of the lighting strategy decision-making subunit specifically includes the following steps: (1) Construct a layered strategy library: The first layer is the basic scene strategy, which determines the scene's color temperature, illuminance, color rendering index, and whether the anti-glare mode is enabled as the basic light parameters based on the user's status. The second layer is a dynamic intervention strategy, which determines the duration of scene color temperature and illuminance based on the duration of the user's state and the frequency of state switching. (2) Query user status: Query the first-level policy library based on the user's current status to obtain the basic optical parameters. ; (3) Optimize user state and make decisions: Receive the state signal S1 output by the attention assessment model. If S1 does not contain intervention signals, then the final instruction is given. When S1 contains an intervention signal, the second-layer policy library is queried to obtain the dynamic light sequence. Final Instructions The signal is sent to the drive circuit; when S1 contains multiple intervention signals, the strategy priority weight and parameter mixing weight are defined to resolve conflicts, and the final parameters are generated by the weighted mixing of each active strategy.

[0057] Step S5: Zoned Lighting Execution. The lighting execution module receives instructions. The driver circuit controls different areas of the multi-zone LED light panel. For example... Figure 3 As shown, the left partition (corresponding to U1) outputs high color rendering white light that conforms to the instructions and performs color temperature gradient; the right partition (corresponding to U2) outputs anti-glare soft light. The lights in the two partitions do not interfere with each other, achieving "one light per person".

[0058] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An adaptive lighting adjustment system based on multimodal non-sensory perception, characterized in that, include: Multimodal sensor fusion module: including millimeter-wave radar and vision camera. The millimeter-wave radar generates point cloud data containing the distance between the user and the lighting device, the user's moving speed, the user's location and the user's micro-motion information by transmitting and receiving electromagnetic waves. The vision camera is responsible for collecting color image information of the target area. Image processing and behavior recognition module: electrically connected to the multimodal sensor fusion module, and recognizes user behavior by processing the point cloud data and color image information; Main control module: electrically connected to the image processing and behavior recognition unit, receiving the recognition results of the image processing and behavior recognition unit and generating lighting adjustment instructions based on the recognition results. It has its own attention assessment model and lighting strategy decision subunit. The attention assessment model receives the recognition results, accumulates the effective attention time based on the recognition results, and sends it to the lighting strategy decision subunit. The lighting strategy decision subunit queries the preset lighting strategy database based on the recognition results and the effective attention time, and generates specific zone lighting adjustment instructions. Lighting execution module; It is electrically connected to the main control module and adjusts the lighting device according to the received lighting adjustment command.

2. The adaptive lighting adjustment system based on multimodal sensorless perception according to claim 1, characterized in that, The image processing and behavior recognition module includes: Human body detection and tracking subunit: Real-time detection and tracking of each user's position and contour from image sequences acquired from visual cameras, and assignment of a unique ID to each user; Key point detection subunit: For each detected human body, identify its skeletal key points, namely the head, shoulders, wrists and elbows; Behavior recognition subunit: Based on the sequence data formed by the identified skeletal key points, the detection and interaction relationship of the target object, and using a pre-trained behavior recognition model, it determines the specific behavior state of each user.

3. The adaptive lighting adjustment system based on multimodal sensorless perception according to claim 2, characterized in that, The pre-trained behavior recognition model is a spatiotemporal graph convolutional network, and the workflow of the behavior recognition sub-unit includes the following steps: (1) Spatiotemporal graph data structure construction: The sequence data is constructed into a spatiotemporal graph G = (V, E), where: Node set V: Each skeletal keypoint in each frame is a node, and each node v i The eigenvectors are their two-dimensional coordinates (x, y), and confidence scores can be attached; the spatial boundary set E s Based on the natural anatomical structure of the human body, adjacent joints are connected within the same frame to form spatial edges; the temporal edge set E t Connect the same key point in adjacent frames to form a time edge; (2) Spatiotemporal graph convolutional network processing flow: The constructed spatiotemporal graph G is input into a pre-trained spatiotemporal graph convolutional network. The network extracts features by alternating between the following two operations: Spatial graph convolution: Convolution operation is performed on the skeleton graph of each frame to learn the spatial configuration features of human posture; Temporal convolution: One-dimensional temporal convolution is performed on the trajectory sequence of each joint along the time edge to capture the motion pattern of the joint over time; Through the stacking of multiple layers of spatial-temporal convolution, discriminative spatiotemporal features are gradually abstracted from the original coordinate sequence. (3) Classification and output: The last layer of the network is connected to a fully connected layer and a Softmax classifier, which maps the learned discriminative spatiotemporal features to the preset behavior categories. Finally, a probability distribution vector is output, which takes the behavior category with the highest probability as the behavior recognition result of the user in that time period, and outputs it together with the confidence score for subsequent attention evaluation model.

4. The adaptive lighting adjustment system based on multimodal sensorless perception according to claim 3, characterized in that, The attention assessment model is a quantitative assessment system based on multi-feature weighted fusion and finite state machine. It is used to convert the raw data acquired by multimodal sensors into an attention index, and to manage the user's state transition and health intervention triggers accordingly.

5. The adaptive lighting adjustment system based on multimodal sensorless perception according to claim 4, characterized in that, The construction of the attention assessment model includes the following steps: The model input features and quantization formulas are constructed. The model input features include: behavioral persistence feature A, posture stability feature B, micro-motion feature C, and visual attention feature D. The quantization formula for persistence feature A is as follows: ,in To focus on the number of frames per second of action, The total number of frames; the quantization formula for the attitude stability feature B is: The quantification formula for the micro-motion feature C is as follows: ,in The normalized spectral entropy value; the quantization formula for the visual attention feature D is: To obtain the overall focus index .

6. The adaptive lighting adjustment system based on multimodal sensorless perception according to claim 5, characterized in that, The decision-making method of the lighting strategy decision-making subunit specifically includes the following steps: (1) Construct a layered strategy library: The first layer is the basic scene strategy, which determines the scene's color temperature, illuminance, color rendering index, and whether the anti-glare mode is enabled as the basic light parameters based on the user's status. The second layer is a dynamic intervention strategy, which determines the duration of scene color temperature and illuminance based on the duration of the user's state and the frequency of state switching. (2) Query user status: Query the first-level policy library based on the user's current status to obtain the basic optical parameters. ; (3) Optimize user state and make decisions: Receive the state signal S1 output by the attention assessment model. If S1 does not contain intervention signals, then the final instruction is given. When S1 contains an intervention signal, the second-layer policy library is queried to obtain the dynamic light sequence. Final Instructions And send it to the drive circuit; When S1 contains multiple intervention signals, the policy priority weights and parameter mixing weights are defined to resolve conflicts. The final parameters are generated by a weighted mixture of each active policy.

7. The adaptive lighting adjustment system based on multimodal sensorless perception according to claim 5, characterized in that, The discriminative spatiotemporal features refer to data representations that can distinguish different behaviors from the original joint coordinate sequence after being processed through multiple layers of a spatiotemporal graph convolutional network.

8. An adaptive lighting adjustment method based on multimodal non-sensory perception, characterized in that, The steps include the following: (1) Perform multimodal sensor fusion: The multimodal sensor includes millimeter-wave radar and vision camera. The millimeter-wave radar generates point cloud data containing the distance between the user and the lighting device, the user's moving speed, the user's location and the user's micro-motion information by transmitting and receiving electromagnetic waves. The vision camera is responsible for collecting color image information of the target area. (2) Perform image processing and behavior recognition: Electrically connected to the multimodal sensor fusion module and recognize the user's behavior by processing the point cloud data and color image information; (3) Generate lighting adjustment instructions based on recognition results: It is electrically connected to the image processing and behavior recognition unit, receives the recognition results of the image processing and behavior recognition unit, and generates lighting adjustment instructions based on the recognition results. It has its own attention assessment model and lighting strategy decision subunit. The attention assessment model receives the recognition results, accumulates the effective attention time based on the recognition results, and sends it to the lighting strategy decision subunit. The lighting strategy decision subunit queries the preset lighting strategy database based on the recognition results and the effective attention time, and generates specific zone lighting adjustment instructions. (4) The lighting device executes the light adjustment command; it is electrically connected to the main control module and adjusts the lighting device according to the received light adjustment command.

Images