Intelligent light control device and method

Abstract

The invention relates to a light intelligent control method and device, and belongs to the technical field of smart home. A radar echo signal in an environment is collected through a radar sensor in the sensor module; temperature distribution in the environment is obtained through a heat-sensitive sensor, and temperature array data are output; the data processing module determines whether a target exists in the environment or not based on the radar echo signal and the temperature array data, and identifies the behavior state of the target when the target is detected; and the illumination control module generates a control instruction for controlling the on-off state and brightness of light in the environment based on the behavior state of the target. According to the method, the privacy leakage risk caused by using the video image is avoided, the recognition accuracy can be greatly improved, and the requirements of a multi-user scene are met.

Description

Technical Field

[0001] The present application belongs to the field of smart home technology, and specifically relates to a lighting intelligent control device and method. Background Art

[0002] In the home environment, lighting equipment is one of the most common and frequently used electrical appliances in daily life, and it often needs to be adjusted and controlled. In order to simplify and optimize the control of lights, many innovative control methods have emerged, including voice-activated switches, infrared sensors, human body sensors, etc. These control methods provide families with a more convenient lighting experience.

[0003] For example, the prior art CN201010275069.7 proposes a method of acquiring a video or image to analyze the state of a human body in the image, establishing a human behavior-brightness comparison table, and after determining the human behavior, determining the required brightness according to the behavior-brightness comparison table, and the lighting control module adjusts the brightness of the lamp according to this brightness.

[0004] Although this method is effective in some scenarios, it has certain limitations. Video image analysis involves privacy issues, especially in places with high privacy requirements such as bedrooms and bathrooms, where video surveillance may not be suitable. Video images are not only easy to infringe on personal privacy, but also require more complex image processing algorithms in lighting control. In addition, due to the limitations of image shooting angles and resolution, it may not be possible to fully and accurately obtain the user's behavior status.

[0005] In addition, the particularity of the bedroom scene requires that lighting control not only depends on the behavior of a single user, but also needs to take into account the status of multiple occupants in the room. For example, when a family member is sleeping, the activity status of other members may also affect the adjustment of lighting equipment, and the applicability and flexibility of existing technologies in multi-user scenarios are relatively low.

[0006] Therefore, although the existing technology has solved the problem of intelligent lighting control to a certain extent, it still has problems such as insufficient privacy protection, limited accuracy, and insufficient control flexibility in complex multi-user scenarios. In order to better meet the requirements for privacy, accuracy, and flexibility in home scenarios, a new intelligent lighting control solution is needed that can accurately identify user status while ensuring privacy, especially suitable for places with high privacy requirements such as bedrooms, and can take into account the behavior status of multiple users to achieve more intelligent lighting adjustment. Summary of the invention

[0007] In view of the above analysis, the embodiments of the present invention aim to provide a lighting intelligent control device and method, aiming to solve the problems of insufficient privacy protection, limited accuracy and insufficient control flexibility in complex multi-user scenarios in the prior art.

[0008] In a first aspect of the present application, a lighting intelligent control device is provided, comprising:

[0009] A sensor module, used to obtain target information in the environment; wherein the sensor module includes a radar sensor and a thermal sensor, the radar sensor is configured to collect radar echo signals in the environment, and the thermal sensor is configured to obtain temperature distribution in the environment and output temperature array data;

[0010] The data processing module is configured to determine whether there is a target in the environment based on the radar echo signal and the temperature array data, and identify the behavior state of the target when the target is detected;

[0011] The lighting control module is configured to generate a control instruction for controlling the switch state and brightness of the lights in the environment based on the behavior state of the target.

[0012] Optionally, the data processing module is configured to process the radar echo signal, determine whether there is a target in the environment, and generate target feature data corresponding to each target when a target is detected; the target feature data includes target point cloud data, motion feature data and physiological feature data.

[0013] Optionally, the data processing module is configured to call a pre-trained machine learning model, the input of the machine learning model is multimodal fusion data, and the output is the behavioral state of the target; the multimodal fusion data includes the target feature data and the temperature array data.

[0014] Optionally, the sensor module further includes: an audio sensor for collecting audio data in the environment;

[0015] The data processing module is configured to generate audio feature data including at least target vocal features or respiratory sound features based on the audio data, generate the multimodal fusion data according to the target feature data, the temperature array data and the audio feature data, and input it into the machine learning model.

[0016] Optionally, the lighting control module is configured to execute any one or any combination of the following lighting control strategies based on the behavior state of the target:

[0017] When it is detected that the environment changes from being without a target to having a target and the ambient light level is lower than a threshold, the main lighting source is turned on;

[0018] When the target is detected to enter the sleeping area, the main lighting source is turned off and the auxiliary lighting source is turned on;

[0019] When it is detected that the target changes from a sitting position to a lying position, the auxiliary lighting source is dimmed or turned off;

[0020] When the target is detected to be in a sleeping state, all lighting sources are turned off;

[0021] When it is detected that the target changes from a lying position to a sitting position or leaves the sleeping area, turning on the auxiliary lighting source;

[0022] When it is detected that a target is already in a sleeping state in the environment and another target enters, the auxiliary lighting source is turned on;

[0023] When it is detected that one of the multiple targets changes from a sleeping state to a non-sleeping state and there are multiple auxiliary lighting sources, only the optimal auxiliary lighting source calculated based on the target position is turned on.

[0024] A second aspect of the present application provides a lighting intelligent control method, comprising the following steps:

[0025] Acquire radar echo signals in the environment;

[0026] Obtain the temperature distribution in the environment and obtain temperature array data;

[0027] Determine whether there is a target in the environment based on the radar echo signal and the temperature array data, and identify the behavior state of the target when the target is detected;

[0028] Based on the behavior state of the target, a control instruction is generated to control the switch state and brightness of the lights in the environment.

[0029] Optionally, after acquiring the radar echo signal in the environment, the method further includes:

[0030] The radar return signal is processed to determine whether there is a target in the environment, and when a target is detected, target feature data corresponding to each target is generated; the target feature data includes target point cloud data, motion feature data and physiological feature data.

[0031] Optionally, determining whether there is a target in the environment based on the radar echo signal and the temperature array data, and identifying the behavior state of the target when the target is detected includes:

[0032] Multimodal fusion data is generated based on the target feature data and the temperature array data, and the multimodal fusion data is input into a pre-trained machine learning model, and the output is the behavior state of the target.

[0033] Optionally, it also includes:

[0034] Get audio data in the environment;

[0035] Based on the audio data, audio feature data including at least the target vocal feature or the respiratory sound feature is generated, and the multimodal fusion data is generated according to the target feature data, the temperature array data and the audio feature data, and is input into the machine learning model, and the output is the target's behavioral state.

[0036] Optionally, the generating of a control instruction for controlling the switch state and brightness of the lights in the environment based on the behavior state of the target includes: executing any one or any combination of the following lighting control strategies based on the behavior state of the target:

[0037] When it is detected that the environment changes from being without a target to having a target and the ambient light level is lower than a threshold, the main lighting source is turned on;

[0038] When the target is detected to enter the sleeping area, the main lighting source is turned off and the auxiliary lighting source is turned on;

[0039] When it is detected that the target changes from a sitting position to a lying position, the auxiliary lighting source is dimmed or turned off;

[0040] When the target is detected to be in a sleeping state, all lighting sources are turned off;

[0041] When it is detected that the target changes from a lying position to a sitting position or leaves the sleeping area, turning on the auxiliary lighting source;

[0042] When it is detected that a target is already in a sleeping state in the environment and another target enters, the auxiliary lighting source is turned on;

[0043] When it is detected that one of the multiple targets changes from a sleeping state to a non-sleeping state and there are multiple auxiliary lighting sources, only the optimal auxiliary lighting source calculated based on the target position is turned on.

[0044] The lighting intelligent control device provided by the present application adopts a radar sensor instead of a video sensor. By collecting radar echo data in the environment, it can accurately capture spatial position change data without infringing privacy, and then infer the user's behavior state, such as sitting state, sleeping state, turning over state, etc. It not only avoids the risk of privacy leakage caused by the use of video images, but also does not need to rely on cameras or image processing, ensuring the feasibility of use in places with high privacy requirements such as bedrooms and bathrooms; it can also obtain important physiological feature data that is difficult to capture with video images, such as the target's breathing feature data and body movement feature data, and this information is crucial to judging the user's behavior state and sleep state, which can greatly improve the accuracy of recognition, and does not need to rely on complex image analysis algorithms. Through the pre-trained machine learning model, even if there are multiple users in the room, it can automatically identify the behavior state of each user, and intelligently adjust the light according to the needs of each user to meet the needs of multi-user scenarios. In addition, the present application also provides a lighting intelligent control method with the above technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0046] Figure 1 This is a structural block diagram of the lighting intelligent control device provided in this application;

[0047] Figure 2 A flowchart of a specific implementation of the lighting intelligent control method provided in this application;

[0048] Figure 3 A control flow chart of a specific implementation method of the lighting intelligent control method;

[0049] Figure 4 This is a control flow chart of the lighting intelligent control method in a multi-person scenario. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0052] To further illustrate the technical solution of this application, Figure 1 A specific embodiment shown in FIG. 1 provides a lighting intelligent control device, and its structural block diagram is shown in FIG. Figure 1 As shown, it includes: a sensor module 11, a data processing module 12 and a lighting control module 13.

[0053] Among them, the sensor module 11 is used to obtain target information in the environment, and specifically includes a radar sensor and a thermal sensor. The radar sensor is configured to collect radar echo signals in the environment, and the thermal sensor is configured to obtain the temperature distribution in the environment and output temperature array data. Through the sensor module 11, the radar point cloud features and thermal radiation features of the target in the space can be obtained at the same time, improving the accuracy of subsequent target detection and state recognition.

[0054] The data processing module 12 is in communication connection with the sensor module 11, and is configured to fuse the radar echo signal and the temperature array data, first determine whether there is a target in the environment; if the target is detected, further identify the behavior state of the target, for example: whether it is in a moving state, a stationary state, a sitting position, a lying position, or even in a sleeping state, etc. Optionally, the data processing module 12 can use a target recognition model based on deep learning, such as a convolutional neural network (CNN) or a long short-term memory network (LSTM), to improve the accuracy and robustness of behavioral state recognition.

[0055] The lighting control module 13 is used to receive the behavior state information output by the data processing module 12, and generate control instructions based on the behavior state to intelligently control the lights in the environment. The control instructions are used to adjust the switch state and brightness of the lights.

[0056] In some embodiments of the present application, the data processing module 12 is configured to process the radar echo signal acquired by the sensor module 11 to determine whether there is a target in the environment, and when a target is detected, further generate target feature data corresponding to each target.

[0057] Specifically, the data processing module 12 first pre-processes the radar echo signal, including but not limited to background filtering, static clutter suppression and signal denoising, to improve the accuracy of target detection. On this basis, the echo signal is analyzed using a target detection algorithm (such as a CFAR algorithm, a DBSCAN clustering algorithm or a detection model based on deep learning) to identify whether there is a moving or stationary target in the environment.

[0058] Once the target is detected, the data processing module 12 further extracts multi-dimensional features of the target to generate target feature data corresponding to each target. In this embodiment, the target feature data includes at least the following three types of information:

[0059] Target point cloud data: Based on the spatial detection capability of the radar array, the point cloud data of the target in three-dimensional space is extracted to reflect the spatial distribution characteristics, volume contour, location information, etc. of the target. This data can be used to distinguish different target individuals and provide a basis for subsequent posture recognition or trajectory tracking.

[0060] Motion feature data: includes information such as the target's speed, acceleration, and direction change, which can be obtained by radar Doppler effect or inter-frame point cloud comparison. These features help determine the target's behavioral changes, such as walking, turning, standing up, or lying down.

[0061] Physiological feature data: including physiological information such as breathing rate and heart rate obtained based on micro-motion signal analysis, which is suitable for identifying the state of the target when it is stationary, such as whether the target is in the resting or sleeping stage. This feature can be extracted by analyzing the periodic small phase changes of the radar waveform.

[0062] By comprehensively processing the above multi-dimensional features, the data processing module 12 can construct a structured target state description, which is provided as input to the subsequent behavior recognition model.

[0063] In some embodiments of the present application, the data processing module 12 is configured to call a pre-trained machine learning model to achieve intelligent recognition of the target behavior state.

[0064] Specifically, after extracting the target feature data corresponding to each target, the data processing module 12 further obtains the temperature array data provided by the thermal sensor, and fuses the above multi-source data to construct multimodal fusion data. The multimodal fusion data includes but is not limited to the following:

[0065] Target characteristic data, including:

[0066] Point cloud data: reflects the target’s posture, position and body structure in space;

[0067] Motion characteristic data: such as the target’s speed, acceleration, and movement trajectory;

[0068] Physiological characteristic data: such as the target's breathing rate, micro-movement amplitude, etc., used to identify vital signs in a static state.

[0069] Temperature array data comes from the thermal distribution of the target and its surroundings collected by the thermal sensor. This data can help identify the target's activity area, posture judgment, and whether it is in a high thermal radiation state (for example, whether someone is lying in bed).

[0070] The data processing module 12 inputs the above multimodal fusion data into a machine learning model that has been pre-trained on a specific population behavior sample. The model is a deep neural network model, such as a convolutional neural network (CNN), a Transformer structure, or a long short-term memory network (LSTM) that integrates temporal information. The model has been trained with a large amount of human behavior data and can identify a variety of typical target behavior states, including but not limited to: whether the target exists; whether the target is sitting, standing, or lying; whether the target is stationary or moving; whether the target is asleep, etc. As a specific implementation, the machine learning model in the data processing module 12 is deployed locally or in the cloud, which is not limited here.

[0071] By calling the pre-trained model, the data processing module 12 can identify the current behavior state of the target with high precision and output the identification result in real time as a basis for the lighting control module 13 to make subsequent intelligent control decisions.

[0072] In this embodiment, by introducing multimodal fusion features as model input, compared with the traditional recognition method that only relies on a single sensor data, it has stronger anti-interference ability and recognition accuracy, and is especially suitable for complex indoor application environments such as night, low light, and multiple people in the same scene.

[0073] On the basis of the above embodiments, in order to further improve the accuracy of target behavior state recognition and the environmental adaptability of the system, in the lighting intelligent control device of the present application, the sensor module 11 may further include an audio sensor for collecting audio data in the environment.

[0074] The audio sensor can be an omnidirectional microphone array or a single sound pickup device, deployed at a specific location indoors, and used to collect indoor background sound information and sound signals emitted by the target in real time. The data processing module 12 is configured to extract features from the collected raw audio data and generate audio feature data, which includes at least one of the following:

[0075] Target vocal features: such as speech, coughing, friction when turning over, walking sounds, etc., using the Sound Event Detection (SED) model to identify different types of target activities;

[0076] Breathing sound characteristics: By analyzing low-frequency continuous signals or breathing interval rhythm changes, it can help determine whether the target is in a sleeping state, deep sleep or light sleep stage;

[0077] Ambient noise level: used to help determine how quiet the environment is and further optimize the lighting response logic (for example, entering low-brightness lighting mode).

[0078] The data processing module 12 fuses the above audio feature data with the previously extracted target feature data (such as point cloud, motion, physiological characteristics) and the temperature array data provided by the thermistor to construct richer multimodal fusion data. The fusion data is input into the pre-trained deep learning model to identify the target behavior state.

[0079] In this embodiment, by introducing the audio modality, the data source is further enriched, the target recognition accuracy of the system in complex environments is enhanced, which helps to achieve a more refined lighting control response and significantly improves the user experience and system intelligence level.

[0080] In some specific embodiments, the lighting control module 13 is configured to execute a variety of intelligent lighting control strategies based on the target behavior state identified by the data processing module 12 to achieve automatic adjustment and optimized response of the lighting in the environment. The lighting control strategies include but are not limited to any one or any combination of the following:

[0081] Scenario 1: When someone enters the environment, the lights automatically turn on

[0082] When it is detected that the environment changes from a target-free state to a target state and the ambient light level is lower than a set threshold (such as after sunset or at night), the lighting control module 13 controls the main lighting source, such as a ceiling lamp, to be turned on to provide basic lighting.

[0083] Scenario 2: The user enters the sleeping area and switches to soft lighting

[0084] When the target is identified to move from the edge area of ​​the room to the area where the bed is located, that is, entering the preset sleeping area, the lighting control module 13 controls to turn off the main lighting source and turn on the auxiliary lighting source (such as bedside lamp, night light) to create a comfortable and quiet bedtime environment.

[0085] Scenario 3: The user is ready to go to sleep and the lights are automatically dimmed

[0086] When it is detected that the target changes from a sitting position to a lying position, indicating that the user is about to enter a resting state, the lighting control module 13 automatically dims the currently turned on auxiliary lighting source, or turns off the light source under the user-set conditions, to reduce light interference and help the user fall asleep faster.

[0087] Scenario 4: Automatically turn off the light after the user falls asleep

[0088] Once the data processing module 12 determines that the user is in a stable sleep state (for example, steady breathing and no body movement for a long time are detected), the lighting control module 13 controls to turn off all lighting sources to achieve a completely dark sleeping environment to improve sleep quality and energy saving effects.

[0089] Scenario 5: When the user gets up at night, low-brightness lighting is automatically turned on

[0090] When it is detected that the target changes from a lying position to a sitting position, or leaves the sleeping area (for example, getting up at night), the lighting control module 13 automatically turns on the night light mode or low-brightness lamp in the auxiliary lighting source to allow the user to walk safely while avoiding strong light stimulation.

[0091] Scenario 6: Avoid disturbing others in a multi-person environment

[0092] When there is already a target in the environment in a sleeping state and another target enters the room, the lighting control module 13 does not turn on the main lighting source, but turns on a low-brightness auxiliary light source to avoid disturbing the sleeping user.

[0093] Scenario 7: Selective lighting in a multi-person environment

[0094] If there are multiple auxiliary lighting sources in the environment, and it is detected that one of the multiple targets changes from a sleeping state to an awake state (such as getting up or sitting up), the lighting control module 13 calculates the relative distance between the target and each auxiliary light source, and only turns on the auxiliary lighting source closest to the target position to achieve precise lighting and avoid affecting other users.

[0095] Through the above strategies, the lighting control module 13 can achieve personalized and flexible lighting control response according to different target behavior states, lighting environments and number of users, significantly improving user experience and living comfort while taking into account energy saving and privacy protection.

[0096] Traditional intelligent lighting control relies on data sources such as video image sensors and infrared sensors, while this application uses radar sensors, which are high-precision, non-contact, and insensitive to ambient light. Without infringing on user privacy, the bedroom lights are intelligently adjusted without manual intervention or operation. By combining ambient light-sensitive information, the prediction results of the presence or absence of the target, the prediction results of the target's behavior state, and the user's preference settings (such as the brightness or switch status of the light), multiple lighting devices in the room can be intelligently adjusted. The entire process ensures the intelligence, personalization, and privacy protection of lighting control through real-time data analysis of radar and thermal sensors and predictions by machine learning models.

[0097] Through the pre-trained machine learning model, even if there are multiple users in the room, it can automatically identify the behavior status of each user and intelligently adjust the lighting according to the needs of each user to meet the needs of multi-user scenarios.

[0098] In order to better understand the intelligent lighting control method provided by the present application, a specific implementation is described below.

[0099] In this embodiment, the lighting intelligent control method is mainly used in indoor lighting systems to achieve automatic lighting adjustment based on user status, such as Figure 2 As shown, the method comprises the following steps:

[0100] S201: Acquire radar echo signals in the environment.

[0101] First, the millimeter-wave radar sensor deployed indoors collects radar echo signals in real time. The sensor can sense the existence, position change, speed, and micro-motion of objects in space, and is used to detect the activity trajectory and posture changes of people.

[0102] S202: Acquire the temperature distribution in the environment and obtain temperature array data.

[0103] Thermal sensors (such as infrared thermal imaging sensors) obtain temperature distribution data in the environment and output a two-dimensional temperature array to reflect the thermal radiation conditions in different areas of the space. The temperature array data can identify whether there are targets in hot areas such as beds and sofas, which helps to assist in determining the target's static or resting state.

[0104] S203: Determine whether there is a target in the environment based on the radar echo signal and the temperature array data, and identify the behavior state of the target when the target is detected.

[0105] The radar echo signal and temperature array data are analyzed and processed to determine whether there is a human target in the environment. After the target is detected, the target's behavior state is further identified, such as whether the target is standing, sitting, lying, moving or still, or sleeping.

[0106] Multi-source data can be fused based on rule-based algorithms (such as multi-threshold judgment) or trained behavior recognition models (such as convolutional neural networks (CNN)) to improve recognition accuracy.

[0107] S204: Based on the behavior state of the target, generate a control instruction for controlling the switch state and brightness of the lights in the environment.

[0108] When the behavior state recognition is completed, the corresponding lighting control instructions are generated according to the recognition results. The control strategy includes adjusting the switch state and brightness level of the lighting equipment. The specific examples are as follows:

[0109] When it is detected that the environment changes from being without a target to having a target and the ambient light level is lower than a threshold, the main lighting source is turned on;

[0110] When the target is detected to enter the sleeping area, the main lighting source is turned off and the auxiliary lighting source is turned on;

[0111] When it is detected that the target changes from a sitting position to a lying position, the auxiliary lighting source is dimmed or turned off;

[0112] When the target is detected to be in a sleeping state, all lighting sources are turned off;

[0113] When it is detected that the target changes from a lying position to a sitting position or leaves the sleeping area, turning on the auxiliary lighting source;

[0114] When it is detected that a target is already in a sleeping state in the environment and another target enters, the auxiliary lighting source is turned on;

[0115] When it is detected that one of the multiple targets changes from a sleeping state to a non-sleeping state and there are multiple auxiliary lighting sources, only the optimal auxiliary lighting source calculated based on the target position is turned on.

[0116] Through the above steps, the intelligent lighting control method provided in this embodiment can dynamically adjust the lighting state according to the environmental perception results, realize non-contact and personalized lighting control, and effectively improve user experience and energy saving effects.

[0117] In some embodiments, after acquiring the radar echo signal in the environment, the intelligent lighting control method further includes signal processing on the echo signal to detect the existence of the target and extract the target features, thereby providing support for subsequent behavior recognition and lighting control.

[0118] Specifically, the echo signal acquired by the radar sensor is a continuous wave or frequency modulated continuous wave radar signal, which reflects the reflection characteristics of the objects in the environment to the transmitted electromagnetic waves. The data processing module first pre-processes the original radar echo signal, including but not limited to: filtering, static background suppression, and inter-frame difference processing.

[0119] After preprocessing, radar signal analysis algorithms or deep learning models are used to identify and determine whether there are one or more human targets in the environment. If a target is detected, target feature extraction is performed to generate target feature data corresponding to each target, which includes the following: target point cloud data, motion feature data, and physiological feature data.

[0120] Through the above processing, it is possible to achieve simultaneous detection and status differentiation of multiple targets, and form structured target feature data, providing an accurate and comprehensive input data basis for subsequent behavioral status recognition and lighting control decisions.

[0121] Furthermore, multimodal fusion data is generated based on the target feature data and the temperature array data, and the multimodal fusion data is input into a pre-trained machine learning model, and the output is the behavior state of the target. This recognition process relies on a multimodal data fusion mechanism and a pre-trained machine learning model.

[0122] In order to achieve more accurate target behavior state judgment, the data processing module fuses the above multi-source data to generate structured multimodal fusion data. The fused feature vector may include current data, historical data or a combination thereof. In this embodiment, the fusion method includes but is not limited to feature-level fusion, and the steps are as follows:

[0123] Standardization processing: normalize the data from radar and thermal sensors to a unified scale; time alignment: align the two modal data according to the acquisition timestamp to ensure synchronous input; feature vector splicing: encode each feature into a high-dimensional vector and then splice it to form a unified input tensor; input to the model: input the fused multimodal data into the pre-trained machine learning model.

[0124] The machine learning model can be a convolutional neural network (CNN), a long short-term memory network (LSTM) or a Transformer model. The model is trained based on a large number of human behavior samples and can accurately identify the following typical behavior states: whether the target exists; whether the target is in a spatial posture such as standing, sitting, or lying; whether the target is active or still; whether the target has fallen asleep.

[0125] The pre-training process of a machine learning model includes the following steps:

[0126] Collect a large amount of labeled data for training machine learning models. This data includes the target point cloud data, motion feature data, physiological feature data (such as breathing rate, body movement, etc.), and the actual behavior state of the target (such as whether sleeping, getting up, lying down, sitting, etc.). This data can be obtained through actual environmental monitoring. A sufficiently diverse data set can be constructed, including data from different individuals, different activity states, and different environmental conditions.

[0127] After collecting the data, extract effective feature data from the data. The extracted feature data is used as input to the machine learning model, and the target's behavior state label is used as output. Through the training method of back propagation algorithm and gradient descent method, the parameters of the model are optimized so that it can predict the target's behavior state as accurately as possible. After multiple rounds of training, the prediction ability of the model is continuously optimized. The model will reduce the prediction error by adjusting the weights and biases.

[0128] In some embodiments, in order to further improve the accuracy of target behavior state recognition and the ability to adapt to complex environments, audio sensors are introduced on the basis of using radar and thermal sensors to obtain target feature data and temperature array data to collect audio data in the environment and perform feature extraction and fusion analysis.

[0129] Specifically, the audio sensor is a sound pickup device (such as an omnidirectional microphone or an array microphone) deployed in the environment, which is used to collect sound signals generated during the target activity in real time, including voice, coughing, turning over, footsteps, breathing sounds, etc.

[0130] The audio data is analyzed and processed to extract audio feature data, including but not limited to the following: target vocalization features and breathing sound features.

[0131] Subsequently, the above audio feature data is fused with the previously extracted target feature data (including point cloud data, motion features, physiological features) and temperature array data to construct a more complete multimodal fusion data set.

[0132] In this embodiment, the fusion method adopts a feature-level fusion strategy, that is, each modal feature is normalized and vectorized, and then spliced ​​into a unified high-dimensional input tensor, and input into the trained machine learning model for behavioral state recognition. After training, the model can output refined target behavioral states, such as: awake and in the room; in bed, ready to fall asleep; lying down but still moving; breathing steadily, already asleep; turning over or getting up at night; one person is awake and others are asleep in a multi-person state, etc.

[0133] By introducing the audio modality, the recognition limitations of radar or thermal imaging in static or weak signal states can be effectively compensated. Especially at night when there is no light and no obvious body movement, audio data can serve as an important auxiliary judgment basis, thereby improving the robustness and accuracy of the multimodal recognition system.

[0134] In some implementations, the lighting control module is driven to execute corresponding lighting control strategies by identifying the target behavior state, so as to adapt to the lighting needs of users in different behavior scenarios and improve user experience and energy saving effects.

[0135] Figure 3 A control flow chart of a specific implementation method of the lighting intelligent control method is shown. The lighting intelligent control method executes corresponding lighting control strategies according to changes in the target behavior state to achieve automatic turning on, off or brightness adjustment of the lights to meet the actual needs of users in different behavior scenarios.

[0136] The entire lighting control process is based on the initial judgment condition of "whether someone is detected" and is divided into the following scenarios:

[0137] When the environment is detected to change from "no one" to "someone", first determine whether the current ambient light level is lower than the set threshold:

[0138] If the light is insufficient, the lighting control module controls the main lighting source (such as a ceiling light) to be turned on to provide ambient lighting.

[0139] The behavior transition and light response in the occupied state are as follows:

[0140] 1. The person is under the bed → the person is on the bed:

[0141] When the target is identified as entering a sleeping area (such as a bed), the following actions are performed: the main lighting source is turned off; and auxiliary light sources such as a bedside lamp or a night light are turned on.

[0142] 2. People sitting on the bed → lying down:

[0143] Recognize that the target changes from a sitting position to a lying position: dim or turn off the auxiliary lighting source to prevent light from disturbing the user's sleep.

[0144] 3. Maintain a stable lying position → Fall asleep:

[0145] If the target is detected to be motionless for a long time and has steady breathing, it is judged to be in a sleeping state: all lights are turned off and the system enters the sleep lighting mode.

[0146] 4. Detect waking up

[0147] If the target is detected to change from lying to sitting, or the behavior of getting out of bed is detected: turn on the auxiliary lighting source (night light) with moderate brightness to ensure safety of walking at night.

[0148] Figure 4 The control flow chart of the lighting intelligent control method in a multi-person scene is shown, referring to Figure 3 , personalized lighting response strategies in multi-person scenarios include:

[0149] 1. Someone is already sleeping in the bed and another person enters the room:

[0150] If it is detected that someone is already sleeping in the room and another person enters the room: Turn on the night light or bedside light, avoid turning on the main light, and reduce disturbance to the sleeping user.

[0151] 2. Someone wakes up during sleep: If one of the multiple targets is identified as awake, while the others are still asleep, only the auxiliary light source closest to the awake target is turned on to implement a directional lighting control strategy.

[0152] 3. Detect that everyone has woken up:

[0153] If all targets are identified to have changed from sleeping state to awake state, and the ambient light is insufficient: turn on the main lighting source to restore normal indoor lighting.

[0154] This embodiment combines the change path of the behavior state and the conversion logic between behaviors, and dynamically adjusts the lighting response through a state-driven method, truly realizing a people-oriented intelligent lighting control system with high adaptability and user experience optimization capabilities, and is particularly suitable for scenarios such as home bedrooms, infant care, and smart elderly care.

[0155] In summary, this application can provide a more comprehensive feature analysis, thereby improving the accuracy of the target's behavior status. Not only can it determine whether the target is still, awake or asleep, but it can also be refined to determine the user's specific posture in bed (sitting, lying, etc.), making the lighting control more personalized and intelligent, and can automatically adjust the light according to the specific activity status of each user, improving the user experience.

[0156] This application automatically adjusts the light intensity and switches according to the user's actual behavior (such as whether sleeping, sitting, lying, etc.). For example, the light is automatically turned off when the user enters the sleeping state to ensure a good sleeping environment; the night light is automatically turned on when the user gets up in the middle of the night to prevent the user from getting hurt.

[0157] Without disturbing the privacy of users (no camera monitoring), the non-invasive radar sensor can accurately sense the state of the target and adjust the environment to improve the living experience of the residents. It is especially suitable for places with high privacy requirements such as bedrooms.

[0158] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0159] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0160] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A lighting intelligent control device, characterized in that: include: A sensor module, used to obtain target information in the environment; wherein the sensor module includes a radar sensor and a thermal sensor, the radar sensor is configured to collect radar echo signals in the environment, and the thermal sensor is configured to obtain temperature distribution in the environment and output temperature array data; The data processing module is configured to determine whether there is a target in the environment based on the radar echo signal and the temperature array data, and identify the behavior state of the target when the target is detected; The lighting control module is configured to generate a control instruction for controlling the switch state and brightness of the lights in the environment based on the behavior state of the target.

2. The lighting intelligent control device according to claim 1, characterized in that: The data processing module is configured to process the radar echo signal, determine whether there is a target in the environment, and generate target feature data corresponding to each target when a target is detected; the target feature data includes target point cloud data, motion feature data and physiological feature data.

3. The intelligent lighting control device according to claim 2, characterized in that: The data processing module is configured to call a pre-trained machine learning model, the input of the machine learning model is multimodal fusion data, and the output is the behavior state of the target; the multimodal fusion data includes the target feature data and the temperature array data.

4. The intelligent lighting control device according to claim 3, characterized in that: The sensor module also includes: an audio sensor for collecting audio data in the environment; The data processing module is configured to generate audio feature data including at least target vocal features or respiratory sound features based on the audio data, generate the multimodal fusion data according to the target feature data, the temperature array data and the audio feature data, and input it into the machine learning model.

5. The intelligent lighting control device according to any one of claims 1 to 4, characterized in that: The lighting control module is configured to execute any one or any combination of the following lighting control strategies based on the behavior state of the target: When it is detected that the environment changes from being without a target to having a target and the ambient light level is lower than a threshold, the main lighting source is turned on; When the target is detected to enter the sleeping area, the main lighting source is turned off and the auxiliary lighting source is turned on; When it is detected that the target changes from a sitting position to a lying position, the auxiliary lighting source is dimmed or turned off; When the target is detected to be in a sleeping state, all lighting sources are turned off; When it is detected that the target changes from a lying position to a sitting position or leaves the sleeping area, turning on the auxiliary lighting source; When it is detected that a target is already in a sleeping state in the environment and another target enters, the auxiliary lighting source is turned on; When it is detected that one of the multiple targets changes from a sleeping state to a non-sleeping state and there are multiple auxiliary lighting sources, only the optimal auxiliary lighting source calculated based on the target position is turned on.

6. A lighting intelligent control method, characterized in that: The following steps are involved: Acquire radar echo signals in the environment; Obtain the temperature distribution in the environment and obtain temperature array data; Determine whether there is a target in the environment based on the radar echo signal and the temperature array data, and identify the behavior state of the target when the target is detected; Based on the behavior state of the target, a control instruction is generated to control the switch state and brightness of the lights in the environment.

7. The lighting intelligent control method according to claim 6, characterized in that: After acquiring the radar echo signal in the environment, the method further includes: The radar return signal is processed to determine whether there is a target in the environment, and when a target is detected, target feature data corresponding to each target is generated; the target feature data includes target point cloud data, motion feature data and physiological feature data.

8. The lighting intelligent control method according to claim 7, characterized in that: The determining whether there is a target in the environment based on the radar echo signal and the temperature array data, and identifying the behavior state of the target when the target is detected, comprises: Multimodal fusion data is generated based on the target feature data and the temperature array data, and the multimodal fusion data is input into a pre-trained machine learning model, and the output is the behavior state of the target.

9. The lighting intelligent control method according to any one of claims 6 to 8, characterized in that: Also includes: Get audio data in the environment; Based on the audio data, audio feature data including at least the target vocal feature or the respiratory sound feature is generated, and the multimodal fusion data is generated according to the target feature data, the temperature array data and the audio feature data, and is input into the machine learning model, and the output is the target's behavioral state.

10. The lighting intelligent control method according to claim 9, characterized in that: The generating of the control instruction for controlling the switch state and brightness of the light in the environment based on the behavior state of the target includes: executing any one or any combination of the following lighting control strategies based on the behavior state of the target: When it is detected that the environment changes from being without a target to having a target and the ambient light level is lower than a threshold, the main lighting source is turned on; When the target is detected to enter the sleeping area, the main lighting source is turned off and the auxiliary lighting source is turned on; When it is detected that the target changes from a sitting position to a lying position, the auxiliary lighting source is dimmed or turned off; When the target is detected to be in a sleeping state, all lighting sources are turned off; When it is detected that the target changes from a lying position to a sitting position or leaves the sleeping area, turning on the auxiliary lighting source; When it is detected that a target is already in a sleeping state in the environment and another target enters, the auxiliary lighting source is turned on; When it is detected that one of the multiple targets changes from a sleeping state to a non-sleeping state and there are multiple auxiliary lighting sources, only the optimal auxiliary lighting source calculated based on the target position is turned on.

Images