Human body sensing-based indoor device intelligent control method and human body sensing device

Abstract

Embodiments of the present disclosure disclose a human body sensing based indoor device intelligent control method and a human body sensing device. A specific embodiment of the method comprises: performing gait feature recognition on a target vibration signal; starting a heat source sensing component to collect a heat source image sequence for a target area, and performing heat source recognition on the heat source image sequence; starting a heat source positioning component to collect a millimeter wave radar signal for the target area, and performing radar signal feature extraction on the millimeter wave radar signal; generating object description information according to the gait feature, the heat source feature and the radar signal feature; and controlling, through a wireless communication component, an intelligent indoor device in the target area corresponding to target linkage information. The embodiment realizes accurate personnel recognition and intention recognition, thereby ensuring the effectiveness and timeliness of intelligent indoor device control.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer technology, human body sensing, and intelligent indoor device control, specifically to an intelligent control method for indoor devices based on human body sensing and a human body sensing device. Background Technology

[0002] Smart home technology integrates and controls smart indoor devices through networking. Achieving more seamless control of these devices has become a hot research topic in the industry. As the primary users of smart homes, effective and accurate human identification and intent recognition are crucial for device control. Currently, the main human sensing technologies are: one is traditional infrared sensing, which detects obstructions by detecting changes in infrared thermal radiation; the other is millimeter-wave radar, which uses the Doppler effect to detect obstructions.

[0003] The above method has the following problems: (1) Infrared sensing is prone to problems of missed identification and false identification. It is also difficult to effectively distinguish between human and non-human objects and to further effectively identify intentions. (2) Millimeter-wave radar has certain power consumption requirements. For human body sensing in non-wired scenarios, it is difficult to effectively meet the power consumption requirements. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure propose an intelligent control method and a human body sensing device for indoor equipment based on human body sensing, in order to solve the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide an intelligent control method for indoor devices based on human body sensing. The method includes: performing gait feature recognition on a target vibration signal to obtain gait features and a first confidence level, wherein the target vibration signal is a vibration signal collected by a vibration sensing component targeting a target area, and the target vibration signal includes: a first vibration signal and a second vibration signal; responding to a first confidence level greater than a first threshold, activating a heat source sensing component to collect a heat source image sequence targeting the target area, and performing heat source identification on the heat source image sequence to obtain heat source features and a second confidence level; responding to a second ... A second threshold is set, and the heat source positioning component is activated to collect millimeter-wave radar signals for the target area. Radar signal features are extracted from the millimeter-wave radar signals to obtain radar signal features. Based on the gait features, heat source features, and radar signal features, object description information is generated, including object type, object intent type, and a third confidence level. In response to the third confidence level being greater than the third threshold, the intelligent indoor device corresponding to the target linkage information in the target area is controlled via a wireless communication component. The target linkage information is determined based on the regional location of the target area and the object description information.

[0007] Secondly, some embodiments of this disclosure provide an intelligent control device for indoor equipment based on human body sensing. The device includes: a gait feature recognition unit configured to perform gait feature recognition on a target vibration signal to obtain gait features and a first confidence level, wherein the target vibration signal is a vibration signal collected by a vibration sensing component targeting a target area, and the target vibration signal includes: a first vibration signal and a second vibration signal; a heat source acquisition and identification unit configured to, in response to a first confidence level greater than a first threshold, activate a heat source sensing component to acquire a heat source image sequence targeting the target area, and perform heat source identification on the heat source image sequence to obtain heat source features and a second confidence level; and a radar signal feature extraction unit configured to... The system is configured to: initiate a heat source positioning component to collect millimeter-wave radar signals targeting the target area and extract radar signal features from the millimeter-wave radar signals to obtain radar signal features; generate object description information based on the gait features, heat source features, and radar signal features, wherein the object description information includes: object type, object intent type, and a third confidence level; and control unit configured to: control the intelligent indoor device in the target area corresponding to the target linkage information via a wireless communication component in response to the third confidence level being greater than the third threshold, wherein the target linkage information is determined based on the regional location of the target area and the object description information.

[0008] Thirdly, some embodiments of this disclosure provide a human body sensing device applied to the aforementioned intelligent control method for indoor devices based on human body sensing, comprising: a vibration sensing component, wherein the vibration sensing component includes: a first vibration sensor and a second vibration sensor, wherein the vibration sensitivity of the first vibration sensor is greater than the vibration sensitivity of the second vibration sensor; a heat source sensing component, wherein the heat source sensing component is triggered by the vibration sensing component and is used to sense heat sources in the sensing area; and a heat source positioning component, wherein the heat source positioning component is triggered by the heat source sensing component and is used to locate the heat source sensed by the heat source sensing component. The system includes: a heat source location component; a control component, wherein the control component is used to perform signal analysis on vibration signals collected by the vibration sensing component, to perform heat source identification on heat source images collected by the heat source sensing component, to perform signal analysis on millimeter-wave radar signals collected by the heat source location component, and to perform component control on the vibration sensing component, the heat source sensing component, and the heat source location component; a wireless communication component, wherein the wireless communication component is used to send control signals to the smart indoor device; and a power supply component, wherein the power supply component is used to supply power to the vibration sensing component, the heat source sensing component, the heat source location component, the control component, and the wireless communication component.

[0009] Fourthly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fifthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect.

[0011] The foregoing various embodiments of the present disclosure have the following beneficial effects: through the intelligent control method for indoor devices based on human body sensing according to some embodiments of the present disclosure, accurate personnel identification and intention recognition are realized, thereby ensuring the effectiveness and timeliness of intelligent indoor device control. Specifically, the infrared sensing method is prone to the problems of missed recognition and false recognition, and it is difficult to effectively distinguish between human and non-human objects, and further difficult to perform effective intention recognition; millimeter-wave radar has certain power consumption requirements, and it is difficult to effectively meet the power consumption demand for human body sensing in non-wiring scenarios. Although there also exists a conventional method combining an infrared sensor and a millimeter-wave radar, which starts the millimeter-wave radar through a triggering mode to reduce power consumption. However, due to the missed recognition problem existing in the infrared sensor, personnel identification and intention recognition still cannot be effectively realized. In view of this, in the intelligent control method for indoor devices based on human body sensing according to some embodiments of the present disclosure, first, gait feature recognition is performed on a target vibration signal to obtain a gait feature and a first confidence, wherein the target vibration signal is a vibration signal for a target area collected by a vibration sensing component, and the target vibration signal includes a first vibration signal and a second vibration signal. In practice, the indoor environment is often a closed area with limited space, and the sources of vibration generated in the indoor environment are relatively limited. Meanwhile, considering that the capture of vibration signals has an extremely wide capture range, which can well solve the missed detection problem, the present disclosure adopts the manner of target vibration signal capture as the trigger condition for controlling the heat source sensing component. At the same time, in order to avoid false triggering, considering that the gait features of personnel have obvious periodicity and other characteristics, the gait feature recognition manner is adopted to capture the gait feature and the first confidence, and it is determined in combination with the first confidence whether to trigger the heat source sensing component. Next, in response to the first confidence being greater than a first threshold, the heat source sensing component is started to collect a heat source image sequence for the target area, and heat source recognition is performed on the heat source image sequence to obtain a heat source feature and a second confidence. Thereby, the capture of the heat source feature and the determination of the second confidence from the perspective of infrared thermal radiation change are realized. Subsequently, in response to the second confidence being greater than a second threshold, the heat source positioning component is started to collect a millimeter-wave radar signal for the target area, and radar signal feature extraction is performed on the millimeter-wave radar signal to obtain a radar signal feature. Thereby, the feature extraction of the radar signal is realized, and the problem of high power consumption of the millimeter-wave radar is effectively alleviated by enabling the components one by one in this linkage manner. Finally, object description information is generated according to the gait feature, the heat source feature and the radar signal feature, wherein the object description information includes an object type, an object intention type and a third confidence. Considering that a single component may have the problems of false recognition and missed recognition, comprehensive personnel identification is performed by combining the gait feature, the heat source feature and the radar signal feature, and the corresponding object intention type is directly determined at the same time.Finally, in response to the third confidence level being greater than the third threshold, the intelligent indoor devices in the aforementioned target area corresponding to the target linkage information are controlled via wireless communication components. The target linkage information is determined based on the regional location of the target area and the object description information. This enables the linkage and automatic control of intelligent indoor devices while maintaining the third confidence level above the third threshold. Through this method, accurate personnel identification and intent recognition are achieved, thereby ensuring the effectiveness and timeliness of intelligent indoor device control. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the intelligent control method for indoor devices based on human body sensing according to the present disclosure; Figure 2 A schematic diagram of a scene where a vibration sensing component or infrared sensor is detecting people. Figure 3 This is a schematic diagram illustrating the generation process of the first local vibration signal sequence; Figure 4 This is a diagram of the device architecture for a human body sensing device; Figure 5 This is a schematic diagram of the structure of some embodiments of the intelligent control device for indoor equipment based on human body sensing according to the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of a human body sensing-based intelligent control method for indoor devices according to the present disclosure. This human body sensing-based intelligent control method for indoor devices includes the following steps: Step 101: Perform gait feature recognition on the target vibration signal to obtain gait features and a first confidence level.

[0021] In some embodiments, the execution subject (e.g., a computing device) of the human body sensing-based indoor device intelligent control method can perform gait feature recognition on the target vibration signal to obtain gait features and a first confidence level.

[0022] The target vibration signal is a vibration signal collected by a vibration sensing component and directed towards a target area. The vibration sensing component can be a sensor used to monitor vibration signals generated within the target area, such as a vibration sensor. The target area can be an indoor area equipped with smart indoor devices, such as a living room, bedroom, or kitchen. Smart indoor devices can be remotely controlled devices installed indoors, such as smart lights, smart TVs, smart air conditioners, and smart speakers. The target vibration signal includes a first vibration signal and a second vibration signal. The first and second vibration signals can be vibration signals simultaneously collected by different vibration sensors within the vibration sensing component. The first confidence level characterizes the confidence that the target vibration signal was generated by a person.

[0023] Optionally, the vibration sensing component includes a first vibration sensor and a second vibration sensor, wherein the vibration sensitivity of the first vibration sensor is greater than that of the second vibration sensor. In practice, for indoor environments, in addition to vibrations caused by people walking, there may also be other noise vibrations (such as vibrations caused by objects falling). Therefore, it is necessary to filter these types of noise vibrations. Specifically, when using a single vibration sensor, if the corresponding vibration sensitivity is low, it will lead to the missed capture of gait-related vibration signals; if the corresponding vibration sensitivity is high, the vibration sensing component will not be able to effectively filter noise vibrations, resulting in frequent triggering of the vibration sensing component. Therefore, the vibration sensing component of this disclosure adopts a dual-sensor mode, that is, vibration sensors with different vibration sensitivities (first vibration sensor and second vibration sensor), to capture vibration signals (first vibration signal and second vibration signal) generated within the effective range, thereby ensuring the effectiveness and robustness of vibration signal acquisition.

[0024] In practice, since human gait characteristics exhibit significant periodicity, the first step is to extract periodically related signal indicators (e.g., signal peaks, signal troughs, signal frequencies, etc.) to capture gait-related features from vibration signals (first and second vibration signals). Then, by combining gait features with a decision tree approach, the corresponding first confidence level is determined.

[0025] As an example, Figure 2 This diagram illustrates a scenario where a vibration sensing component or infrared sensor is detecting people. Figure 2 Scenario A and Scenario B are illustrated. Scenario A corresponds to the scenario where a vibration sensing component captures the vibration signal generated by a target object (e.g., a person), while Scenario B corresponds to the scenario where an infrared sensor detects the target object. Assuming the vibration sensing component and the infrared sensor are positioned at the same location within the target area, analysis of Scenario B reveals that when the detection area of ​​the infrared sensor is smaller than the target area, the infrared sensor cannot effectively detect the target object even if it is already within the target area. Analysis of Scenario A shows that as the target object moves into the target area, the vibration signal generated by its movement is collected by the vibration sensing component.

[0026] It should be noted that the aforementioned computing device can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here. In particular, the aforementioned computing device can be a human body sensing device.

[0027] In some optional implementations of some embodiments, the aforementioned execution entity performs gait feature recognition on the target vibration signal to obtain gait features and a first confidence level, including: Step S1: Perform signal denoising on the first vibration signal and the second vibration signal respectively to obtain the denoised first vibration signal and the denoised second vibration signal.

[0028] In practice, since the target vibration signal needs to be transmitted to the vibration sensing component through structures such as walls and floors, there may also be situations where noise signals are captured by the vibration sensing component. Therefore, appropriate signal denoising is required to improve the quality of the vibration signal. Specifically, a Kalman filter can be used to denoise both the first and second vibration signals to obtain denoised first and second vibration signals.

[0029] Step S2: Extract the first signal description information corresponding to the first vibration signal after noise reduction.

[0030] The first signal description information includes: a trough position sequence and a peak position sequence. Since the first vibration signal is a time-series signal, the trough position represents the signal position of the trough signal value in the first vibration signal after noise reduction, and the peak position represents the signal position of the peak signal value in the first vibration signal after noise reduction.

[0031] In practice, firstly, the first-order difference corresponding to the first vibration signal after noise reduction can be obtained. Next, the point where the derivative of the first-order difference changes from negative to positive is identified as the trough position in the first vibration signal after noise reduction. Then, the point where the derivative of the first-order difference changes from positive to negative is identified as the peak position in the first vibration signal after noise reduction.

[0032] Step S3: Generate a first sliding window based on the first signal description information described above.

[0033] In practice, since the movement of people is not entirely periodic, to ensure that the window size corresponding to the first sliding window is appropriate, firstly, multiple periodic values ​​are obtained based on the trough and peak positions. Then, through statistical analysis, the periodic value that occurs most frequently is determined as the window size of the first sliding window.

[0034] Step S4: Based on the first sliding window, the first vibration signal after noise reduction is slid segmented to obtain the first local vibration signal sequence.

[0035] In practice, the window length of the first sliding window can be W, and the movement step size of the first sliding window is W×n%. This is because, since the window size of the first sliding window is statistically obtained and the movement of a person is not entirely periodic, using W as the movement step size might result in the first local vibration signal within the first sliding window not completely corresponding to a periodic gait. To ensure the effectiveness of the first local vibration signal within the first sliding window, the movement step size is controlled by n. The value of n can be [50, 100]. In particular, the value of W×n% may be a decimal; therefore, when W×n% is a decimal, it is rounded up to the nearest integer as the movement step size of the first sliding window.

[0036] As an example, see Figure 3 The diagram shows the generation process of the first local vibration signal sequence. In this process, the first vibration signal after noise reduction corresponding to the actual gait is segmented according to the movement step size through the first sliding window to obtain the first local vibration signal sequence.

[0037] Step S5: Extract the second signal description information corresponding to the second vibration signal after noise reduction.

[0038] The second signal description information includes: the trough position sequence and the peak position sequence.

[0039] Step S6: Generate a second sliding window based on the second signal description information described above.

[0040] Step S7: According to the second sliding window, the noise-reduced second vibration signal is slid segmented to obtain the second local vibration signal sequence.

[0041] In practice, the specific implementation methods for generating the second local vibration signal sequence in steps S5 to S7 are the same as those for generating the first local vibration signal sequence in steps S1 to S4, and will not be repeated here.

[0042] Step S8: Generate the gait features and the first confidence level based on the pre-trained gait feature extraction model, the first local vibration signal sequence, and the second local vibration signal sequence.

[0043] The gait feature extraction model described above is a dual-path feature extraction model, consisting of a first gait feature extraction model, a second gait feature extraction model, a feature fusion layer, and a Softmax layer. The first and second gait feature extraction models share local parameters. Specifically, both the first and second gait feature extraction models use a U-Net network as their backbone. In particular, the convolutional layers located at the same network position and corresponding to 2N layers in both models share weights, where N is an odd number. The feature fusion layer includes a feature stacking module and a downsampling module. The feature stacking module fuses the features extracted by the first and second gait feature extraction models to obtain the gait features. Then, the Softmax layer outputs a first confidence score. Specifically, the downsampling module downsamples the gait features and uses this downsampling as input to the Softmax layer. The reason for not using the conventional dual-path network with weight sharing between layers is that the first vibration signal and the second vibration signal correspond to different vibration sensitivities, that is, gait features are expressed under different sensitivities. Using the conventional method described above, the feature representation will fail due to the weight sharing between layers. Therefore, this disclosure uses weight sharing between some convolutional layers, which can ensure the correlation between the first gait feature extraction model and the second gait feature extraction model, while avoiding excessive mutual influence.

[0044] Step 102: In response to the first confidence level being greater than the first threshold, the heat source sensing component is activated to acquire a heat source image sequence for the target area, and to identify the heat source in the heat source image sequence to obtain heat source features and a second confidence level.

[0045] In some embodiments, the execution entity may, in response to a first confidence level greater than a first threshold, initiate the heat source sensing component to acquire a heat source image sequence for the target area, and perform heat source identification on the heat source image sequence to obtain heat source features and a second confidence level.

[0046] The heat source sensing component can be implemented using an infrared camera. Specifically, since the main objective of this disclosure is effective sensing of people and the location is indoors (e.g., living room, bedroom, etc.), a conventional method would be to use a full-color camera for image acquisition. However, this requires optimized algorithms to filter sensitive information to avoid privacy leaks. This process involves additional data processing, which affects the sensing speed and places additional computing power demands on the hardware. The increase in redundant computing power will further increase hardware costs. Since infrared images are mainly based on thermal radiation changes, they naturally have the characteristic of filtering sensitive information. Therefore, this disclosure uses an infrared camera as the heat source sensing component to capture heat source images. Furthermore, by using a first confidence level and a first threshold to trigger the heat source sensing component to acquire heat source images, especially in scenarios where power lines cannot be laid, the energy consumption problem caused by long-term operation of the heat source sensing component can be effectively reduced. The second confidence level represents the confidence level of the person corresponding to the heat source characteristics.

[0047] In practice, firstly, the heat source sensing component can be triggered to acquire continuous heat source images of the target area over a specified period, forming a heat source image sequence. Secondly, a recurrent neural network (RNN) with a "one-to-many" architecture can be used to identify heat sources in the heat source image sequence, obtaining heat source features. In particular, a classifier can be set after the recurrent neural network to generate a second confidence level.

[0048] In some optional implementations of certain embodiments, the execution entity performs heat source identification on the heat source image sequence to obtain heat source features and a second confidence level, including: Step S1: For each heat source image in the above heat source image sequence, perform the following processing steps: Step S11: Perform edge detection on the above heat source image to obtain edge features.

[0049] In practice, edge detection algorithms based on the Sobel operator can be used to detect edges in heat source images and obtain edge features.

[0050] Step S12: Based on the above edge features, perform image segmentation on the above heat source image to obtain a first segmented image and a second segmented image.

[0051] The first segmented image is a local image in the heat source image that is enclosed by edge features, and the second segmented image is a local image in the heat source image that is enclosed by inverse selection of edge features.

[0052] Step S13: Determine the image ratio.

[0053] The image ratio is the ratio of the number of pixels in the first segmented image to the number of pixels in the second segmented image.

[0054] Step S14: In response to the image ratio being less than the preset image ratio, the pixel weights of the first segmented image are updated according to the first weight to obtain the updated first segmented image.

[0055] The value range of the first weight is (1, 1.3).

[0056] In practice, the pixel values ​​in the first segmented image can be updated using a first weight value to obtain an updated first segmented image. In particular, when the image ratio is less than a preset image ratio, the position representing the infrared target may be relatively small. Therefore, updating the pixel values ​​using the first weight highlights the image difference from the second segmented image.

[0057] Step S15: In response to the image ratio being greater than or equal to the preset image ratio, the pixel weights of the second segmented image are updated according to the second weights to obtain the updated second segmented image.

[0058] The value of the second weight can be in the range of [0.85, 1].

[0059] In practice, the pixel values ​​in the second segmented image can be updated using a second weight value to obtain the updated second segmented image. Specifically, when the image ratio is greater than or equal to a preset image ratio, it indicates a larger location corresponding to the infrared target. In this case, due to the larger corresponding area, a certain level of effective feature capture can be guaranteed. However, to further highlight the difference from the second segmented image, a second weight value is used to weaken the second segmented image to some extent. By combining the first and second weight values, only a small area needs to have its pixel values ​​updated, effectively reducing the amount of data processing.

[0060] Step S16: Perform in-situ image stitching on the updated first segmented image and the updated second segmented image to obtain the updated heat source image.

[0061] In practice, since the first segmented image after the update has undergone the first weight update, or the second segmented image after the update has undergone the second weight update, the corresponding pixel values ​​of the image may be greater than the pixel value boundary. Therefore, after the images are stitched together in situ, they need to be re-standardized with 0 and 256 as the upper and lower boundaries to obtain the updated heat source image.

[0062] Step S2: Based on the updated heat source image sequence and the pre-trained heat source feature extraction model, generate the aforementioned heat source features and the aforementioned second confidence level.

[0063] The aforementioned heat source feature extraction model includes an optical flow feature extraction model and a heat source localization model. The optical flow feature extraction model is used to extract optical flow features from the updated heat source image sequence at a block-level granularity. The heat source localization model is used to extract heat source features and locate heat sources based on the extracted optical flow features. The optical flow feature extraction model includes a Feature Pyramid Network (FPN) and a block-level optical flow feature extractor. The FPN network is used to extract features from the image. The block-level optical flow feature extractor first performs corner detection on the extracted image features to obtain a set of corner points. Corner detection uses a block-level pixel search (e.g., 3×3, a total of 9 pixels). Compared to the conventional pixel-by-pixel search method, this abstracts 9 pixels into 1 pixel for searching. This approach has two advantages: first, it can enhance the extraction of weak corner points through the joint representation of surrounding pixels; second, it can compress the computational load (equivalent to reducing the image size, i.e., performing corner search at a smaller size), thereby improving search efficiency. Secondly, optical flow is calculated using the Lucas-Kanade method to obtain optical flow features. The heat source localization model uses the Faster R-CNN model as its backbone network. Specifically, since Faster R-CNN is a typical two-stage object detection model, dividing the detection task into region proposal and region classification and localization, and the optical flow feature extractor disclosed in this paper has already extracted the continuous transformation of the heat source in multiple frames of heat source images by extracting optical flow features, Faster R-CNN, in the region proposal part, uses the corner positions included in the optical flow features as constraints to limit the candidate boxes proposed in the region proposal part. Specifically, the corner positions included in the optical flow features should be located within the candidate boxes and close to the candidate box boundaries. This method can further reduce the number of proposed candidate boxes and improve the localization speed.

[0064] Step 103: In response to the second confidence level being greater than the second threshold, the heat source positioning component is activated to collect millimeter-wave radar signals for the target area, and radar signal feature extraction is performed on the millimeter-wave radar signals to obtain radar signal features.

[0065] In some embodiments, the aforementioned execution entity may, in response to a second confidence level greater than a second threshold, initiate the heat source positioning component to acquire millimeter-wave radar signals for the target area, and extract radar signal features from the millimeter-wave radar signals to obtain radar signal features.

[0066] The heat source location component can be a millimeter-wave radar. In particular, considering human body sensing in non-wired scenarios, a low-power millimeter-wave radar can be selected as the heat source location component.

[0067] In practice, firstly, a two-dimensional fast Fourier transform (FFT) is performed on the millimeter-wave radar signal to obtain the processed millimeter-wave radar signal. Secondly, a constant false alarm rate (CFAR) detection method is used to capture and track targets in the processed millimeter-wave radar signal, thereby estimating the target's range and velocity. Next, the direction of arrival (DOA) is used to estimate the target's azimuth and elevation angles by utilizing the phase difference between signals received from different channels. Finally, a point cloud image is constructed based on the target's range, velocity, azimuth, and elevation angles, serving as the radar signal features.

[0068] In some optional implementations of certain embodiments, the execution entity performs radar signal feature extraction on the millimeter-wave radar signal to obtain radar signal features, including: Step S1: Perform clutter filtering on the millimeter-wave radar signal to obtain the filtered millimeter-wave radar signal.

[0069] In practice, firstly, pulse compression processing can be performed on the millimeter-wave radar signal to obtain a compressed millimeter-wave radar signal. Secondly, a moving target indication (MTI) algorithm is used to filter clutter from the compressed millimeter-wave radar signal to obtain a filtered millimeter-wave radar signal. Specifically, pulse compression can improve the range resolution of the millimeter-wave radar signal. Furthermore, considering the difference in Doppler frequencies between stationary clutter and moving targets, a moving target indication algorithm is used to distinguish between them and separate the clutter, thereby improving the signal quality of the millimeter-wave radar signal.

[0070] Step S2: Generate a sparse point cloud map based on the filtered millimeter-wave radar signal.

[0071] In practice, firstly, the filtered millimeter-wave radar signal can be processed using a two-dimensional fast Fourier transform to obtain the processed millimeter-wave radar signal. Secondly, targets in the processed millimeter-wave radar signal are captured and tracked using constant false alarm rate (CFAR) detection, thereby estimating the target's range and velocity. Next, the target's azimuth and elevation angles are estimated using the direction of arrival (DOA) based on the phase difference between signals received from different channels. Finally, a point cloud image is constructed based on the target's range, velocity, azimuth, and elevation angles, serving as a sparse point cloud map.

[0072] Step S3: Perform the following processing steps on the above sparse point cloud map: Step S31: Extract a first number of point clouds from the above sparse point cloud map as a candidate point cloud set.

[0073] The candidate point cloud set includes a second number of historical point clouds. When the current iteration has 1 extraction iteration, the second number is 0. Historical point clouds are those retained from the first number of point clouds extracted in the previous iteration. The overlap between the sampling regions of any two candidate point clouds in the candidate point cloud set is less than a preset overlap. The sampling region is a spherical region centered on the candidate point cloud with a preset radius. Specifically, although the extraction process references some point clouds extracted in the previous iteration (the second number of historical point clouds), the extraction process still has a certain degree of randomness, which may result in small intervals between extracted point clouds, affecting the subsequent upsampling quality. Therefore, by setting sampling regions for the point clouds, the distance between any two extracted point clouds is controlled.

[0074] As an example, suppose the current iteration has 2 extractions, and N (the first number) point clouds need to be extracted from the sparse point cloud graph as a candidate point cloud set. The candidate point cloud set includes M (the second number) historical point clouds. These M historical point clouds are the point clouds selected from the N point clouds when the iteration had 1 extraction. Specifically, N > M ≥ 0.

[0075] Step S32: Determine the area of ​​the union region of the sampling region set corresponding to the candidate point cloud set.

[0076] In practice, since each candidate point cloud corresponds to a spherical sampling region, the sampling region corresponding to the candidate point cloud can first be projected onto a two-dimensional plane. Then, the area of ​​the union region is the union of the projected sampling regions corresponding to each candidate point cloud in the candidate point cloud set. In particular, the region union is often an irregularly shaped region; therefore, the number of pixels contained in the region union is used as the corresponding union region area.

[0077] Step S33: Determine the area coverage.

[0078] The region coverage is the ratio of the area of ​​the union region to the area of ​​the sparse point cloud map. In particular, to ensure scale consistency, the sparse point cloud map can be projected onto a two-dimensional plane (the same projection plane as the sampling region), and the number of pixels contained therein can be used as the area of ​​the sparse point cloud map.

[0079] Step S34: In response to the convergence of regional coverage or the current iteration number being equal to the preset iteration number, select candidate point clouds that meet the selection criteria from the candidate point cloud set as target point clouds to obtain the target point cloud set, and perform point cloud upsampling on the sparse point cloud map based on the target point cloud set to obtain radar signal features.

[0080] The selection criteria are as follows: the point cloud density of the sampling region corresponding to the candidate point cloud is greater than a first density threshold and less than a second density threshold. Point cloud density represents the number of point clouds located in the sampling region corresponding to the candidate point cloud. In particular, since the candidate point cloud is an extracted point cloud, its corresponding spherical sampling region may contain other point clouds. Therefore, the number of point clouds in the sampling region corresponding to the candidate point cloud can be used as the point cloud density.

[0081] In practice, firstly, since each iteration selects the second number of historical point clouds extracted in the previous iteration as candidate point clouds in the candidate point cloud set, the entire process approximates convergence with continuous iteration. To avoid excessively slow convergence in extreme cases, a preset number of iterations is set to control the end of the iteration. Secondly, when convergence is achieved or the preset number of iterations is reached, it indicates that the selected candidate point cloud set can represent the point cloud distribution in the sparse point cloud map to a certain extent. Therefore, point cloud upsampling is needed based on this to further improve the graph resolution corresponding to the sparse point cloud map. However, upsampling all target point clouds is problematic because: firstly, the data processing volume is large, affecting real-time performance; secondly, some selected target point clouds lack surrounding point clouds, resulting in poor upsampling performance for these target point clouds. Therefore, candidate point clouds are filtered and selected by combining point cloud density. Finally, point cloud interpolation, such as point cloud interpolation, can be used to perform point cloud interpolation on the target point clouds, thereby achieving the effect of point cloud upsampling on the sparse point cloud map. This method enables targeted upsampling of the core point cloud (target point cloud), effectively increasing the point cloud density of the sparse point cloud map while reducing the amount of data processing and ensuring timely response.

[0082] Step S35: In response to the failure of the region coverage to converge and the current iteration number being less than the preset iteration number, the above processing steps are re-executed.

[0083] Step 104: Generate object description information based on gait characteristics, heat source characteristics, and radar signal characteristics.

[0084] In some embodiments, the aforementioned execution entity can generate object description information based on gait characteristics, heat source characteristics, and radar signal characteristics.

[0085] The object description information includes: object type, object intent type, and third confidence level. The object type represents the identified target type. For example, the object type can include: personnel type and non-personnel type. The object intent type represents the identified object's behavioral intent. The third confidence level represents the confidence level corresponding to the object type.

[0086] In practice, multimodal models, such as LLM (Large Language Model), can be used to obtain object description information by taking gait features, heat source features, and radar signal features as inputs.

[0087] In particular, as the features involved become more and more abundant, the first confidence level, second confidence level, and third confidence level in the human body sensing process should be gradually increased. Therefore, different thresholds (first threshold, second threshold, and third threshold) should be set accordingly as triggers for different stages, thereby controlling the start-up timing of different components in the entire human body sensing process, especially in non-wired scenarios, in order to achieve the purpose of power consumption control.

[0088] In some optional implementations of certain embodiments, the execution entity generates object description information based on the gait characteristics, heat source characteristics, and radar signal characteristics, including: Step S1: Generate fused features based on the multimodal feature extractor, the above gait features, the above heat source features, and the above radar signal features.

[0089] The multimodal feature extractor includes three feature encoders, a feature fusion unit, and a feature decoder. The feature encoders employ a Transformer-based encoder. Specifically, considering that gait features and radar signal features are essentially signal features, an encoder designed for text processing can be used. Furthermore, considering that heat source features are essentially image features, an encoder designed for image processing (e.g., the encoder part of the ViT-BERT model) is used. In addition, for multimodal data processing, conventional methods often use a single encoder to jointly encode multimodal data. However, gait features, the aforementioned heat source features, and the aforementioned radar signal features do not have obvious feature relationships before feature encoding, which can lead to poor encoding quality. Therefore, this disclosure uses three independent encoders for independent encoding. Secondly, the feature fusion unit concatenates the features output from the three feature encoders. The feature decoder uses one decoder. Since the three concatenated features are in the same feature space, using three decoders results in two problems: firstly, a large number of parameters leading to a large data processing volume; and secondly, it fails to capture the relationships between features. Therefore, this disclosure uses only one decoder for overall decoding to obtain the fused features.

[0090] Step S2: Generate the object description information based on the above-mentioned fusion features, object type classifier, and object intent discriminator.

[0091] The multimodal feature extractor, object type classifier, and object intent discriminator are included in the object description information generation model. Specifically, during the model training phase, the object description information generation model employs supervised training to fix the parameters of each included model component. Both the object type classifier and the object intent discriminator can be multi-classifiers. In particular, considering the categories of object types, the object type classifier is a binary classifier. The object intent discriminator is a multi-classifier.

[0092] Step 105: In response to the third confidence level being greater than the third threshold, the smart indoor device in the target area corresponding to the target linkage information is controlled via the wireless communication component.

[0093] In some embodiments, in response to a third confidence level greater than a third threshold, a smart indoor device in the target area corresponding to the target linkage information is controlled via a wireless communication component.

[0094] The target linkage information is determined based on the regional location of the target area and the aforementioned object description information. Target linkage information may include: the device communication address, device type, device status, and control logic of the controllable smart indoor devices.

[0095] In practice, to ensure the accuracy of smart indoor device control, corresponding human body sensors are often set up in different areas, and the corresponding smart indoor devices in those areas are linked for control. Therefore, when the location of the target area is known, the controllable smart indoor devices can be identified. Furthermore, different control logics are often preset for different smart indoor devices. For example, for smart lights, when the object's intent type indicates a person passing by, the smart lights are controlled to turn on at low brightness; when the object's intent type indicates a person staying, the smart lights are controlled to turn on at high brightness. Therefore, given the controllable smart indoor devices, it is necessary to combine the object description information to match the corresponding control logic of the controllable smart devices, thereby achieving further screening of smart indoor devices and obtaining target linkage information. Based on this, wireless communication components are used, such as WiFi, Bluetooth, WiMAX, Zigbee, UWB (Ultra Wide Band), and other currently known or future wireless connection methods, to control the smart indoor devices in the target area that correspond to the target linkage information.

[0096] In some optional implementations of some embodiments, in response to a third confidence level greater than a third threshold, controlling the smart indoor device in the target area corresponding to the target linkage information via a wireless communication component includes: Step S1: Determine the network connection status of the smart indoor device corresponding to the target linkage information through the above wireless communication components.

[0097] The device network status includes: device online status, device offline status, or device communication abnormal status.

[0098] In practice, this method can be used to determine the network connectivity status of controllable smart indoor devices.

[0099] Step S2: In response to the device being online, generate an initial control command based on the aforementioned target linkage information.

[0100] In practice, corresponding initial control commands can be generated based on the control logic for smart indoor devices included in the target linkage information.

[0101] Step S3: Based on the device type of the smart indoor device corresponding to the above target linkage information, determine whether to translate the above initial control command.

[0102] In practice, due to differences in the types and brands of smart indoor devices, inconsistencies in control standards prevent direct control of these devices. Therefore, instruction translation is necessary. This requires considering the device type to determine if control standards are consistent, and thus whether instruction translation of the initial control commands is required.

[0103] Step S4: Response Therefore, the above initial control command is translated to obtain the updated control command.

[0104] In practice, plugins like HomeAssistant can be used to translate initial control commands into updated commands. HomeAssistant is an open-source, interoperable integration platform that can control and manage smart home devices from different brands.

[0105] Step S5: In response to no, determine the initial control command as the updated control command; Step S6: Send the updated control command to the smart indoor device corresponding to the target linkage information through the wireless communication component.

[0106] In practice, the updated control commands can be sent to the smart indoor devices corresponding to the target linkage information via wireless communication components, using WiFi, Bluetooth, WiMAX, Zigbee, UWB (Ultra Wide Band), and other known or future wireless connection methods.

[0107] The above various embodiments of the present disclosure have the following beneficial effects: through the intelligent control method for indoor devices based on human body sensing according to some embodiments of the present disclosure, accurate personnel identification and intention recognition are realized, thereby ensuring the effectiveness and timeliness of intelligent indoor device control. Specifically, the infrared sensing method is prone to the problems of missed identification and false identification, at the same time, it is difficult to effectively distinguish between human and non-human objects, and it is difficult to further perform effective intention recognition; millimeter-wave radar has certain power consumption requirements, and it is difficult to effectively meet the power consumption requirement for human body sensing in non-wiring scenarios. Although there also exists a conventional method combining an infrared sensor and a millimeter-wave radar, which starts the millimeter-wave radar through a triggering mode to reduce power consumption. However, due to the missed identification problem of the infrared sensor, personnel identification and intention recognition still cannot be effectively realized. In view of this, in the intelligent control method for indoor devices based on human body sensing according to some embodiments of the present disclosure, first, gait feature recognition is performed on a target vibration signal to obtain a gait feature and a first confidence, wherein the target vibration signal is a vibration signal for a target area collected by a vibration sensing component, and the target vibration signal includes: a first vibration signal and a second vibration signal. In practice, the indoor environment is often a closed area with limited space, and the vibration generation sources in the indoor environment are relatively limited. Meanwhile, considering that the capture of vibration signals has an extremely wide capture range, which can well solve the missed detection problem, the present disclosure adopts the manner of target vibration signal capture as the trigger condition for controlling the heat source sensing component. At the same time, in order to avoid false triggering, considering that the gait features of personnel have obvious periodicity and other characteristics, the gait feature recognition manner is adopted to capture the gait feature and the first confidence, and the first confidence is combined to judge whether to trigger the heat source sensing component. Secondly, in response to the first confidence being greater than a first threshold, the heat source sensing component is started to collect a heat source image sequence for the target area, and heat source recognition is performed on the heat source image sequence to obtain a heat source feature and a second confidence. Thereby, the capture of the heat source feature and the determination of the second confidence from the perspective of infrared thermal radiation change are realized. Then, in response to the second confidence being greater than a second threshold, the heat source positioning component is started to collect a millimeter-wave radar signal for the target area, and radar signal feature extraction is performed on the millimeter-wave radar signal to obtain a radar signal feature. Thereby, the feature extraction of the radar signal is realized, and the problem of high power consumption existing in the millimeter-wave radar is effectively alleviated by enabling the components one by one in this linkage manner. Finally, object description information is generated according to the gait feature, the heat source feature and the radar signal feature, wherein the object description information includes: an object type, an object intention type and a third confidence. Considering that a single component may have the problems of false identification and missed identification, the gait feature, the heat source feature and the radar signal feature are combined for comprehensive personnel identification, and the corresponding object intention type is directly determined at the same time.Finally, in response to the third confidence level being greater than the third threshold, the intelligent indoor devices in the aforementioned target area corresponding to the target linkage information are controlled via wireless communication components. The target linkage information is determined based on the regional location of the target area and the object description information. This enables the linkage and automatic control of intelligent indoor devices while maintaining the third confidence level above the third threshold. Through this method, accurate personnel identification and intent recognition are achieved, thereby ensuring the effectiveness and timeliness of intelligent indoor device control. Further reference Figure 4 The diagram illustrates the device architecture of a human body sensing device according to some embodiments of the present disclosure. The human body sensing device includes: a vibration sensing component, a heat source sensing component, a heat source positioning component, a control component, a wireless communication component, and a power supply component. The aforementioned vibration sensing component includes a first vibration sensor and a second vibration sensor, wherein the vibration sensitivity of the first vibration sensor is greater than that of the second vibration sensor. The vibration sensing component of this disclosure employs a dual-sensor mode, namely vibration sensors with different vibration sensitivities (the first vibration sensor and the second vibration sensor), to capture vibration signals (the first vibration signal and the second vibration signal) generated within the effective range, thereby ensuring the effectiveness and robustness of vibration signal acquisition.

[0108] The aforementioned heat source sensing component is triggered by the aforementioned vibration sensing component, and is used to sense heat sources in the sensing area. The heat source sensing component can be implemented using an infrared camera.

[0109] The aforementioned heat source location component is triggered by the heat source sensing component, and is used to locate the heat source sensed by the heat source sensing component. The heat source location component can be a millimeter-wave radar. In particular, considering human body sensing in non-wired scenarios, a low-power millimeter-wave radar can be selected as the heat source location component.

[0110] The aforementioned control component is used for signal analysis of the vibration signals collected by the vibration sensing component, for heat source identification of the heat source images collected by the heat source sensing component, for signal analysis of the millimeter-wave radar signals collected by the heat source positioning component, and for component control of the vibration sensing component, the heat source sensing component, and the heat source positioning component. The control component consists of a control circuit.

[0111] The aforementioned wireless communication component is used to send control signals to smart indoor devices. The wireless communication component may include at least one of the following: a Wi-Fi transceiver, a Bluetooth transceiver, a Zigbee transceiver, and a UWB transceiver, along with a corresponding communication control circuit.

[0112] The power supply component provides power to the aforementioned vibration sensing component, heat source sensing component, heat source positioning component, and control component. Considering human body sensing in non-wired scenarios, the power supply component uses a button battery for power. Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an intelligent control device for indoor equipment based on human body sensing. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this human body sensing-based indoor device intelligent control device can be specifically applied to various electronic devices.

[0113] like Figure 5 As shown, some embodiments of an indoor device intelligent control device 500 based on human body sensing include: a gait feature recognition unit 501, a data acquisition and heat source identification unit 502, a data acquisition and radar signal feature extraction unit 503, a generation unit 504, and a control unit 505. The gait feature recognition unit 501 is configured to perform gait feature recognition on a target vibration signal to obtain gait features and a first confidence level. The target vibration signal is a vibration signal acquired by a vibration sensing component targeting a target area, and includes a first vibration signal and a second vibration signal. The data acquisition and heat source identification unit 502 is configured to, in response to a first confidence level greater than a first threshold, activate a heat source sensing component to acquire a heat source image sequence targeting the target area, and perform heat source identification on the heat source image sequence to obtain the heat source. Features and a second confidence level; a radar signal feature extraction unit 503 is configured to, in response to the second confidence level being greater than a second threshold, activate the heat source positioning component to acquire millimeter-wave radar signals for the target area, and extract radar signal features from the millimeter-wave radar signals to obtain radar signal features; a generation unit 504 is configured to generate object description information based on the gait features, the heat source features, and the radar signal features, wherein the object description information includes: object type, object intent type, and a third confidence level; a control unit 505 is configured to, in response to the third confidence level being greater than a third threshold, control the intelligent indoor device in the target area corresponding to the target linkage information via a wireless communication component, wherein the target linkage information is determined based on the regional location of the target area and the object description information.

[0114] It is understandable that the units described in the human body sensing-based indoor equipment intelligent control device 500 are similar to those in the reference device. Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the human body sensing-based indoor intelligent control device 500 and the units contained therein, and will not be repeated here. The following is for reference. Figure 6 It illustrates a schematic diagram of the structure of an electronic device (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of this disclosure. Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory may include a non-volatile storage medium and internal memory. The non-volatile storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause the processor to perform any of the methods described above. The processor provides computational and control capabilities to support the operation of the entire computer device. The internal memory provides an environment for the execution of the computer program in the non-volatile storage medium; when executed by the processor, the computer program causes the processor to perform any of the methods described above. The network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0116] In one embodiment, the processor is configured to run a computer program stored in a memory to perform the following steps: gait feature recognition on a target vibration signal to obtain gait features and a first confidence level, wherein the target vibration signal is a vibration signal collected by a vibration sensing component targeting a target area, and the target vibration signal includes: a first vibration signal and a second vibration signal; in response to a first confidence level greater than a first threshold, initiating a heat source sensing component to collect a heat source image sequence targeting the target area, and performing heat source recognition on the heat source image sequence to obtain heat source features and a second confidence level; in response to a second confidence level greater than a first threshold, initiating a heat source sensing component to collect a heat source image sequence targeting the target area, and performing heat source recognition on the heat source image sequence to obtain heat source features and a second confidence level; in response to a second confidence level greater than a first threshold, initiating a heat source recognition component to collect a heat source image sequence targeting a target area, and performing a heat source recognition component ... A second threshold is set, and the heat source positioning component is activated to collect millimeter-wave radar signals for the target area. Radar signal features are extracted from the millimeter-wave radar signals to obtain radar signal features. Based on the gait features, heat source features, and radar signal features, object description information is generated, including object type, object intent type, and a third confidence level. In response to the third confidence level being greater than the third threshold, the intelligent indoor device corresponding to the target linkage information in the target area is controlled via a wireless communication component. The target linkage information is determined based on the regional location of the target area and the object description information.

[0117] This disclosure also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can be referred to the various embodiments of the methods described above.

[0118] The aforementioned computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. Alternatively, the aforementioned computer-readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0119] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0120] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for intelligent control of indoor equipment based on human body sensing, characterized in that, include: Gait feature recognition is performed on the target vibration signal to obtain gait features and a first confidence level. The target vibration signal is a vibration signal collected by a vibration sensing component and targeted at the target area. The target vibration signal includes a first vibration signal and a second vibration signal. In response to a first confidence level greater than a first threshold, the heat source sensing component is activated to acquire a heat source image sequence for the target area, and to identify the heat source in the heat source image sequence to obtain heat source features and a second confidence level. In response to the second confidence level being greater than the second threshold, the heat source positioning component is activated to collect millimeter-wave radar signals for the target area, and radar signal feature extraction is performed on the millimeter-wave radar signals to obtain radar signal features; Based on the gait features, the heat source features, and the radar signal features, object description information is generated, wherein the object description information includes: object type, object intent type, and third confidence level; In response to a third confidence level exceeding a third threshold, the system controls, via a wireless communication component, the smart indoor device in the target area corresponding to the target linkage information, wherein the target linkage information is determined based on the regional location of the target area and the object description information. The step of generating object description information based on the gait features, the heat source features, and the radar signal features includes: Based on the multimodal feature extractor, the gait features, the heat source features, and the radar signal features, a fused feature is generated. The multimodal feature extractor includes three feature encoders, a feature fusion unit, and a feature decoder. The object description information is generated based on the fused features, object type classifier, and object intent discriminator, wherein the multimodal feature extractor, object type classifier, and object intent discriminator are included in the object description information generation model. The step of controlling the smart indoor device in the target area corresponding to the target linkage information via a wireless communication component in response to a third confidence level greater than a third threshold includes: The wireless communication component is used to determine the network status of the smart indoor device corresponding to the target linkage information. The network status includes: device online status, device offline status, or device communication abnormal status. In response to the device being online due to network connectivity, an initial control command is generated based on the target linkage information; Based on the device type of the smart indoor device corresponding to the target linkage information, determine whether to translate the initial control command; In response, the initial control command is translated to obtain the updated control command; Whether or not the initial control command is determined as the updated control command; The updated control command is sent to the smart indoor device corresponding to the target linkage information via the wireless communication component.

2. The method according to claim 1, characterized in that, The step of performing gait feature recognition on the target vibration signal to obtain gait features and a first confidence level includes: The first vibration signal and the second vibration signal are respectively denoised to obtain the denoised first vibration signal and the denoised second vibration signal; Extract the first signal description information corresponding to the first vibration signal after noise reduction, wherein the first signal description information includes: trough position sequence and peak position sequence; A first sliding window is generated based on the first signal description information; According to the first sliding window, the noise-reduced first vibration signal is slid segmented to obtain the first local vibration signal sequence; Extract the second signal description information corresponding to the noise-reduced second vibration signal, wherein the second signal description information includes: trough position sequence and peak position sequence; A second sliding window is generated based on the second signal description information; Based on the second sliding window, the noise-reduced second vibration signal is subjected to sliding segmentation to obtain a second local vibration signal sequence; The gait features and the first confidence score are generated based on the pre-trained gait feature extraction model, the first local vibration signal sequence, and the second local vibration signal sequence. The gait feature extraction model is a dual-path feature extraction model, which consists of a first gait feature extraction model, a second gait feature extraction model, a feature fusion layer, and a Softmax layer. The first gait feature extraction model and the second gait feature extraction model share local parameters.

3. The method according to claim 2, characterized in that, The step of identifying heat sources from the heat source image sequence to obtain heat source features and a second confidence level includes: For each heat source image in the heat source image sequence, perform the following processing steps: Edge detection is performed on the heat source image to obtain edge features; Based on the edge features, the heat source image is segmented to obtain a first segmented image and a second segmented image, wherein the first segmented image is a local image of the heat source image enclosed by edge features, and the second segmented image is a local image of the heat source image enclosed by inverse selection of edge features. Determine the image ratio, where the image ratio is the ratio of the number of corresponding pixels in the first segmented image to the number of corresponding pixels in the second segmented image; In response to an image ratio being less than a preset image ratio, the pixel weights of the first segmented image are updated according to the first weight to obtain the updated first segmented image; In response to an image ratio greater than or equal to a preset image ratio, the pixel weights of the second segmented image are updated according to the second weights to obtain the updated second segmented image; The updated first segmented image and the updated second segmented image are stitched together in situ to obtain the updated heat source image. Based on the updated heat source image sequence and the pre-trained heat source feature extraction model, the heat source features and the second confidence score are generated. The heat source feature extraction model includes an optical flow feature extraction model and a heat source localization model. The optical flow feature extraction model is used to extract optical flow features from the updated heat source image sequence at the block granularity, and the heat source localization model is used to extract heat source features and locate the heat source based on the extracted optical flow features.

4. The method according to claim 3, characterized in that, The step of extracting radar signal features from the millimeter-wave radar signal to obtain radar signal features includes: Clutter filtering is applied to the millimeter-wave radar signal to obtain the filtered millimeter-wave radar signal. Based on the filtered millimeter-wave radar signal, a sparse point cloud map is generated. The following processing steps are performed on the sparse point cloud image: A first number of point clouds are extracted from the sparse point cloud map to form a candidate point cloud set. The candidate point cloud set includes a second number of historical point clouds. When the current iteration extraction count is 1, the second number is 0. The historical point clouds are the point clouds retained from the first number of point clouds extracted in the previous iteration. The overlap between the sampling regions corresponding to any two candidate point clouds in the candidate point cloud set is less than a preset overlap. The sampling region is a spherical region centered on the candidate point cloud with a preset radius. Determine the area of ​​the union region of the sampling regions corresponding to the candidate point cloud set; Determine the region coverage, where the region coverage is the ratio of the area of ​​the union region to the area of ​​the corresponding sparse point cloud map; In response to the convergence of regional coverage or the current iteration number being equal to the preset iteration number, candidate point clouds that meet the selection criteria are selected from the candidate point cloud set as target point clouds to obtain the target point cloud set. Based on the target point cloud set, the sparse point cloud map is upsampled to obtain radar signal features. The selection criteria are: the point cloud density of the sampling area corresponding to the candidate point cloud is greater than the first density threshold and less than the second density threshold. The point cloud density represents the number of point clouds located in the sampling area corresponding to the candidate point cloud. If the region coverage does not converge and the current iteration number is less than the preset iteration number, the processing steps are re-executed.

5. A human body sensing device, applied to the intelligent control method for indoor equipment based on human body sensing as described in any one of claims 1 to 4, characterized in that, include: A vibration sensing component, comprising: a first vibration sensor and a second vibration sensor, wherein the vibration sensitivity of the first vibration sensor is greater than the vibration sensitivity of the second vibration sensor. A heat source sensing component, wherein the heat source sensing component is triggered by the vibration sensing component, and the heat source sensing component is used to sense the heat source in the sensing area; A heat source positioning component, wherein the heat source positioning component is triggered by a heat source sensing component, and the heat source positioning component is used to locate the heat source sensed by the heat source sensing component. A control component, wherein the control component is used to perform signal analysis on the vibration signal collected by the vibration sensing component, to perform heat source identification on the heat source image collected by the heat source sensing component, to perform signal analysis on the millimeter-wave radar signal collected by the heat source positioning component, and to perform component control on the vibration sensing component, the heat source sensing component, and the heat source positioning component. A wireless communication component, wherein the wireless communication component is used to send control signals to a smart indoor device; A power supply component, wherein the power supply component is used to supply power to the vibration sensing component, the heat source sensing component, the heat source positioning component, the control component and the wireless communication component.

6. A human body sensing-based intelligent control device for indoor equipment, used to execute the method according to any one of claims 1 to 4, characterized in that, include: A gait feature recognition unit is configured to perform gait feature recognition on a target vibration signal to obtain gait features and a first confidence level, wherein the target vibration signal is a vibration signal collected by a vibration sensing component for a target area, and the target vibration signal includes: a first vibration signal and a second vibration signal; The acquisition and heat source identification unit is configured to, in response to a first confidence level greater than a first threshold, activate the heat source sensing component to acquire a heat source image sequence for the target area, and perform heat source identification on the heat source image sequence to obtain heat source features and a second confidence level; The acquisition and radar signal feature extraction unit is configured to, in response to a second confidence level greater than a second threshold, activate the heat source positioning component to acquire millimeter-wave radar signals for the target area, and perform radar signal feature extraction on the millimeter-wave radar signals to obtain radar signal features; The generation unit is configured to generate object description information based on the gait features, the heat source features, and the radar signal features, wherein the object description information includes: object type, object intent type, and third confidence level; The control unit is configured to control, via a wireless communication component, a smart indoor device in the target area corresponding to target linkage information in response to a third confidence level greater than a third threshold, wherein the target linkage information is determined based on the regional location of the target area and the object description information.

7. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 4.

8. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images