Smart home central control system and method based on multi-mode perception

Abstract

The invention relates to the technical field of smart home control, and particularly discloses a smart home central control system and method based on multi-mode perception, and the system comprises the steps: synchronously collecting a voice audio signal, a gesture image signal, an infrared thermal imaging signal and a millimeter wave radar signal through a plurality of groups of sensors; performing blind source separation processing on the voice and gesture signals, extracting a voice command component and a gesture action component which are independent in statistics, and performing space-time alignment and Kalman filtering fusion on the infrared and radar signals to generate a dynamic environment sensing graph; voice intention features, gesture track features and environment anomaly features are extracted through Mel frequency cepstrum coefficient analysis, skeleton key point tracking and multi-level convolution processing; constructing a three-dimensional decision matrix based on the features, performing weighted evaluation through a fuzzy logic rule base to generate a control instruction priority sequence, and dynamically adjusting an equipment operation mode according to the priority; according to the invention, the problems of control conflict and response delay caused by multi-mode signal coupling are solved.

Description

Technical Field

[0001] This invention relates to the field of smart home control technology, specifically to a smart home central control system and method based on multimodal perception. Background Technology

[0002] With the popularization of smart home systems, existing central control systems mostly rely on single-modality or simply superimposed multi-sensor data for environmental perception and interactive control. In practical applications, users often interact with the system simultaneously through multiple methods such as voice and gestures, while environmental conditions are perceived by multiple sensors such as infrared and radar. However, due to the significant differences in time-frequency characteristics, data structures, and physical meanings of different modal signals, existing technologies struggle to achieve effective decoupling and fusion of multimodal signals, leading to the following technical problems: severe coupling and interference exist between different modal signals; voice commands are susceptible to environmental noise contamination; gesture recognition is affected by changes in lighting; and environmental perception data lacks spatiotemporal consistency. Ultimately, this results in inaccurate understanding of user intentions, unreliable environmental condition judgments, conflicting control commands, or delayed responses.

[0003] The existing technology has the following shortcomings: Existing technologies lack the ability to decouple mixed voice and gesture signals through blind source separation technology, achieve spatiotemporal fusion of infrared and radar signals through Kalman filtering, construct a multimodal decision matrix and perform weighted evaluation in combination with a fuzzy logic rule base, thereby realizing accurate intent recognition and priority control under multimodal signal interaction, eliminating control conflicts caused by signal coupling, and improving the system's response speed and control accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a smart home central control system and method based on multimodal perception to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions: A smart home central control method based on multimodal perception includes the following steps: S1: During the operation of the smart home central control system, user interaction signals and environmental status signals are collected simultaneously. The user interaction signals include voice audio signals and gesture image signals, and the environmental status signals include infrared thermal imaging signals and millimeter-wave radar signals. S2: The speech audio signal and gesture image signal are used as a mixed observation signal to separate the statistically independent speech command component and gesture action component; the infrared thermal imaging signal and the millimeter-wave radar signal are spatiotemporally aligned and fused using the Kalman filter algorithm to generate a dynamic environment perception map; S3: Perform Mel-frequency cepstral coefficient analysis on the speech command components to extract speech intent features; perform skeletal keypoint tracking on the gesture motion components to extract gesture trajectory features; and simultaneously process the dynamic environment perception map using a convolutional neural network to extract environmental anomaly features. S4: Based on voice intent features, gesture trajectory features, and environmental anomaly features, a multimodal decision matrix is ​​constructed. The decision matrix is ​​weighted and evaluated using a preset fuzzy logic rule base to generate a control command priority sequence. The operation mode of the smart home device is dynamically adjusted according to the priority sequence, and an adaptive control strategy is output.

[0006] As a further aspect of the present invention: S2 specifically includes: Multi-scale time-frequency analysis is performed on the speech audio signal to obtain the speech time-frequency spectrum, and three-dimensional convolution feature extraction is performed on the gesture image signal to obtain the gesture motion sequence. The speech time-frequency spectrum and gesture motion sequence are input into the blind source separation network. By maximizing mutual information and minimizing the loss function, statistically independent speech command components and gesture motion components are separated. The infrared thermal imaging signal and the millimeter-wave radar signal are synchronized in time and registered in space to generate an aligned multimodal signal. The signal is then dynamically fused using a Kalman filter algorithm to output a dynamic environmental perception map.

[0007] As a further aspect of the present invention: the separation of statistically independent voice command components and gesture action components specifically includes: The speech time-frequency spectrum is divided into time windows, and the spectral energy distribution features within each time window are extracted to form a speech spectral feature vector; at the same time, the motion trajectory of the gesture motion sequence is sampled, and the rate of change of the coordinates of the gesture key points in each frame is extracted to form a gesture motion feature vector. The speech spectral feature vector and the gesture motion feature vector are fused in a multimodal manner. Principal component analysis is used to reduce the dimensionality and generate a mixed feature matrix. The mixed feature matrix is ​​then preprocessed by whitening to eliminate correlation. The whitened mixed feature matrix is ​​input into the blind source separation algorithm. The mutual information value of the mixed feature matrix is ​​calculated as the loss function. The separation matrix is ​​iteratively optimized using the stochastic gradient descent method to minimize the loss function until the separation matrix converges. The converged separation matrix is ​​used to perform a linear transformation on the mixed feature matrix to obtain statistically independent speech command components and gesture components.

[0008] As a further aspect of the present invention: S3 specifically includes: The speech command components are preprocessed by framing and windowing, the Mel frequency cepstral coefficients are calculated, and the coefficients are mapped to a high-dimensional space through nonlinear transformation to obtain the speech intent feature vector. Using the speech intent feature vector as a temporal reference, the motion trajectory of the gesture component is segmented. Based on the segmentation results, the spatial coordinate sequence of the skeletal key points is extracted, and the gesture trajectory feature vector is generated by calculating the rate of change of Euclidean distance between the key points. The gesture trajectory feature vector and the voice intent feature vector are concatenated into a multimodal guidance tensor. The multimodal guidance tensor is then used to apply regional attention weighting to the dynamic environment perception map to highlight abnormally sensitive areas. Multi-level convolution operations are performed on the weighted dynamic environment perception map, shallow and deep features are fused through residual connections, and finally, channel pooling is used to extract environmental anomaly features.

[0009] As a further aspect of the present invention: the highlighted abnormally sensitive area specifically includes: The gesture trajectory feature vector and the voice intent feature vector are concatenated into a three-dimensional tensor, with the time dimension aligned and the spatial dimension expanded, to form a multimodal guidance tensor. A nonlinear transformation is performed on the multimodal guidance tensor to generate an attention weight matrix that matches the size of the dynamic environment perception map; The attention weight matrix is ​​multiplied element-wise with the dynamic environment perception map to complete the regional attention weighting, which strengthens the response value of abnormally sensitive areas and suppresses the response value of non-critical areas. Local extremum detection is performed on the weighted dynamic environment perception map, and regions with response values ​​exceeding a preset threshold are marked as the final abnormal sensitive regions.

[0010] As a further aspect of the present invention: the nonlinear transformation of the multimodal guidance tensor to generate an attention weight matrix matching the size of the dynamic environment perception map specifically includes: The multimodal guidance tensor is decomposed into multiple feature sub-tensors along the channel dimension, and each feature sub-tensor corresponds to a different modal interaction mode. Each feature sub-tensor is transformed by a piecewise nonlinear function, where the slope of the piecewise nonlinear function is dynamically adjusted according to the distribution of tensor values ​​to generate a preliminary weight distribution. The initial weight distributions are fused across channels, and an unadjusted attention weight matrix is ​​generated by element-wise addition and global average pooling. The unadjusted attention weight matrix is ​​scaled using bilinear interpolation to make its spatial dimensions completely consistent with the dynamic environment perception map, thus forming the final attention weight matrix.

[0011] As a further aspect of the present invention: S4 specifically includes: By aligning voice intent features, gesture trajectory features, and environmental anomaly features along the time dimension, a three-dimensional decision matrix is ​​constructed, where the first dimension represents the feature type, the second dimension represents the time series, and the third dimension represents the feature intensity. The three-dimensional decision matrix is ​​input into a preset fuzzy logic rule base, and each feature is fuzzified through a membership function to generate a fuzzy decision set. Rule-based reasoning is performed on the fuzzy decision set, and the weight coefficients are dynamically adjusted according to the correlation strength between features to calculate the comprehensive priority score of each control command. The control command priority sequence is generated by sorting the comprehensive priority scores from high to low, with environmental anomaly features having the highest weight coefficient, followed by voice intent features, and gesture trajectory features having a basic weight coefficient.

[0012] As a further aspect of the present invention: the construction process of the three-dimensional decision matrix is ​​as follows: The speech intent features, gesture trajectory features, and environmental anomaly features are processed using a dynamic time warping algorithm to ensure that the three feature sequences have the same temporal length. The processed time series are mapped according to time points, and the timestamps of each feature are aligned using cubic spline interpolation to form a time-synchronized multimodal feature group. The feature intensity of the time-synchronized multimodal feature group is normalized to uniformly map the numerical range of each feature to a preset interval. The normalized features are arranged and combined according to three dimensions: feature type, time series, and feature intensity, to construct the complete three-dimensional decision matrix.

[0013] As a further aspect of the present invention: the formation of time-synchronized multimodal feature sets specifically includes: Based on the start and end times of all time-equal length sequences, a uniformly sampled reference time axis is generated, which covers the common time range of all sequences. For each time series of equal time, the original time point is used as the interpolation node. The cubic spline interpolation method is applied to calculate the feature value of each sampling point on the reference time axis, generating a continuous and smooth interpolated feature series. Boundary condition optimization is performed on each interpolation feature sequence by symmetrically expanding the data points at both ends of the sequence; The optimized interpolated feature sequences are combined according to the time points of the reference time axis to form a time-synchronized multimodal feature group.

[0014] A smart home central control system based on multimodal perception includes: The multimodal signal acquisition module is used to simultaneously acquire user interaction signals and environmental status signals during the operation of the smart home central control system. The user interaction signals include voice audio signals and gesture image signals, and the environmental status signals include infrared thermal imaging signals and millimeter-wave radar signals. Signal decoupling and fusion are used to treat speech audio signals and gesture image signals as mixed observation signals, and separate statistically independent speech command components and gesture action components; infrared thermal imaging signals and millimeter-wave radar signals are spatiotemporally aligned and fused using a Kalman filter algorithm to generate a dynamic environment perception map. The multimodal feature extraction module is used to perform Mel-frequency cepstral coefficient analysis on the speech command components to extract speech intent features; to perform skeletal keypoint tracking on the gesture action components to extract gesture trajectory features; and to process the dynamic environment perception map using a convolutional neural network to extract environmental anomaly features. The intelligent control decision module constructs a multimodal decision matrix based on voice intent features, gesture trajectory features, and environmental anomaly features. It then performs a weighted evaluation of the decision matrix using a preset fuzzy logic rule base to generate a control command priority sequence. Based on the priority sequence, it dynamically adjusts the operation mode of the smart home devices and outputs an adaptive control strategy.

[0015] The beneficial effects of this invention are: (1) By employing blind source separation technology to separate and process the mixed voice and gesture signals, environmental noise and intermodal interference can be effectively eliminated, and independent voice commands and gesture features can be accurately extracted. By combining multimodal environmental perception data fusion of infrared thermal imaging and millimeter-wave radar, the system can maintain stable perception capabilities under complex scenarios such as changes in lighting and noise interference, which significantly improves the accuracy of interactive recognition and the reliability of environmental status judgment of the smart home system in real environments.

[0016] (2) By constructing a multimodal decision matrix and applying a fuzzy logic rule base, the system can comprehensively evaluate multi-dimensional features such as voice intent, gesture trajectory, and environmental anomalies, and dynamically generate a sequence of control command priorities. This decision-making mechanism based on multi-feature weighted evaluation enables the system to adaptively adjust the device control strategy according to the real-time environmental status and user interaction intent, ensuring priority handling of security anomalies and achieving precise control in daily interaction scenarios, thereby improving the intelligence level and user experience of the smart home system. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 As shown, this invention is a smart home central control method based on multimodal perception, comprising the following steps: S1: During the operation of the smart home central control system, user interaction signals and environmental status signals are collected simultaneously. The user interaction signals include voice audio signals and gesture image signals, and the environmental status signals include infrared thermal imaging signals and millimeter-wave radar signals. S2: The speech audio signal and gesture image signal are used as a mixed observation signal to separate the statistically independent speech command component and gesture action component; the infrared thermal imaging signal and the millimeter-wave radar signal are spatiotemporally aligned and fused using the Kalman filter algorithm to generate a dynamic environment perception map; S3: Perform Mel-frequency cepstral coefficient analysis on the speech command components to extract speech intent features; perform skeletal keypoint tracking on the gesture motion components to extract gesture trajectory features; and simultaneously process the dynamic environment perception map using a convolutional neural network to extract environmental anomaly features. S4: Based on voice intent features, gesture trajectory features, and environmental anomaly features, a multimodal decision matrix is ​​constructed. The decision matrix is ​​weighted and evaluated using a preset fuzzy logic rule base to generate a control command priority sequence. The operation mode of the smart home device is dynamically adjusted according to the priority sequence, and an adaptive control strategy is output.

[0021] In S1, during the operation of the smart home central control system, user interaction signals and environmental status signals are collected simultaneously. The user interaction signals include voice audio signals and gesture image signals, while the environmental status signals include infrared thermal imaging signals and millimeter-wave radar signals. Specifically, these include: During the operation of the smart home control system, the system synchronously collects user interaction signals and environmental status signals through multiple sets of dedicated sensors. User interaction signals include voice audio signals and gesture image signals, while environmental status signals include infrared thermal imaging signals and millimeter-wave radar signals.

[0022] Voice audio signals are acquired using a high-sensitivity microphone array. This array, with a ring-shaped distribution structure, is mounted on the surface of the smart home control device, enabling omnidirectional acquisition of user voice data. During acquisition, the microphone array simultaneously records the raw audio waveform data and automatically filters out background steady-state noise, retaining only the valid audio segments containing voice commands.

[0023] Gesture image signals are acquired using a depth vision camera. This camera, equipped with an infrared laser projector and an infrared image sensor, is capable of acquiring real-time sequences of gesture images containing depth information. During acquisition, the camera covers the user's interaction area from a fixed perspective, continuously capturing hand movement images and generating gesture motion data containing a three-dimensional coordinate point cloud.

[0024] Infrared thermal imaging signals from the environmental status data are acquired using an uncooled infrared thermal imager. This imager employs a focal plane array detector to generate a thermal image spectrum reflecting temperature distribution by detecting the infrared radiation intensity on the surface of objects. Simultaneously, millimeter-wave radar signals are acquired using a frequency-modulated continuous wave radar sensor. This sensor transmits millimeter-wave signals and receives reflected echoes to obtain information on the speed, distance, and orientation of objects in the environment.

[0025] During signal acquisition, the system uses a unified time synchronization module to accurately timestamp all sensor data, ensuring that the multimodal signals have the same time reference. The raw data acquired by each sensor is transmitted to the central processing unit through a dedicated interface to form a time-aligned multimodal signal dataset.

[0026] In S2, the speech audio signal and gesture image signal are used as a mixed observation signal, and statistically independent speech command components and gesture action components are separated. Infrared thermal imaging signals and millimeter-wave radar signals are spatiotemporally aligned and fused using a Kalman filter algorithm to generate a dynamic environment perception map, specifically including: In signal processing, multi-scale time-frequency analysis is first performed on the speech audio signal. This analysis employs a windowed short-time Fourier transform method, using a Hanning window with a window length of 256 sampling points and an overlap interval of 128 sampling points. A fast Fourier transform is then performed on the speech signal within each time window, converting the time-domain signal to a frequency-domain representation. The amplitude value of each frequency component is calculated, forming a speech time-frequency spectrum. The horizontal axis of this spectrum represents the time series, the vertical axis represents the frequency components, and the numerical values ​​represent the energy intensity at the corresponding time and frequency.

[0027] Simultaneously, 3D convolutional feature extraction is performed on the gesture image signal. This process uses multiple consecutive frames of depth images as input, employing a 3×3 convolution kernel for feature extraction in the spatial dimension and a 5-frame sliding window for motion feature capture in the temporal dimension. The specific calculation process of the convolution operation is as follows: for each pixel in the input image, its 3×3 neighborhood is multiplied element-wise with the convolution kernel, the sums are obtained, and then the feature value is output through an activation function. After three layers of convolution operations, a gesture motion sequence is extracted, which contains the motion trajectory information of key hand points.

[0028] The speech time-frequency spectrum and the gesture motion sequence are used as mixed observation signals input into the blind source separation process. First, the speech time-frequency spectrum is divided into time windows, each 20 milliseconds long. The spectral energy distribution features within each time window are extracted, and the mean and variance of the energy values ​​in each frequency band are calculated to form a speech spectral feature vector. Then, the gesture motion sequence is sampled at a frequency of 30 Hz. The coordinate change rate of key points in each frame is extracted, and the Euclidean distance and angular change of key point movement between adjacent frames are calculated to form a gesture motion feature vector.

[0029] Multimodal fusion of speech spectral feature vectors and gesture motion feature vectors is performed. The fusion process employs feature concatenation, connecting the two vectors along their feature dimensions to form a hybrid feature vector. Principal component analysis (PCA) is then applied to this hybrid feature vector for dimensionality reduction. The covariance matrix of the hybrid feature vector is calculated, and its eigenvalues ​​and eigenvectors are solved. The eigenvectors corresponding to the k largest eigenvalues ​​are selected as the projection matrix, projecting the original hybrid feature vector onto the lower-dimensional space to generate a hybrid feature matrix. The hybrid feature matrix undergoes whitening preprocessing. By calculating the square root of the inverse of the covariance matrix of the hybrid feature matrix, a linear transformation is applied to the original data to eliminate correlations between dimensions.

[0030] The whitened mixed feature matrix is ​​input into the blind source separation algorithm. This algorithm uses the mutual information value of the mixed feature matrix as the loss function, calculated using a difference method based on edge entropy and joint entropy. The separation matrix is ​​iteratively optimized using stochastic gradient descent. In each iteration, the gradient of the loss function with respect to the separation matrix is ​​calculated, and the separation matrix parameters are updated with a learning rate of 0.001 to minimize the loss function. The iteration terminates when the change in the loss function is less than 0.0001 after 10 consecutive iterations, at which point the separation matrix converges.

[0031] The converged separation matrix is ​​used to perform a linear transformation on the mixed feature matrix. The linear transformation is calculated by multiplying the separation matrix and the mixed feature matrix to obtain statistically independent speech command components and gesture components. The speech command components retain the semantic information in the speech time-frequency spectrum, while the gesture components retain the spatial motion information in the gesture motion sequence.

[0032] Time synchronization and spatial registration were performed between infrared thermal imaging signals and millimeter-wave radar signals. Time synchronization employed a linear interpolation method, using the timestamp of the infrared thermal imaging signal as a reference, and resampling the millimeter-wave radar signal to ensure both signals had the same time sequence. Spatial registration was achieved through coordinate transformation, converting the polar coordinate system data of the millimeter-wave radar to a Cartesian coordinate system consistent with the infrared thermal imaging, establishing a spatial correspondence between the two signals, and generating aligned multimodal signals.

[0033] The aligned multimodal signals are dynamically fused using a Kalman filter algorithm. The Kalman filter algorithm comprises two stages: prediction and update. In the prediction stage, predicted state values ​​and covariance values ​​are calculated based on the system state equations. State variables include target position, velocity, and temperature information. In the update stage, the predicted values ​​are corrected using observed values, the Kalman gain matrix is ​​calculated, and then the final state estimate is obtained by combining the observed and predicted values, outputting a dynamic environmental perception map. This environmental perception map integrates temperature distribution information from infrared thermal imaging and motion trajectory information from millimeter-wave radar, accurately reflecting the dynamic changes in the environment.

[0034] In S3, Mel-frequency cepstral coefficient analysis is performed on the speech command components to extract speech intent features; skeletal keypoint tracking is performed on the gesture motion components to extract gesture trajectory features; simultaneously, convolutional neural network processing is applied to the dynamic environment perception map to extract environmental anomaly features, specifically including: In the feature extraction process, the speech command components are first preprocessed by framing and windowing. Framing divides the speech command components into 25-millisecond frames with a frame shift of 10 milliseconds. Windowing uses the Hanning window function, which is expressed as 0.5 minus 0.5 multiplied by the cosine function value. The parameter of the cosine function is 2π multiplied by n divided by N minus 1, where n represents the current sampling point number and N represents the window length of 256 sampling points. Mel frequency cepstral coefficients are calculated for each frame of signal. The specific process includes: performing a fast Fourier transform on each frame of signal to obtain the spectrum; passing the spectrum through a 40-Melb filter bank; the Mel frequency is calculated as 2595 multiplied by a logarithmic function to the base 10, where the parameter of the logarithmic function is 1 plus the linear frequency divided by 700; taking the logarithm of the filter bank output and performing a discrete cosine transform; and taking the first 13 coefficients as the basic features. The coefficients are mapped to a high-dimensional space through a nonlinear transformation. The nonlinear transformation uses a hyperbolic tangent function to map each coefficient value to the interval from negative 1 to positive 1, thus obtaining the speech intent feature vector.

[0035] Using the speech intent feature vector as a temporal reference, the motion trajectory of the gesture components is segmented. Motion trajectory segmentation is based on the time point of the speech intent feature vector, dividing continuous gestures into multiple time segments. Based on the segmentation results, spatial coordinate sequences of skeletal keypoints are extracted, including 15 joints such as the wrist, elbow, and shoulder. The rate of change of Euclidean distance between keypoints is calculated. The Euclidean distance is calculated as the square root of the sum of the squares of the differences in the coordinates of two keypoints. The rate of change is calculated as the ratio of the distance difference between adjacent frames to the time interval, generating a gesture trajectory feature vector.

[0036] The gesture trajectory feature vector and the speech intent feature vector are concatenated into a 3D tensor. During the concatenation process, the temporal dimensions of the two vectors are first aligned using linear interpolation to ensure they have the same time length. Spatial expansion is then performed, extending the dimension of each feature vector from two to three dimensions, with the third dimension representing the number of feature channels. The concatenation operation connects the two 3D features along the channel dimension, forming a multimodal guidance tensor. The three dimensions of this tensor represent the time step, feature dimension, and number of channels, respectively.

[0037] A nonlinear transformation is performed on the multimodal guidance tensor to generate an attention weight matrix that matches the size of the dynamic environment perception map. First, the multimodal guidance tensor is decomposed along the channel dimension into multiple feature sub-tensors, with every two channels forming a new sub-tensor. A piecewise nonlinear function is applied to each feature sub-tensor, dynamically adjusting its slope based on the tensor value distribution: a slope of 0.1 when the tensor value is less than 0, a slope of 1 when the tensor value is between 0 and 1, and a slope of 0.5 when the tensor value is greater than 1. The initial weight distributions are then fused across channels using element-wise addition and global average pooling. Element-wise addition adds weights at the same position, while global average pooling averages all weights for each channel. This generates an unadjusted attention weight matrix.

[0038] The unadjusted attention weight matrix is ​​scaled using bilinear interpolation. Bilinear interpolation uses the weight values ​​of four adjacent points for linear interpolation calculation. The interpolation formula is based on the linear interpolation results in two directions, followed by another linear interpolation. Through this interpolation operation, the spatial dimensions of the attention weight matrix are made completely consistent with the dynamic environment perception map, forming the final attention weight matrix.

[0039] The attention weight matrix is ​​multiplied element-wise with the dynamic environment perception map. Element-wise multiplication means multiplying the weight values ​​at the same locations with the perception map values ​​to complete the region attention weighting. In the weighted perception map, the response values ​​of abnormally sensitive regions are enhanced by 2 to 5 times the original weight values, while the response values ​​of non-critical regions are suppressed by 0.1 to 0.5 times the original weight values.

[0040] Local extremum detection is performed on the weighted dynamic environment perception map. A 3x3 sliding window is used for local extremum detection, comparing the response values ​​of the center point with those of its eight surrounding points within the window. When the response value of the center point is simultaneously greater than that of all its neighboring points and exceeds a preset threshold of 0.7, the area is marked as an abnormally sensitive region. The preset threshold is dynamically adjusted based on historical data, with an adjustment increment of 1.5 times the average response value over the most recent 100 detection periods.

[0041] Multi-level convolutional operations are performed on the weighted dynamic environment-aware map. The first convolutional layer uses 64 3x3 kernels with a stride of 1 and padding of 1. The second convolutional layer uses 128 3x3 kernels with a stride of 2 and padding of 1. The third convolutional layer uses 256 3x3 kernels with a stride of 2 and padding of 1. Shallow and deep features are fused through residual connections, which add the outputs of the first and third convolutional layers. Before adding, the number of channels is adjusted using a 1x1 convolution. Finally, channel pooling is used to extract environmental anomaly features. Channel pooling takes the maximum value of all channels at each spatial location to form the final environmental anomaly feature vector.

[0042] In S4, a multimodal decision matrix is ​​constructed based on voice intent features, gesture trajectory features, and environmental anomaly features. The decision matrix is ​​then weighted and evaluated using a pre-defined fuzzy logic rule base to generate a control command priority sequence. Based on this priority sequence, the operating mode of the smart home devices is dynamically adjusted, and an adaptive control strategy is output, specifically including: In the decision-making and control process, the voice intent features, gesture trajectory features, and environmental anomaly features are first time-aligned. A dynamic time warping algorithm is used to process the three feature sequences. This algorithm finds the optimal alignment path by constructing a cumulative cost matrix. The cumulative cost matrix is ​​calculated as follows: each element equals the current feature distance plus the minimum cumulative cost of the three adjacent directions. The feature distance is calculated using Euclidean distance. By backtracking to the optimal path, the three feature sequences are adjusted to the same temporal length, eliminating the time-scale inconsistency caused by differences in sampling frequencies.

[0043] Based on the start and end times of all time-equal duration sequences, a uniformly sampled reference timeline is generated. The start time of the reference timeline is the maximum of the start times of the three sequences, the end time is the minimum of the end times of the three sequences, and the sampling interval is the least common multiple of the original sampling intervals of the three sequences. The reference timeline covers the common time range of all sequences, ensuring the integrity of time synchronization.

[0044] For each time series of equal duration, using the original time points as interpolation nodes, cubic spline interpolation is applied to calculate the eigenvalues ​​of each sampling point on the reference time axis. Cubic spline interpolation requires the interpolation function to have continuous first and second derivatives at the nodes. The specific calculation process is as follows: First, solve the tridiagonal linear equation system to obtain the second derivative value of each interval segment; the coefficients of the equation system are determined by the distance between adjacent nodes. Then, use the Hermitian interpolation formula to calculate the eigenvalues ​​of each point on the reference time axis, generating a continuous and smooth interpolated eigenvalue sequence.

[0045] Boundary conditions are optimized for each interpolated feature sequence. Natural boundary conditions are used, meaning the second derivative at the sequence endpoints is zero. This is achieved by symmetrically expanding the data points at both ends of the sequence: adding two virtual data points before the start point with values ​​equal to the start point value, and adding two virtual data points after the end point with values ​​equal to the end point value. This eliminates boundary oscillations during interpolation, ensuring the smoothness of the sequence boundaries.

[0046] The optimized interpolated feature sequences are combined according to the time points of the reference time axis. Each time point contains the synchronized values ​​of three features, forming a time-synchronized multimodal feature group. The data structure of the feature group is a three-dimensional array, where the first dimension represents the time point number, the second dimension represents the feature type, and the third dimension represents the feature value.

[0047] Feature intensity normalization is performed on the time-synchronized multimodal feature set. A min-max normalization method is used to linearly map the numerical range of each feature to the interval 0 to 1. The normalization formula is: the normalized value equals the original value minus the minimum value, divided by the difference between the maximum and minimum values. The minimum and maximum values ​​are taken as the minimum and maximum observed values ​​of the feature across all time points, respectively.

[0048] The normalized features are arranged and combined according to three dimensions: feature type, time series, and feature intensity. The first dimension, feature type, is arranged in a fixed order: speech intent features, gesture trajectory features, and environmental anomaly features. The second dimension, time series, is arranged in the order of time points on the reference time axis. The third dimension, feature intensity, stores the normalized feature values. This arrangement constructs a three-dimensional decision matrix.

[0049] The three-dimensional decision matrix is ​​input into a pre-defined fuzzy logic rule base. The rule base contains 25 fuzzy rules, each consisting of a premise and a conclusion. The premise is a combination of fuzzy conditions for feature strength, and the conclusion is the activation strength of the corresponding control instruction. Each feature is fuzzified using a membership function, which is a triangular function. Each feature is assigned three fuzzy sets: "low," "medium," and "high."

[0050] Rule-based reasoning is performed on the fuzzy decision set. The Mamdani inference method is used, with the rule activation strength set to the minimum membership degree of the premise. Fuzzy clustering is performed on the output of each rule, with the cluster center set being the center value of the fuzzy set of the rule's conclusion. Weight coefficients are dynamically adjusted based on the correlation strength between features. The correlation strength is calculated using the Pearson correlation coefficient between features; features with a correlation coefficient greater than 0.7 have a weight adjustment coefficient of 1.2, and features with a correlation coefficient less than 0.3 have a weight adjustment coefficient of 0.8.

[0051] Calculate the overall priority score for each control command. The overall score consists of three parts: a base score, which is the fuzzy value output by rule inference; a time weight, which decays exponentially based on the time of feature occurrence; and a feature weight, which is set with a fixed coefficient according to the feature type. The weight coefficient for environmental anomaly features is 0.5, the weight coefficient for voice intent features is 0.3, and the weight coefficient for gesture trajectory features is 0.2. The final score is calculated as a weighted sum of the scores for each component.

[0052] The control command priority sequence is generated by sorting commands from highest to lowest based on their overall priority scores. The sorting algorithm uses quicksort, with a time complexity of O(nlogn). Commands with the same score are arranged in chronological order, with earlier commands having higher priority. The generated control command priority sequence is used to dynamically adjust the operating mode of smart home devices.

[0053] The operating modes of smart home devices are dynamically adjusted based on a priority sequence. The adjustment strategy is set based on the priority score difference. When the score difference between adjacent commands is greater than 0.1, a mode switch is executed; when the score difference is between 0.05 and 0.1, parameter adjustments are performed; and when the score difference is less than 0.05, the current mode is maintained. This adjustment method outputs an adaptive control strategy to achieve optimized control of the smart home system.

[0054] Please see Figure 2 As shown, a smart home central control system based on multimodal perception includes: The multimodal signal acquisition module is used to simultaneously acquire user interaction signals and environmental status signals during the operation of the smart home central control system. The user interaction signals include voice audio signals and gesture image signals, and the environmental status signals include infrared thermal imaging signals and millimeter-wave radar signals. Signal decoupling and fusion are used to treat speech audio signals and gesture image signals as mixed observation signals, and separate statistically independent speech command components and gesture action components; infrared thermal imaging signals and millimeter-wave radar signals are spatiotemporally aligned and fused using a Kalman filter algorithm to generate a dynamic environment perception map. The multimodal feature extraction module is used to perform Mel-frequency cepstral coefficient analysis on the speech command components to extract speech intent features; to perform skeletal keypoint tracking on the gesture action components to extract gesture trajectory features; and to process the dynamic environment perception map using a convolutional neural network to extract environmental anomaly features. The intelligent control decision module constructs a multimodal decision matrix based on voice intent features, gesture trajectory features, and environmental anomaly features. It then performs a weighted evaluation of the decision matrix using a preset fuzzy logic rule base to generate a control command priority sequence. Based on the priority sequence, it dynamically adjusts the operation mode of the smart home devices and outputs an adaptive control strategy.

[0055] The working principle of this invention is as follows: By simultaneously acquiring speech audio signals, gesture image signals, infrared thermal imaging signals, and millimeter-wave radar signals, statistically independent speech command components and gesture action components are extracted from the speech and gesture signals using blind source separation technology. A dynamic environment perception map is generated from the infrared and radar signals through spatiotemporal alignment and Kalman filtering. Then, speech intent features, gesture trajectory features, and environmental anomaly features are extracted through Mel-frequency cepstral coefficient analysis, skeletal keypoint tracking, and multi-level convolution processing, respectively. Finally, a three-dimensional decision matrix is ​​constructed based on these features, and a weighted evaluation using a fuzzy logic rule base is performed to generate a control command priority sequence. The device operation mode is dynamically adjusted according to the priority to achieve an adaptive control strategy.

[0056] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A multi-modal perception based smart home central control method, characterized in that, The method comprises the following steps: S1: in the process of running the smart home control system, synchronously collecting user interaction signals and environment state signals, wherein the user interaction signals include voice audio signals and gesture image signals, and the environment state signals include infrared thermal imaging signals and millimeter wave radar signals; S2: taking the voice audio signals and gesture image signals as mixed observation signals, separating out statistically independent voice command components and gesture action components; performing time and space alignment on the infrared thermal imaging signals and millimeter wave radar signals, and generating a dynamic environment perception map through Kalman filtering algorithm fusion; S3: performing mel-frequency cepstrum coefficient analysis on the voice command components to extract voice intention features; performing skeleton key point tracking on the gesture action components to extract gesture trajectory features; and simultaneously performing convolutional neural network processing on the dynamic environment perception map to extract environment anomaly features; S4: based on the voice intention features, gesture trajectory features and environment anomaly features, constructing a multi-modal decision matrix, performing weighted evaluation on the decision matrix through a pre-set fuzzy logic rule base to generate a control instruction priority sequence; and dynamically adjusting the operation mode of the smart home device according to the priority sequence to output an adaptive control strategy. 2.The multi-modal perception based smart home central control method of claim 1, wherein, The S2 specifically comprises: performing multi-scale time-frequency analysis on the voice audio signals to obtain a voice time-frequency spectrum, and simultaneously performing three-dimensional convolution feature extraction on the gesture image signals to obtain a gesture motion sequence; inputting the voice time-frequency spectrum and the gesture motion sequence into a blind source separation network to separate out statistically independent voice command components and gesture action components by maximizing mutual information and minimizing a loss function; performing time synchronization and space registration on the infrared thermal imaging signals and the millimeter wave radar signals to generate aligned multi-modal signals, and outputting a dynamic environment perception map through Kalman filtering algorithm dynamic fusion. 3.The multi-modal perception based smart home central control method of claim 2, wherein, The separation of the statistically independent voice command components and gesture action components specifically comprises: dividing the voice time-frequency spectrum into time windows to extract the spectral energy distribution features in each time window to form a voice spectral feature vector; and simultaneously sampling the gesture motion sequence to extract the gesture key point coordinate change rate of each frame to form a gesture motion feature vector; performing multi-modal fusion on the voice spectral feature vector and the gesture motion feature vector, performing dimension reduction processing through principal component analysis to generate a mixed feature matrix, and performing white pre-processing on the mixed feature matrix to eliminate correlation; inputting the white mixed feature matrix into a blind source separation algorithm, calculating the mutual information value of the mixed feature matrix as a loss function, and using a stochastic gradient descent method to iteratively optimize a separation matrix to minimize the loss function until the separation matrix converges; using the converged separation matrix to perform linear transformation on the mixed feature matrix to obtain the statistically independent voice command components and gesture action components.

4. The multi-modal perception based smart home central control method of claim 1, wherein, The S3 specifically comprises: performing frame division and windowing preprocessing on the voice command components, calculating mel-frequency cepstrum coefficients, and mapping the coefficients to a high-dimensional space through nonlinear transformation to obtain a voice intention feature vector; The gesture action component is motion trajectory segmented by taking the speech intention feature vector as a time sequence reference, a spatial coordinate sequence of a skeleton key point is extracted based on a segmentation result, and a gesture trajectory feature vector is generated by calculating a Euclidean distance change rate between the key points; The gesture trajectory feature vector and the speech intention feature vector are tensor spliced to form a multi-modal guidance tensor, and the multi-modal guidance tensor is used to perform regional attention weighting on the dynamic environment perception graph to highlight the abnormal sensitive area; A multi-level convolution operation is performed on the weighted dynamic environment perception graph, shallow and deep features are fused through residual connection, and finally an environmental anomaly feature is extracted through channel pooling.

5. The multi-modal perception based smart home central control method of claim 4, wherein, The highlighted abnormal sensitive area specifically includes: The gesture trajectory feature vector and the speech intention feature vector are three-dimensional tensor spliced, wherein the time dimension is aligned and the space dimension is expanded to form a multi-modal guidance tensor; A nonlinear transformation is performed on the multi-modal guidance tensor to generate an attention weight matrix matching the size of the dynamic environment perception graph; The attention weight matrix and the dynamic environment perception graph are multiplied element by element to complete regional attention weighting, so that the response value of the abnormal sensitive area is strengthened and the response value of the non-key area is inhibited; Local extreme value detection is performed on the weighted dynamic environment perception graph, and the area with a response value exceeding a preset threshold is marked as the final abnormal sensitive area. 6.The multi-modal perception based smart home central control method of claim 5, wherein, The nonlinear transformation of the multi-modal guidance tensor to generate an attention weight matrix matching the size of the dynamic environment perception graph specifically includes: The multi-modal guidance tensor is decomposed into multiple feature sub-tensors along the channel dimension, each feature sub-tensor corresponding to a different modal interaction mode; A segmented nonlinear function is applied to each feature sub-tensor for transformation, wherein the segmented nonlinear function dynamically adjusts the slope according to the distribution of the tensor value to generate a preliminary weight distribution; The preliminary weight distributions are cross-channel fused to generate an unadjusted attention weight matrix through element-wise addition and global average pooling operations; The unadjusted attention weight matrix is bilinearly interpolated and scaled to make its spatial size completely consistent with that of the dynamic environment perception graph to form the final attention weight matrix.

7. The multi-modal perception based smart home central control method of claim 1, wherein, The S4 specifically includes: Align the speech intention feature, gesture trajectory feature, and environmental anomaly feature along the time dimension to construct a three-dimensional decision matrix, wherein the first dimension represents the feature type, the second dimension represents the time sequence, and the third dimension represents the feature intensity; Input the three-dimensional decision matrix into a preset fuzzy logic rule base, fuzzify each feature through a membership function to generate a fuzzy decision set; Rule inference is performed on the fuzzy decision set, the weight coefficients are dynamically adjusted according to the correlation strength between the features, and the comprehensive priority score of each control instruction is calculated; The control instruction priority sequence is generated in descending order of the comprehensive priority score, wherein the environmental anomaly feature has the highest weight coefficient, the speech intention feature has the second highest weight coefficient, and the gesture trajectory feature has a basic weight coefficient. 8.The multi-modal perception based smart home central control method of claim 7, wherein, The construction process of the three-dimensional decision matrix is: The speech intention feature, gesture trajectory feature, and environmental anomaly feature are processed respectively using a dynamic time warping algorithm to make the three feature sequences have the same time sequence length; The processed equal-length time sequences are corresponded according to time points, the timestamps of each feature are aligned through a cubic spline interpolation method, and a time-synchronized multi-modal feature group is formed; The time-synchronized multi-modal feature group is subjected to feature intensity normalization processing, and the numerical range of each feature is uniformly mapped to a preset interval; The normalized features are arranged and combined according to three dimensions of feature type, time sequence and feature intensity, and a completed three-dimensional decision matrix is constructed. 9.The multi-modal perception based smart home central control method of claim 8, wherein, The time-synchronized multi-modal feature group is formed, specifically including: Based on the start and end time points of all equal-length time sequences, a uniformly sampled reference time axis is generated, covering the common time range of all sequences; For each equal-length time sequence, the original time points are used as interpolation nodes, and a cubic spline interpolation method is applied to calculate the feature values of each sampling point on the reference time axis to generate a continuous and smooth interpolation feature sequence; Each interpolation feature sequence is subjected to boundary condition optimization by symmetrically expanding the data points at both ends of the sequence; The optimized interpolation feature sequences are corresponded and combined according to the time points of the reference time axis to form a time-synchronized multi-modal feature group.

10. A multi-modal perception based smart home central control system, characterized in that, A multi-modal perception-based smart home central control method according to any one of claims 1-9, comprising: A multi-modal signal acquisition module for synchronously acquiring user interaction signals and environmental state signals during the operation of the smart home central control system, wherein the user interaction signals include voice audio signals and gesture image signals, and the environmental state signals include infrared thermal imaging signals and millimeter wave radar signals; Signal decoupling and fusion, for separating the voice audio signals and gesture image signals as mixed observation signals into statistically independent voice command components and gesture action components; and performing time and space alignment on the infrared thermal imaging signals and millimeter wave radar signals to generate a dynamic environment perception map through Kalman filtering algorithm fusion; A multi-modal feature extraction module for performing Mel frequency cepstrum coefficient analysis on the voice command components to extract voice intent features, performing skeleton key point tracking on the gesture action components to extract gesture trajectory features, and simultaneously performing convolutional neural network processing on the dynamic environment perception map to extract environmental anomaly features; An intelligent control decision module for constructing a multi-modal decision matrix based on the voice intent features, gesture trajectory features and environmental anomaly features, performing weighted evaluation on the decision matrix through a preset fuzzy logic rule base to generate a control instruction priority sequence, and dynamically adjusting the operation mode of the smart home devices according to the priority sequence to output an adaptive control strategy.

Images