Smart home privacy protection monitoring system based on multi-mode sensor fusion

Abstract

The invention discloses a smart home privacy protection monitoring system based on multi-modal sensor fusion, and relates to the technical field of privacy protection, a multi-modal sensing module is arranged to collect scene data, a privacy object recognition module is arranged to obtain a privacy object recognition result from the scene data, and the privacy object recognition result is sent to the privacy protection module. Setting a multi-modal feature fusion module to generate fusion feature data, setting a privacy protection processing module, generating a preliminary protection processing result according to privacy object position information in the fusion feature data, and setting a dynamic protection adjustment module to adjust the privacy object position information according to a user distance parameter, a scene importance parameter and a time dimension parameter. Adjusting a protection strength parameter in a preliminary protection processing result, generating a self-adaptive protection result, setting a multi-mode interaction module, receiving the self-adaptive protection result sent by a dynamic protection adjustment module, and modifying the self-adaptive protection result according to a user input parameter, thereby effectively solving the privacy protection problem in smart home monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of privacy protection, and particularly relates to an intelligent home privacy protection monitoring system based on multi-modal sensor fusion. BACKGROUND

[0002] With the rapid development of smart home technology, home monitoring systems have become an important part of ensuring home safety. However, traditional monitoring systems, while providing security, also pose a serious risk of privacy leakage. Existing smart home monitoring systems mainly have the following problems: Traditional monitoring systems usually use a single sensor (such as a camera) for data collection, which is easily disturbed by factors such as light changes, obstructions, and noise in complex environments, resulting in unstable monitoring results and an inability to fully perceive scene information. The limitations of this single data source make it difficult for the system to adapt to changing home environments, reducing the reliability and effectiveness of monitoring.

[0003] Existing privacy protection technologies mostly use static, pre-set protection strategies, such as fixed area blurring or mosaic processing, which cannot adaptively adjust protection strength and methods according to scene dynamics and user needs. This rigid protection mechanism makes it difficult to balance protection effectiveness and user experience, either over-protecting and affecting monitoring effectiveness, or under-protecting and causing privacy leakage.

[0004] In handling occlusion scenarios, existing systems are ineffective when privacy objects are partially occluded, often failing to accurately identify and protect them, increasing the risk of privacy leakage. Due to the lack of multi-modal data fusion and advanced recognition algorithms, the system cannot infer information from the occluded part, resulting in discontinuous or ineffective protection.

[0005] Therefore, there is an urgent need for an intelligent home privacy protection monitoring system that can fully perceive the environment, intelligently identify privacy objects, dynamically adjust protection strategies, provide diversified protection methods, and support multi-modal interaction, to solve the above technical problems.

[0006] To this end, the present application proposes an intelligent home privacy protection monitoring system based on multi-modal sensor fusion. SUMMARY

[0007] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes an intelligent home privacy protection monitoring system based on multi-modal sensor fusion, effectively solving the problem of privacy protection in smart home monitoring.

[0008] In order to achieve the above object, a kind of intelligent home privacy protection monitoring system based on multi-modal sensor fusion is presented, including multi-modal perception module, privacy object identification module, multi-modal feature fusion module, privacy protection processing module, dynamic protection adjustment module and multi-modal interaction module;Wherein, each module is connected by electrical mode; Multi-modal perception module is composed of high-resolution camera, millimeter wave radar, IMU sensor and environmental sensor, for collecting scene data, and the scene data is sent to privacy object identification module and multi-modal feature fusion module; Privacy object identification module inputs image data in the scene data into improved YOLOv7 model, executes privacy object identification algorithm, and obtains privacy object identification result by combining point cloud data in the scene data and the IMU data, and sends the privacy object identification result to multi-modal feature fusion module; Multi-modal feature fusion module inputs the scene data and the privacy object identification result into Transformer architecture, executes multi-modal feature fusion algorithm, the multi-modal feature fusion algorithm includes attention mechanism processing module, generates fusion feature data, and sends the fusion feature data to privacy protection processing module and dynamic protection adjustment module; Privacy protection processing module executes multi-level privacy protection processing according to privacy object position information in the fusion feature data, and the multi-level privacy protection processing includes applying pixel-level mosaic algorithm, region blurring algorithm and scene semantic replacement algorithm to the privacy object, generates preliminary protection processing result, and sends the preliminary protection processing result to dynamic protection adjustment module; Dynamic protection adjustment module extracts user distance parameter, scene importance parameter and time dimension parameter from the fusion feature data, adjusts protection intensity parameter in the preliminary protection processing result according to the user distance parameter, the scene importance parameter and the time dimension parameter, generates adaptive protection result, and sends the adaptive protection result to multi-modal interaction module; Multi-modal interaction module is composed of gesture recognition module, voice instruction module and mobile terminal APP module, receives the adaptive protection result sent by the dynamic protection adjustment module, receives user input parameter, the user input parameter includes privacy area coordinate and protection strategy parameter, modifies the adaptive protection result according to the user input parameter, and displays the modified adaptive protection result.

[0009] The collection scene data includes the following steps: Step 11: configure high-resolution camera to collect image data, the high-resolution camera is installed on the top of indoor space and wall surface; Step 12: Deploy millimeter wave radar to collect point cloud data to perceive human body position and motion state; Step 13: Integrate IMU sensor to collect IMU data to monitor user motion posture, the IMU sensor includes three-axis accelerometer, three-axis gyroscope and three-axis magnetometer; Step 14: Arrange environmental sensors to collect indoor environmental parameters, the environmental sensors include temperature sensor, humidity sensor, light sensor and sound sensor; Step 15: Time synchronize and spatially register the image data collected by the high-resolution camera, the point cloud data perceived by the millimeter wave radar, the IMU data perceived by the IMU sensor and the indoor environmental parameters collected by the environmental sensor, to form the scene data; The execution of the privacy object recognition algorithm includes the following steps: Step 21: Construct a privacy object detection network based on the improved YOLOv7 model, the privacy object includes face, electronic device screen, file, personal item and privacy part; Step 22: Input the image data in the scene data into the improved YOLOv7 model, execute the forward inference process, and generate a feature map; Step 23: Apply anchor box prediction and non-maximum suppression algorithm to the feature map to obtain the recognition result of the privacy object, the recognition result includes the bounding box, class and confidence of the privacy object; Step 24: Combine the recognition result with the point cloud data and the IMU data to perform multi-modal verification on the detection result of the privacy object, and the verified recognition result is used as the final privacy object recognition result; The execution of the multi-modal feature fusion algorithm includes the following steps: Step 31: Construct a multi-modal feature fusion network based on the Transformer architecture, the Transformer architecture includes an encoder and a decoder; Step 32: Feature encoding of the privacy object recognition result to generate a visual feature sequence; The feature encoding first converts the bounding box, class, and confidence information in the privacy object recognition result into a feature vector. The bounding box information includes center coordinates, width and height, and area, which are mapped to the 0-1 range through normalization processing. The class information is represented by one-hot encoding, and the confidence information is directly used as a feature dimension. Then the feature vector is fused with the intermediate layer feature map of the improved YOLOv7 model to extract the visual features of the privacy object. The visual features are encoded by spatial position to increase the position information, and mapped to the input dimension of the Transformer through a linear projection layer. The length of the visual feature sequence is equal to the number of detected privacy objects, and each sequence element corresponds to a feature representation of a privacy object.

[0010] Step 33: Feature extraction of human body position and motion state collected by the millimeter wave radar to generate a radar feature sequence; The feature extraction of human body position and motion state collected by the millimeter wave radar first voxelizes the point cloud data generated by the millimeter wave radar, divides the three-dimensional space into uniform grids, and calculates the statistical features of the points in each grid, including the number of points, average reflection intensity, and velocity vector. Then the spatial features of the point cloud are extracted by a three-dimensional convolutional neural network, which contains 4 convolutional blocks, each containing two 3D convolutional layers, a batch normalization layer, and a ReLU activation layer. The output feature map of the three-dimensional convolutional neural network is converted into a feature vector through global average pooling and a fully connected layer. The length of the radar feature sequence is equal to the number of detected targets, and each sequence element corresponds to a radar feature representation of a target.

[0011] Step 34: Feature extraction of the IMU sensor and the environmental parameter data to generate an IMU feature sequence and an environmental feature sequence, respectively; The IMU feature extraction uses an LSTM network to process time series data to capture the time dependence of user motion. The LSTM network contains two layers of bidirectional LSTM units with a hidden state dimension of 256, and the input is the original data sequence of the IMU sensor, including acceleration, angular velocity, and magnetic field strength. The output of the LSTM network is weighted and summed through an attention layer to generate a fixed-length feature vector. The environmental feature extraction uses a multi-layer perceptron to process environmental parameter data. The MLP contains three fully connected layers with hidden layer dimensions of 128, 64, and 32, respectively, and the activation function is ReLU. The environmental feature sequence is sampled through a time window sliding window with a window length of 5 seconds and a step length of 1 second to capture the short-term trend of environmental parameters.

[0012] Step 35: Introduce an attention mechanism for the Transformer architecture to handle occlusion scenarios to enhance the recognition ability of partially visible privacy objects; The attention mechanism includes two branches of spatial attention and channel attention. The spatial attention branch generates a spatial attention map through convolution operation, highlights the significant area in the image, and suppresses background interference. The channel attention branch generates channel descriptors through global average pooling and global maximum pooling, learns the correlation between channels through a fully connected layer, and generates channel attention weights. The spatial attention and channel attention are applied to the feature map through element-level multiplication to enhance the expression of key features. For the occlusion scene, the attention mechanism combines temporal information and multi-modal information, keeps track of the occluded object through the memory mechanism, and supplements the spatial information of the occluded part through radar point cloud data, to realize the robust recognition of partially visible private objects.

[0013] Step 36: input the visual feature sequence, the radar feature sequence, the IMU feature sequence and the environment feature sequence into the attention mechanism added Transformer architecture, perform multi-modal feature fusion, and generate fusion feature data; The step of performing multi-level privacy protection processing to generate the preliminary protection processing result includes the following steps: Step 41: according to the private object position information in the fusion feature data, constructing a privacy protection area mapping to determine the image area that needs to be protected; Step 42: applying a pixel-level mosaic algorithm to the private object to realize basic privacy protection; Step 43: applying a region blurring algorithm to the private object to enhance the protection effect; Step 44: applying a scene semantic replacement algorithm to the private object to realize visual-friendly privacy protection; Step 45: applying a gradual transition algorithm to the edge area of the private object to improve the visual naturalness; Step 46: fusing the processing results of the pixel-level mosaic algorithm, the region blurring algorithm, the scene semantic replacement algorithm and the gradual transition algorithm to generate a preliminary protection processing result; The step of adjusting the protection strength parameter in the preliminary protection processing result to generate an adaptive protection result includes the following steps: Step 51: extracting a user distance parameter from the fusion feature data, and adjusting the protection strength based on the spatial relationship between the user and the private object; Step 52: extracting a scene importance parameter from the fusion feature data, and adjusting the protection strategy based on the semantic importance of the private object; Step 53: extracting a time dimension parameter from the fusion feature data, and adjusting the protection dynamics based on the time persistence of the private object; Step 54: According to the user distance parameter, the scene importance parameter and the time dimension parameter, the comprehensive protection strength coefficient is calculated; Step 55: According to the comprehensive protection strength coefficient, the protection parameters in the preliminary protection processing result are adjusted to generate an adaptive protection result; The receiving user input parameters and modifying the adaptive protection result according to the user input parameters include the following steps: Step 61: A multi-modal interaction layer is constructed to collect and process user input signals; Step 62: According to the user input parameters collected by the multi-modal interaction layer, the adaptive protection result is modified; Compared with the prior art, the beneficial effects of the present application are: Firstly, the present application constructs a multi-modal perception layer by high-resolution cameras, millimeter wave radars, IMU sensors and environmental sensors to collect scene data; then the collected scene data is input into an improved YOLOv7 model to execute a privacy object recognition algorithm; then multi-modal feature fusion is performed using a Transformer architecture, and an attention mechanism is introduced to handle occlusion scenarios; then multi-level privacy protection processing is performed according to the fused feature data, and the protection strength is dynamically adjusted based on user distance, scene importance and time dimension parameters; finally, the multi-modal interaction layer receives user input parameters, modifies the adaptive protection result and outputs. The present application effectively solves the privacy protection problem in smart home monitoring through multi-modal sensor fusion and adaptive privacy protection strategy, and realizes more intelligent, more accurate and more flexible privacy protection effect. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 Figure 1 is a module connection relationship diagram of a smart home privacy protection monitoring system based on multi-modal sensor fusion in Embodiment 1 of the present application. DETAILED DESCRIPTION

[0015] The technical solutions of the present application will be described below in conjunction with embodiments, obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0016] Embodiment 1 As shown in Figure 1, a smart home privacy protection monitoring system based on multi-modal sensor fusion includes a multi-modal perception module, a privacy object recognition module, a multi-modal feature fusion module, a privacy protection processing module, a dynamic protection adjustment module and a multi-modal interaction module; wherein each module is connected by electrical means. Figure 1 ​A multi-modal perception module composed of a high-resolution camera, a millimeter wave radar, an IMU sensor, and an environmental sensor, is used to collect scene data and send the scene data to a private object recognition module and a multi-modal feature fusion module; A private object recognition module inputs image data in the scene data into an improved YOLOv7 model, executes a private object recognition algorithm, and combines point cloud data and IMU data in the scene data to obtain a private object recognition result, which is sent to a multi-modal feature fusion module; A multi-modal feature fusion module inputs the scene data and the private object recognition result into a Transformer architecture, executes a multi-modal feature fusion algorithm, which includes an attention mechanism processing module, to generate fusion feature data, which is sent to a privacy protection processing module and a dynamic protection adjustment module; A privacy protection processing module executes a multi-level privacy protection process according to private object position information in the fusion feature data, which includes applying a pixel-level mosaic algorithm, a region blurring algorithm, and a scene semantic replacement algorithm to the private object to generate a preliminary protection processing result, which is sent to a dynamic protection adjustment module; A dynamic protection adjustment module extracts user distance parameters, scene importance parameters, and time dimension parameters from the fusion feature data, adjusts protection intensity parameters in the preliminary protection processing result according to the user distance parameters, the scene importance parameters, and the time dimension parameters to generate an adaptive protection result, which is sent to a multi-modal interaction module; A multi-modal interaction module composed of a gesture recognition module, a voice instruction module, and a mobile APP module receives the adaptive protection result sent by the dynamic protection adjustment module, receives user input parameters including private region coordinates and protection strategy parameters, modifies the adaptive protection result according to the user input parameters, and displays the modified adaptive protection result.

[0017] The collection of scene data includes the following steps: Step 11: Configure a high-resolution camera to collect image data, which is installed on the top of the indoor space and the wall surface; Specifically, the high-resolution camera adopts a 4K resolution RGB-IR dual-spectrum imaging sensor, supporting image acquisition in the visible and infrared spectrum. The high-resolution cameras are distributed in a grid pattern on the top and walls of the indoor space, ensuring complete coverage of the monitoring area without blind spots. Each high-resolution camera has a field of view of 120 degrees, and there is a 30% overlap area between the fields of view of adjacent cameras, which is used for subsequent image stitching and three-dimensional scene reconstruction.

[0018] Step 12: Deploy millimeter wave radar to collect point cloud data to perceive human position and motion state; Specifically, the millimeter wave radar uses FMCW (Frequency Modulated Continuous Wave) technology, with a transmission frequency of 24 GHz and a bandwidth of 2 GHz, achieving centimeter-level distance resolution. The millimeter wave radar calculates the radial velocity and angle information of the target by measuring the Doppler shift of electromagnetic waves. The millimeter wave radar uses a MIMO (Multiple Input Multiple Output) antenna array design, containing 8 transmit antennas and 16 receive antennas, forming 128 virtual channels, achieving three-dimensional imaging capability in horizontal and vertical directions. The millimeter wave radar has a scanning period of 100 ms, and completes a complete spatial scan in each scanning period to generate point cloud data, which includes the position coordinates, velocity vector, and reflection intensity information of the target.

[0019] Step 13: Integrate IMU sensor to collect IMU data to monitor user motion posture, the IMU sensor includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer; Specifically, the IMU sensor uses a 9-axis fusion design, integrating a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. The three-axis accelerometer has a measurement range of ±16g and a resolution of 16 bits; the three-axis gyroscope has a measurement range of ±2000° / s and a resolution of 16 bits; the three-axis magnetometer has a measurement range of ±4900μT and a resolution of 14 bits. The IMU sensor fuses the data of the three sensors through a Kalman filter algorithm, and calculates the attitude angle (Euler angle or quaternion representation) and motion acceleration of the user in real time. The sampling frequency of the IMU sensor is 200 Hz, and the data is transmitted to the system's data processing unit through the Bluetooth Low Energy protocol.

[0020] Step 14: Arrange environmental sensors to collect indoor environmental parameters, including temperature sensors, humidity sensors, light sensors, and sound sensors; Specifically, the temperature sensor adopts a digital temperature sensor, with a measurement range of -40℃ to 125℃ and an accuracy of ±0.5℃; the humidity sensor adopts a capacitive humidity sensor, with a measurement range of 0%RH to 100%RH and an accuracy of ±3%RH; the illumination sensor adopts a photodiode, with a measurement range of 0.1 lux to 100000 lux and a dynamic range of 120dB; the sound sensor adopts a MEMS microphone array, containing 6 omnidirectional microphones, with a sampling rate of 48kHz and a signal-to-noise ratio greater than 65dB. The environmental sensor forms a sensor network through a ZigBee wireless network, collects environmental data at a frequency of 1Hz, and transmits the data to the data processing unit of the system.

[0021] Step 15: Time synchronization and spatial registration of the image data collected by the high-resolution camera, the point cloud data perceived by the millimeter wave radar, the IMU data perceived by the IMU sensor, and the indoor environmental parameters collected by the environmental sensor to form the scene data; Specifically, the time synchronization adopts a combination of Network Time Protocol (NTP) and Precision Time Protocol (PTP), ensuring that the timestamp error of each sensor is less than 1ms. The system assigns a unique clock source to each sensor, achieving global clock synchronization through master-slave clock architecture. The spatial registration adopts multi-sensor calibration technology, first determining the intrinsic and extrinsic parameters of the high-resolution camera through a calibration board, and then determining the spatial transformation matrix of the millimeter wave radar, the IMU sensor, and the environmental sensor relative to the high-resolution camera using a feature point matching algorithm. The spatial registration process optimizes the transformation matrix using the Iterative Closest Point (ICP) algorithm, so that the data collected by each sensor is represented in a unified world coordinate system.

[0022] Further, the execution of the privacy object recognition algorithm to obtain a privacy object recognition result includes the following steps: Step 21: Building a privacy object detection network based on the improved YOLOv7 model, the privacy objects including faces, electronic device screens, files, personal items, and private parts; Specifically, the improved YOLOv7 model is optimized on the basis of the original YOLOv7 architecture, including introducing an attention mechanism, enhancing a feature extraction network, and optimizing a loss function. The attention mechanism adopts a combination of spatial attention and channel attention to enhance the model's perception of privacy object detail features. The feature extraction network adopts CSPDarknet53 as the backbone network and adds a depth separable convolution layer to reduce the parameter amount while improving the feature extraction capability. The loss function combines cross-entropy loss, IoU loss, and focal loss to balance detection accuracy and recall rate and improve small target detection performance.

[0023] Step 22: Input the image data from the scene data into the improved YOLOv7 model, perform the forward inference process, and generate feature maps; Specifically, the forward inference process first extracts multi-scale features through the CSPDarknet53 backbone network, generating five feature maps with different resolutions, corresponding to high-resolution features in the shallow layer and low-resolution features in the deep layer. Then, feature pyramid network (FPN) and path aggregation network (PAN) are used to fuse upsampling and downsampling of the features, enhancing their semantic and spatial information. The feature fusion process employs residual connections and skip connections to alleviate the gradient vanishing problem and improve feature propagation efficiency. Finally, an attention module weights the feature maps, highlighting key features of privacy-sensitive objects and suppressing background interference. The generated feature maps contain rich semantic and spatial information, providing a foundation for subsequent object detection.

[0024] Step 23: Apply anchor box prediction and nonmaximum suppression algorithms to the feature map to obtain the identification results of privacy objects. The identification results include the bounding box, category, and confidence score of the privacy objects. Specifically, the anchor box prediction is based on a preset anchor box size and scale, predicting the center coordinates, width, height, object confidence, and class probability of the bounding box for each grid cell of the feature map. The prediction process uses the sigmoid function to normalize the bounding box coordinates to the range of 0-1, and the softmax function to calculate the class probability distribution. The non-maximum suppression algorithm first filters low-confidence predicted boxes based on a confidence threshold (default 0.25), then calculates the IoU (Intersection over Union) between predicted boxes, merging or suppressing predicted boxes with an IoU greater than the threshold (default 0.45), retaining the predicted box with the highest confidence. The non-maximum suppression process uses a soft NMS algorithm, attenuating the confidence based on the IoU value rather than binary suppression, improving the robustness of detection.

[0025] Step 24: Based on the recognition result, combined with the point cloud data and the IMU data, perform multimodal verification on the detection result of the privacy object, and use the verified recognition result as the final privacy object recognition result; Specifically, the multi-modal verification first projects the point cloud data generated by the millimeter wave radar to the image plane, and spatially aligns with the image detection result. Then the spatial consistency of the point cloud target and the image target is calculated, including the position overlap, size ratio and motion consistency. The IMU data is used to analyze the motion state and posture of the user, and to predict the gaze direction and interaction intention of the user. The multi-modal verification process adopts a Bayesian probability model, which comprehensively considers the detection confidence and consistency of each modality, and calculates the fusion confidence. For the detection result with fusion confidence lower than the threshold, the historical detection trajectory and context information are used for correction or filtering, to improve the accuracy and stability of the detection.

[0026] Further, the execution of the multi-modal feature fusion algorithm to generate the fusion feature data includes the following steps: Step 31: Construct a multi-modal feature fusion network based on a Transformer architecture, which includes an encoder and a decoder; Specifically, the multi-modal feature fusion network adopts a variant structure of Vision Transformer, which is suitable for the characteristics of multi-modal data. The encoder contains a multi-head self-attention mechanism and a feedforward neural network, which is used to extract the internal association of each modality data. The decoder contains a multi-head cross-attention mechanism and a feedforward neural network, which is used to establish the association between different modalities. The input of the Transformer architecture includes visual feature sequences, radar feature sequences, IMU feature sequences and environmental feature sequences, each of which is mapped to the same feature dimension (default 512) through a linear projection layer. The depth of the Transformer architecture is 6 layers, each layer contains 8 attention heads, the hidden layer dimension is 2048, and the GELU activation function and LayerNorm normalization layer are used.

[0027] Step 32: Feature encoding is performed on the privacy object recognition result to generate a visual feature sequence; Specifically, the feature encoding first converts the bounding box, class and confidence information in the privacy object recognition result into a feature vector. The bounding box information includes center coordinates, width and height, and area, which are mapped to the range of 0-1 through normalization processing. The class information is represented by one-hot encoding, and the confidence information is directly used as a feature dimension. Then the feature vector is fused with the intermediate layer feature map of the improved YOLOv7 model to extract the visual features of the privacy object. The visual features are encoded with spatial position to increase the position information, and are mapped to the input dimension of the Transformer through a linear projection layer. The length of the visual feature sequence is equal to the number of detected privacy objects, and each sequence element corresponds to a feature representation of a privacy object.

[0028] Step 33: Feature extraction is performed on the human position and motion state collected by the millimeter wave radar to generate a radar feature sequence; Specifically, the feature extraction first voxelizes the point cloud data generated by the millimeter wave radar, divides the three-dimensional space into a uniform grid, and calculates the statistical features of the points in each grid, including the number of points, the average reflection intensity, and the velocity vector. Then the spatial features of the point cloud are extracted through a three-dimensional convolutional neural network, which includes 4 convolutional blocks, each containing two 3D convolutional layers, a batch normalization layer, and a ReLU activation layer. The output feature map of the three-dimensional convolutional neural network is converted into a feature vector through global average pooling and a fully connected layer. The length of the radar feature sequence is equal to the number of detected targets, and each sequence element corresponds to a radar feature representation of a target.

[0029] Step 34: Feature extraction is performed on the IMU sensor and environmental parameter data to generate an IMU feature sequence and an environmental feature sequence, respectively; Specifically, the IMU feature extraction uses a long short-term memory network (LSTM) to process time series data, capturing the time dependence of user motion. The LSTM network contains two layers of bidirectional LSTM units with a hidden state dimension of 256, and the input is the original data sequence of the IMU sensor, including acceleration, angular velocity, and magnetic field intensity. The output of the LSTM network is weighted and summed through an attention layer to generate a fixed-length feature vector. The environmental feature extraction uses a multi-layer perceptron (MLP) to process environmental parameter data, which includes three fully connected layers with hidden layer dimensions of 128, 64, and 32, respectively, and a ReLU activation function. The environmental feature sequence is sampled through a time window sliding window with a window length of 5 seconds and a step size of 1 second, capturing the short-term trend of environmental parameter changes.

[0030] Step 35: Introduce an attention mechanism for the Transformer architecture to handle occlusion scenarios and enhance the recognition ability of partially visible private objects; Specifically, the attention mechanism includes two branches: spatial attention and channel attention. The spatial attention branch generates a spatial attention map through convolutional operations, highlighting the salient regions in the image and suppressing background interference. The channel attention branch generates channel descriptors through global average pooling and global maximum pooling, learns the correlation between channels through a fully connected layer, and generates channel attention weights. The spatial attention and channel attention are applied to the feature map through element-level multiplication to enhance the expression of key features. For occlusion scenarios, the attention mechanism combines time series information and multi-modal information, maintains tracking of occluded objects through a memory mechanism, and supplements spatial information for the occluded part through radar point cloud data, achieving robust recognition of partially visible private objects.

[0031] Step 36: input the visual feature sequence, the radar feature sequence, the IMU feature sequence, and the environment feature sequence into the attention mechanism added Transformer architecture, perform multi-modal feature fusion, and generate fusion feature data; Specifically, the multi-modal feature fusion first maps each modal feature to the same feature space with a dimension of 512 through a modal-specific linear projection layer. Then, modal type embedding and position embedding are added to distinguish different modalities and sequence positions. The Transformer encoder processes the feature association within each modality through self-attention mechanism, calculates the attention weight matrix, and aggregates the feature information with weighting. The Transformer decoder processes the feature association between different modalities through cross-attention mechanism, establishing the correspondence of multi-modal features. The multi-modal feature fusion process adopts residual connection and layer normalization to ensure stable gradient propagation and effective feature fusion. The final output fusion feature contains complementary information of each modality, forming a comprehensive understanding of the scene.

[0032] Further, the performing multi-level privacy protection processing to generate the preliminary protection processing result includes the following steps: Step 41: constructing a privacy protection area mapping according to the privacy object position information in the fusion feature data to determine the image area that needs to be protected; Specifically, the privacy protection area mapping is constructed based on the privacy object bounding box coordinates and category information in the fusion feature data. First, the privacy object bounding box coordinates are mapped from the feature space back to the original image space, and an affine transformation matrix is used for coordinate conversion. For each privacy object, the system sets different protection area expansion coefficients according to its category: the face area expansion coefficient is 1.2, the electronic device screen expansion coefficient is 1.1, the file area expansion coefficient is 1.15, the personal item expansion coefficient is 1.05, and the privacy part expansion coefficient is 1.3. The expanded area is generated as a closed polygon through a convex hull algorithm as the final protection area. The system uses a region growing algorithm to handle the overlap problem of adjacent protection areas, merging the overlapping areas into a single protection area to avoid discontinuous protection effect. The privacy protection area mapping is stored in the form of a binary mask, where the pixels with a value of 1 represent the areas that need to be protected, and the pixels with a value of 0 represent the areas that do not need to be protected.

[0033] Step 42: applying a pixel-level mosaic algorithm to the privacy object to achieve basic privacy protection; Specifically, the pixel-level mosaic algorithm adopts an adaptive block size design, dynamically adjusting the size of the mosaic block according to the category and size of the privacy object. For the face area, the mosaic block size is set to 1 / 20 of the diagonal length of the bounding box; for the electronic device screen, the mosaic block size is set to 1 / 15 of the diagonal length of the bounding box; for the file area, the mosaic block size is set to 1 / 12 of the diagonal length of the bounding box; for personal items, the mosaic block size is set to 1 / 10 of the diagonal length of the bounding box; for the privacy parts, the mosaic block size is set to 1 / 25 of the diagonal length of the bounding box. The mosaic processing process first divides the original image into a grid according to the mosaic block size, then calculates the average RGB value of all pixels in each grid, and finally replaces all pixels in the grid with the average value. To improve processing efficiency, the system uses parallel computing technology to accelerate the mosaic processing process using GPU, with a processing speed of up to 120 frames per second (based on NVIDIA RTX 3080 graphics card). The mosaic algorithm also includes edge smoothing processing, which processes the edges of the mosaic area using a Gaussian blur algorithm to reduce the visual sense of discontinuity with the non-mosaic area.

[0034] Step 43: Apply a region blurring algorithm to the privacy object to enhance the protection effect; Specifically, the region blurring algorithm adopts an adaptive Gaussian blur technique, dynamically adjusting the blur radius and intensity according to the importance level of the privacy object and the user distance. The blur radius calculation formula is R = base_radius × importance_factor × distance_factor, where base_radius is the base blur radius (default value is 15 pixels), importance_factor is the importance factor (range is 0.8-1.5), and distance_factor is the distance factor (range is 0.6-1.2). The Gaussian blur uses a two-dimensional Gaussian kernel for convolution operation, with a Gaussian kernel size of 2×R+1 and a standard deviation σ of R / 3. To achieve a gradual blur effect, the system applies a blur intensity gradient at the edge of the privacy area, gradually increasing the blur effect from the edge to the center. The region blurring algorithm also includes frequency domain filtering enhancement, which converts the image to the frequency domain through Fast Fourier Transform (FFT), applies a low-pass filter to suppress high-frequency details, and then converts the image back to the spatial domain through Inverse Fast Fourier Transform (IFFT), further enhancing the blur effect and computational efficiency.

[0035] Step 44: Apply a scene semantic replacement algorithm to the privacy object to achieve visually friendly privacy protection; Specifically, the scene semantic replacement algorithm is implemented based on a generative adversarial network (GAN), which contains an encoder-decoder architecture and a discriminator. The encoder uses ResNet-50 as the backbone network to extract the semantic features of the privacy area; the decoder uses deconvolution and up-sampling layers to generate replacement content according to the semantic features; the discriminator uses a PatchGAN structure to evaluate the authenticity of the generated content. The system has pre-trained five types of replacement models, corresponding to faces, electronic device screens, files, personal items, and private parts, and each model is trained using more than 100,000 images of the corresponding category. The replacement process first identifies the category and pose of the privacy object, then selects the most matching replacement template from the corresponding replacement model library, adjusts the position, size, and angle of the template through affine transformation, and finally seamlessly integrates the replacement content into the original image through the Poisson blending algorithm. For the face area, the system generates a cartoonized avatar; for the electronic device screen, the system generates a blurred color pattern; for the file area, the system generates a random text texture; for personal items, the system generates a similar general item; for private parts, the system generates appropriate clothing coverage effects.

[0036] Step 45: Apply a gradual transition algorithm to the edge area of the privacy object to improve visual naturalness; Specifically, the gradual transition algorithm uses alpha blending technology to create a 15-pixel-wide transition zone at the edge of the privacy protection area. The alpha value in the transition zone changes linearly from the edge to the inside, with an alpha value of 0 at the edge (displaying the original image completely) and an alpha value of 1 on the inside (displaying the processed image completely). The system calculates the blending value of each pixel in the transition zone through bilinear interpolation to ensure smooth and natural visual transition. For multiple adjacent privacy areas, the system uses the Voronoi diagram algorithm to calculate the dividing line between the areas and applies symmetric gradual transitions on both sides of the dividing line to avoid discontinuity in the protection effect. The gradual transition algorithm also includes a texture preservation mechanism that evaluates the texture difference between the original image and the processed image using the structural similarity (SSIM) index, preserving the texture details of the image as much as possible while ensuring privacy protection, thereby improving the visual quality of the processed image.

[0037] Step 46: Fuse the processing results of the pixel-level mosaic algorithm, the area blurring algorithm, the scene semantic replacement algorithm, and the gradual transition algorithm to generate a preliminary protection processing result; Specifically, the fusion process adopts a weighted fusion strategy, which sets the weights of different algorithms according to the categories, importance and user preferences of privacy objects. The system defines a fusion weight matrix W, which has a dimension of 5x4, corresponding to the combination of 5 categories of privacy objects and 4 protection algorithms. For the face area, the weight of the mosaic algorithm is 0.2, the weight of the blurring algorithm is 0.3, the weight of the semantic replacement algorithm is 0.4, and the weight of the gradual transition algorithm is 0.1; for the electronic device screen, the corresponding weights are 0.3, 0.4, 0.2 and 0.1; for the file area, the corresponding weights are 0.25, 0.35, 0.3 and 0.1; for personal items, the corresponding weights are 0.2, 0.3, 0.4 and 0.1; for privacy parts, the corresponding weights are 0.15, 0.25, 0.5 and 0.1. The fusion process calculates the final value of each pixel by weighted average, and the fusion formula is P_final = Σ(Wi × Pi), where Wi is the weight of the i-th algorithm and Pi is the pixel value after the i-th algorithm processing. The system uses parallel computing technology to speed up the fusion process, and the processing delay is controlled within 10ms, ensuring real-time performance. The preliminary protection processing result retains the global structure of the original image and the detail information of the non-private area, while effectively protecting the privacy area, with natural and coordinated visual effects.

[0038] Further, the adjusting the protection intensity parameter in the preliminary protection processing result to generate an adaptive protection result comprises the following steps: Step 51: Extract the user distance parameter from the fusion feature data, and adjust the protection intensity based on the spatial relationship between the user and the privacy object; Specifically, the user distance parameter is calculated by the fusion data of the millimeter wave radar and the high-resolution camera. First, identify the user's position in the scene, then calculate the Euclidean distance between the user and each privacy object. The distance calculation considers three-dimensional space coordinates, including horizontal distance and vertical height difference. The system defines a distance threshold interval, which maps the distance between the user and the privacy object to the protection intensity coefficient. The closer the distance, the lower the protection intensity coefficient; the farther the distance, the higher the protection intensity coefficient. The specific mapping relationship is: when the distance is less than 1 meter, the protection intensity coefficient is 0.5; when the distance is between 1 meter and 3 meters, the protection intensity coefficient increases linearly from 0.5 to 1.0; when the distance is greater than 3 meters, the protection intensity coefficient is 1.0. The protection intensity coefficient directly affects the size of the mosaic block, the blur radius and the detail retention degree of the replacement content.

[0039] Step 52: Extract the scene importance parameter from the fusion feature data, and adjust the protection strategy based on the semantic importance of the privacy object; Specifically, the scene importance parameter is derived by semantic understanding of scene content through a deep learning model. A pre-trained scene understanding network is used to identify scene types (such as meetings, leisure, work, etc.) and activity types (such as talking, reading, operating equipment, etc.). According to the scene and activity types, the system assigns importance weights to different categories of privacy objects. For example, in a meeting scene, the importance weight of documents and screen content is high; in a leisure scene, the importance weight of faces and personal items is high. The importance weight ranges from 0 to 1, and the higher the weight, the stronger the privacy protection demand. The system dynamically selects the protection method according to the importance weight. For privacy objects with high importance weight, the scene semantic replacement algorithm is preferred; for privacy objects with medium importance weight, the pixel-level mosaic algorithm is selected; for privacy objects with low importance weight, the region blurring algorithm or no processing is performed.

[0040] Step 53: Extracting time dimension parameters from the fusion feature data, adjusting the protection dynamics based on the time duration of privacy objects; Specifically, the time dimension parameter is derived by tracking the time sequence of privacy objects. The system uses a multi-target tracking algorithm to assign a unique identifier to each detected privacy object and establish a link between consecutive frames. The multi-target tracking algorithm is based on Kalman filtering and the Hungarian algorithm, which realizes the prediction and matching of privacy object position, speed and appearance features. The system records the duration of appearance, frequency of appearance and state changes (such as stationary, moving, occlusion, etc.) of each privacy object. According to the time dimension parameter, the system adjusts the dynamics of the protection strategy: for privacy objects that appear briefly, a fixed strength protection process is used; for privacy objects that exist for a long time, a periodically varying protection strength is used to avoid visual fatigue of the protection effect; for privacy objects that frequently change state, an adaptive protection strategy is used to adjust the protection method and strength in real time according to the state changes.

[0041] Step 54: Calculate the comprehensive protection strength coefficient according to the user distance parameter, the scene importance parameter and the time dimension parameter; Specifically, the comprehensive protection intensity coefficient is calculated by weighted fusion of the user distance parameter, the scene importance parameter and the time dimension parameter. The system adopts an adaptive weight distribution mechanism to dynamically adjust the weights of each parameter according to the scene type and user preference. The default weight distribution is: user distance parameter weight 0.4, scene importance parameter weight 0.4, and time dimension parameter weight 0.2. The calculation formula of the comprehensive protection intensity coefficient is: comprehensive coefficient = distance coefficient x distance weight + importance coefficient x importance weight + time coefficient x time weight. The range of the comprehensive protection intensity coefficient is 0 to 1, and the higher the coefficient, the higher the protection intensity. The system maps the comprehensive protection intensity coefficient to specific protection parameters, including mosaic block size (8x8 to 32x32 pixels), blur radius (3 to 15 pixels), and detail level of replacement content (low, medium, and high levels).

[0042] Step 55: According to the comprehensive protection intensity coefficient, adjust the protection parameters in the preliminary protection processing result to generate an adaptive protection result. Specifically, the protection parameter adjustment process adopts a piecewise linear mapping function to map the comprehensive protection intensity coefficient to the specific parameters of each protection method. For the pixel-level mosaic algorithm, mosaic block size = basic size + (max size - basic size) x intensity coefficient, basic size is 8x8 pixels, and max size is 32x32 pixels. For the regional blurring algorithm, blur radius = basic radius + (max radius - basic radius) x intensity coefficient, basic radius is 3 pixels, and max radius is 15 pixels. For the scene semantic replacement algorithm, detail retention degree = max detail x (1 - intensity coefficient), and max detail corresponds to the structure and texture features of the original scene. The protection parameter adjustment process also includes a time smoothing mechanism to avoid visual jumps caused by sudden changes in parameters, and the parameter changes are smoothed by an exponential moving average algorithm with a time window of 10 frames.

[0043] The receiving user input parameters and modifying the adaptive protection result according to the user input parameters includes the following steps: Step 61: Build a multi-modal interaction layer to collect and process user input signals. Specifically, the multi-modal interaction layer includes a gesture recognition unit, a voice instruction unit, and a touch input unit. The gesture recognition unit collects user gesture data through the high-resolution camera and the IMU sensor, applies a double-flow neural network to extract spatial features and timing features, and recognizes predefined gesture commands. The voice instruction unit collects user speech through the sound sensor in the environmental sensor, applies an acoustic model and a language model to convert the voice signal into a text command. The touch input unit receives operation instructions input by the user through a touch device. The multi-modal interaction layer uniformly converts the signals of each input unit into a standard command format, including command type, parameter value, and priority label, and delivers it to the command processing module of the system.

[0044] Step 62: modifying the adaptive protection result according to the user input parameters collected by the multi-modal interaction layer; Specifically, the system first parses the user input parameters to extract privacy region coordinates and protection policy parameters. The privacy region coordinates define the image region that needs to be protected, which can be a rectangle, an ellipse, or a polygon shape; the protection policy parameters specify the protection method (mosaic, blur, or replacement) and the protection intensity level. The system merges the user-defined privacy region with the automatically detected privacy object region, and generates a comprehensive protection region using a region union operation. The system covers the automatically calculated parameters of the corresponding region according to the protection policy parameters, and applies the user-specified protection method and intensity level. For regions without specified parameters, the system keeps the original adaptive protection parameters unchanged. The system applies the modified protection parameters to the original image through an image processing pipeline to generate the final protection result, and outputs the processed video stream through a display device.

[0045] In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive repetition.

[0046] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0047] The above embodiments are only used to illustrate the technical methods of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical methods of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical methods of the present application.

Claims

1. A smart home privacy protection monitoring system based on multi-modal sensor fusion, characterized in that, The multi-modal perception module, the privacy object recognition module, the multi-modal feature fusion module, the privacy protection processing module, the dynamic protection adjustment module, and the multi-modal interaction module are connected through electrical connection. The multi-modal perception module is composed of a high-resolution camera, a millimeter wave radar, an IMU sensor, and an environmental sensor, which is used to collect scene data and send the scene data to the privacy object recognition module and the multi-modal feature fusion module. The privacy object recognition module inputs image data in the scene data into an improved YOLOv7 model, executes a privacy object recognition algorithm, and combines point cloud data and IMU data in the scene data to obtain a privacy object recognition result, which is sent to the multi-modal feature fusion module. The multi-modal feature fusion module inputs the scene data and the privacy object recognition result into a Transformer architecture, executes a multi-modal feature fusion algorithm, which includes an attention mechanism processing module, generates fusion feature data, and sends the fusion feature data to the privacy protection processing module and the dynamic protection adjustment module. The privacy protection processing module executes a multi-level privacy protection process according to privacy object position information in the fusion feature data, which includes applying a pixel-level mosaic algorithm, a region blurring algorithm, and a scene semantic replacement algorithm to the privacy object, generates a preliminary protection processing result, and sends the preliminary protection processing result to the dynamic protection adjustment module. The dynamic protection adjustment module extracts user distance parameters, scene importance parameters, and time dimension parameters from the fusion feature data, adjusts protection intensity parameters in the preliminary protection processing result according to the user distance parameters, the scene importance parameters, and the time dimension parameters, generates an adaptive protection result, and sends the adaptive protection result to the multi-modal interaction module. The multi-modal interaction module is composed of a gesture recognition module, a voice instruction module, and a mobile APP module, receives the adaptive protection result sent by the dynamic protection adjustment module, receives user input parameters including privacy region coordinates and protection strategy parameters, modifies the adaptive protection result according to the user input parameters, and displays the modified adaptive protection result. 2.The smart home privacy protection monitoring system based on multi-modal sensor fusion of claim 1, wherein, The collection of scene data includes the following steps: Step 11: Configure a high-resolution camera to collect image data, which is installed on the top of the indoor space and the wall surface. Step 12: Deploy a millimeter wave radar to collect point cloud data to sense human position and motion state. Step 13: Integrate an IMU sensor to collect IMU data to monitor user motion posture, which includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. Step 14: Arrange an environmental sensor to collect indoor environmental parameters, which includes a temperature sensor, a humidity sensor, a light sensor, and a sound sensor. Step 15: Time synchronization and spatial registration are performed on the image data collected by the high-resolution camera, the point cloud data sensed by the millimeter wave radar, the IMU data sensed by the IMU sensor, and the indoor environment parameters collected by the environment sensor to form the scene data. 3.The smart home privacy protection monitoring system based on multi-modal sensor fusion of claim 2, characterized in that, The execution of the privacy object recognition algorithm includes the following steps: Step 21: A privacy object detection network is constructed based on the improved YOLOv7 model, and the privacy objects include faces, electronic device screens, files, personal items, and private parts. Step 22: The image data in the scene data is input into the improved YOLOv7 model to perform a forward inference process and generate a feature map. Step 23: Anchor box prediction and non-maximum suppression algorithms are applied to the feature map to obtain the recognition result of the privacy object, which includes the bounding box, class, and confidence of the privacy object. Step 24: The recognition result is combined with the point cloud data and the IMU data to perform multi-modal verification on the detection result of the privacy object, and the verified recognition result is taken as the final privacy object recognition result. 4.The smart home privacy protection monitoring system based on multi-modal sensor fusion of claim 3, characterized in that, The execution of the multi-modal feature fusion algorithm to generate the fusion feature data includes the following steps: Step 31: A multi-modal feature fusion network based on a Transformer architecture is constructed, which includes an encoder and a decoder. Step 32: Feature encoding is performed on the privacy object recognition result to generate a visual feature sequence. Step 33: Feature extraction is performed on the human body position and motion state collected by the millimeter wave radar to generate a radar feature sequence. Step 34: Feature extraction is performed on the IMU sensor and the environment parameter data to generate an IMU feature sequence and an environment feature sequence, respectively. Step 35: An attention mechanism is introduced into the Transformer architecture to handle occlusion scenarios and enhance the recognition ability of partially visible privacy objects. Step 36: The visual feature sequence, the radar feature sequence, the IMU feature sequence, and the environment feature sequence are input into the Transformer architecture with the added attention mechanism to perform multi-modal feature fusion and generate fusion feature data. 5.The smart home privacy protection monitoring system based on multi-modal sensor fusion of claim 4, characterized in that, The feature encoding first converts the bounding box, class, and confidence information in the privacy object recognition result into a feature vector. The bounding box information includes center coordinates, width and height, and area, which are mapped to the 0-1 range through normalization processing. The class information is represented by one-hot encoding, and the confidence information is directly used as a feature dimension. Then, the feature vector is fused with the intermediate layer feature map of the improved YOLOv7 model to extract the visual features of the privacy object. The visual features are encoded with spatial position to increase the position information and mapped to the input dimension of the Transformer through a linear projection layer. The length of the visual feature sequence is equal to the number of detected privacy objects, and each sequence element corresponds to a feature representation of a privacy object. 6.The smart home privacy protection monitoring system based on multi-modal sensor fusion according to claim 5, characterized in that, The human body position and motion state collected by the millimeter wave radar are subjected to feature extraction. Firstly, voxelization processing is performed on the point cloud data generated by the millimeter wave radar, and a three-dimensional space is divided into uniform grids. The statistical features of the points in each grid are calculated, including the number of points, the average reflection intensity and the velocity vector. Then, the spatial features of the point cloud are extracted through a three-dimensional convolutional neural network. The three-dimensional convolutional neural network includes four convolutional blocks, each of which includes two 3D convolutional layers, a batch normalization layer and a ReLU activation layer. The output feature map of the three-dimensional convolutional neural network is converted into a feature vector through global average pooling and a fully connected layer. The length of the radar feature sequence is equal to the number of detected targets, and each sequence element corresponds to a radar feature representation of a target. 7.The smart home privacy protection monitoring system based on multi-modal sensor fusion of claim 6, characterized in that, The IMU feature extraction adopts an LSTM network to process time series data to capture the time dependence of user motion. The LSTM network includes two layers of bidirectional LSTM units with a hidden state dimension of 256. The input is the original data sequence of the IMU sensor, including acceleration, angular velocity and magnetic field intensity. The output of the LSTM network is weighted and summed through an attention layer to generate a fixed-length feature vector. The environment feature extraction adopts a multi-layer perceptron to process environment parameter data. The MLP includes three fully connected layers with hidden layer dimensions of 128, 64 and 32, respectively, and the activation function is ReLU. The environment feature sequence is sampled through a time window sliding window with a window length of 5 seconds and a step length of 1 second to capture the short-term trend of environmental parameters.

8. The intelligent home privacy protection monitoring system based on multi-modal sensor fusion according to claim 7, characterized in that, The attention mechanism includes spatial attention and channel attention branches. The spatial attention branch generates a spatial attention map through convolutional operations to highlight significant areas in the image and suppress background interference. The channel attention branch generates channel descriptors through global average pooling and global maximum pooling, learns the correlation between channels through a fully connected layer, and generates channel attention weights. The spatial attention and channel attention are applied to the feature map through element-level multiplication to enhance the expression of key features. For occlusion scenarios, the attention mechanism combines time series information and multi-modal information, maintains tracking of occluded objects through a memory mechanism, and supplements spatial information for occluded parts through radar point cloud data to achieve robust recognition of partially visible private objects. 9.The smart home privacy protection monitoring system based on multi-modal sensor fusion of claim 8, wherein, The execution of multi-level privacy protection processing to generate the preliminary protection processing result includes the following steps: Step 41: According to the private object position information in the fusion feature data, a privacy protection area mapping is constructed to determine the image area that needs to be protected. Step 42: Apply a pixel-level mosaic algorithm to the private object to achieve basic privacy protection. Step 43: Apply a region blurring algorithm to the private object to enhance the protection effect. Step 44: Apply a scene semantic replacement algorithm to the private object to achieve visually friendly privacy protection. Step 45: Apply a gradual transition algorithm to the edge area of the private object to improve visual naturalness. Step 46: fuse the processing results of the pixel-level mosaic algorithm, the region blurring algorithm, the scene semantic replacement algorithm and the gradient transition algorithm to generate a preliminary protection processing result. 10.The smart home privacy protection monitoring system based on multi-modal sensor fusion of claim 9, wherein, The step of adjusting the protection strength parameter in the preliminary protection processing result to generate an adaptive protection result comprises the following steps: Step 51: extract a user distance parameter from the fused feature data, and adjust the protection strength based on the spatial relationship between the user and the privacy object; Step 52: extract a scene importance parameter from the fused feature data, and adjust the protection strategy based on the semantic importance of the privacy object; Step 53: extract a time dimension parameter from the fused feature data, and adjust the protection dynamics based on the time duration of the privacy object; Step 54: calculate a comprehensive protection strength coefficient according to the user distance parameter, the scene importance parameter and the time dimension parameter; Step 55: adjust the protection parameter in the preliminary protection processing result according to the comprehensive protection strength coefficient to generate an adaptive protection result.

Images