A visual privacy protection method for intelligent monitoring based on three-axis accelerometer

Abstract

The present invention discloses a smart monitoring visual privacy protection method based on a three-axis accelerometer, which integrates a smart monitoring camera and a smart wearable device. The three-axis accelerometer data is collected from the smart wearable device to build a fall detection model, and video data is collected from the smart monitoring camera at the same time. When the smart wearable device detects that a dangerous behavior such as a fall occurs, CycleGAN is first used to perform style transfer on the currently collected video frame to achieve non-realistic processing of the full frame, protect the privacy information appearing in the video frame, and then send the processed video frame to the server for fall detection, thereby improving the accuracy of fall detection and enhancing the privacy protection of the monitoring video. When the detection model does not detect that a dangerous behavior such as a fall occurs, the current video frame is discarded to protect the user's normal behavior activities from being leaked. The present invention effectively enhances the user's privacy protection, improves the fall detection rate, and also reduces bandwidth consumption.

Description

Technical Field

[0001] The present invention belongs to the field of privacy protection in smart elderly care, and in particular relates to an intelligent monitoring visual privacy protection method based on a three-axis accelerometer. Background Art

[0002] With the advent of the "big data" era, the concept of "smart city" has been proposed, and "smart elderly care" has also emerged. The "Guiding Opinions on Actively Promoting the "Internet +" Action" clearly stated that it will further promote the development of the smart health and elderly care industry, based on advanced technologies and products, to achieve the effective integration of different subjects such as individuals, families, communities and institutions. Ensure that health and elderly care services are more "smart" and achieve an all-round upgrade in efficiency and reliability. At present, China uses home-based elderly care as the main form, assisted by communities and institutions, to set up a diversified elderly care system. The development of the "smart elderly care" model is conducive to remote monitoring, real-time positioning, and unified platform information interaction, meeting the modernization, scientific and humanization needs of modern family elderly care, and is also an important aspect of the development of smart cities.

[0003] However, there are also certain hidden dangers behind the application of technology. When various data are uploaded to the public smart elderly care service platform, the privacy of the elderly and their families cannot be effectively protected. To a certain extent, the current difficulties in the development of smart elderly care are closely related to the use of information technology and the leakage of elderly information. In the process of using smart elderly care technology and collecting information about the elderly, the most important issue that has attracted much attention from the elderly and ensured the smooth development of smart elderly care is the protection of the privacy rights of the elderly. The healthy development of smart home-based elderly care urgently needs a series of effective measures to protect personal privacy and provide a good environment for the privacy of the elderly and their families.

[0004] According to a survey, among the numerous smart elderly care products, the development of smart cameras has a large audience group. Because it breaks through the boundaries of time and space, it brings great convenience to the elderly and their families and is deeply loved by everyone. The installation of smart home cameras can monitor the living conditions of the elderly in real time, and can help them in the first time when the elderly encounter emergencies. From this level, the installation of smart home cameras provides more convenient services for the elderly living alone. However, as it develops rapidly and brings many conveniences to the elderly, the privacy issues of smart cameras also follow.

[0005] In the prior art, traditional video privacy protection is generally achieved by hiding the visual effects of the target, such as using encryption, blurring, pixelation, replacement, occlusion and other related technologies to filter out facial information in the video. The privacy protection level of blurring and pixelation methods depends largely on the size of the blur radius. Too small values ​​provide too little protection, and too large values ​​affect usability. Although encryption, occlusion and other methods protect the privacy of personal identity, they cannot meet the subsequent artificial intelligence detection tasks and will reduce the usability of the video. In recent years, the vigorous development of deep learning has provided new ideas for video privacy protection. The powerful style transfer ability of the generative adversarial network is used to perform full-frame non-realistic processing to achieve video privacy protection. However, since most of the current privacy protection processing is processed 24 hours a day, it not only increases the consumption of network bandwidth, but also leaks the normal life behavior of users. Summary of the invention

[0006] The present invention provides an intelligent monitoring visual privacy protection method based on a three-axis accelerometer to solve the problems of insufficient privacy protection and high transmission energy consumption in the existing intelligent monitoring privacy protection methods. By using three-axis accelerometer data, abnormal behaviors such as user falls are identified in advance, thereby realizing intelligent monitoring privacy protection.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] An intelligent monitoring visual privacy protection method based on a three-axis accelerometer comprises the following steps:

[0009] S1. Preprocessing the raw data of the three-axis accelerometer collected by the smart wearable device;

[0010] S2. Analyze the processed raw data and extract corresponding features;

[0011] S3. Using the change in the acceleration intensity value to set a threshold-based fall detection module for the smart wearable device;

[0012] S4. Use visual privacy enhancement CycleGAN to set up a visual privacy enhancement module for intelligent monitoring to hide full-frame privacy information;

[0013] S5. Send the fall video processed by the visual privacy enhancement module to the remote server for subsequent intelligent recognition tasks. If no abnormal behavior such as fall occurs, it will not be sent.

[0014] Furthermore, the step S1 is specifically as follows:

[0015] S11. According to the raw signal data (ACC_X, ACC_Y and ACC_Z) collected by the three-axis accelerometer built into the smart wearable device, the data offset is calibrated. First, the coordinate system is determined. Since the theoretical values ​​of ACC_X and ACC_Y should be 0, the offsets of these two readings are calibrated by statistical mean. ACC_Z requires one more step, that is, in the process of statistical offset, the theoretical value must be added to each reading, and then the statistical mean calibration is performed;

[0016] S12. Use Kalman filter to remove noise from the original data.

[0017] Furthermore, the step S2 is specifically as follows:

[0018] S21. Use sliding window to extract features. The sliding window consists of two key variables: sliding window size and sliding step size. The sliding window size refers to the amount of data processed at one time. The sliding window size selected by the present invention is an exponential power of 2. Among them, f is the sampling frequency of the sensor, which ensures the smooth implementation of Fourier transform when extracting frequency domain features;

[0019] S22. Use t to represent the size of a time window, and use signal amplitude area SMA to represent the time domain characteristics. Signal amplitude area refers to the sum of the areas enclosed by the acceleration reading curves of the x, y, and z axes and the three coordinate axes. Generally speaking, the characteristics are more obvious in static and moving states. The calculation formula is:

[0020]

[0021] Where: x(t), y(t) and z(t) are the acceleration values ​​of the three axes in time window t.

[0022] S23. For the characteristics of the acceleration value of a single axis, the average acceleration value of the single axis is used to represent it. The calculation formula of the average acceleration value of the y-axis at sampling time n is:

[0023]

[0024] Where: y[i] is the acceleration value of the y-axis at sampling time i, and N is the number of samples.

[0025] Furthermore, the step S3 is specifically as follows:

[0026] S31. Set threshold T1 according to the strength of the signal amplitude area (SMA) value to perform coarse-grained fall event detection on human activity. Mainly consider static actions (including standing, sitting and lying) and motion actions (including walking, going up and down stairs and falling, etc.). When the user performs motion actions, the SMA value is much higher than the SMA value of static actions;

[0027] S32. A threshold T2 is set according to the intensity of the average acceleration value (y_mean) of the y-axis to perform fine-grained fall event detection, which distinguishes a fall action from actions similar to a fall action (such as lying down and standing up quickly) to improve the accuracy of fall detection.

[0028] Furthermore, the step S4 is specifically as follows:

[0029] S41. The edge camera needs to protect the privacy of the frame content before sending the collected video frames containing the fall behavior to the remote server based on the fall signal of the threshold-based fall detection module. The visual privacy enhanced CycleGAN is used to perform unrealistic processing on the video frames where the fall occurs. The training dataset first applies a bilateral filter (an edge-preserving smoothing filter) to reduce the number of colors. Then, a simple image segmentation is performed to quickly locate the contours and edges of possible objects. Finally, the segmented parts are recolored with the average value of the original color of their corresponding areas to achieve unrealistic processing of the entire frame and achieve visual privacy protection;

[0030] S42. During training, a key is assigned to each smart camera. The key is a two-dimensional matrix with the same height and width as the original video frame. Each variable in the matrix is ​​a random number between 0 and 255. During the training phase, the device's key is attached to the training data of the original 3-channel RGB image as a fourth channel watermark. Through visual privacy enhancement, CycleGAN restores the processed video frame to the original video frame, which is only available to authorized users.

[0031] Furthermore, the step S5 is specifically as follows:

[0032] S51. When the fall detection system detects an abnormal behavior such as a fall, the smart camera sends the processed video frames of the fall time period to the remote server for subsequent intelligent detection to improve the accuracy of fall detection and provide timely assistance to the user;

[0033] S52. When the fall detection system does not detect abnormal behavior such as a fall, the smart camera will not send any video frames to the remote server to protect the user's daily life from being leaked and improve user privacy.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention integrates the intelligent monitoring camera and the intelligent wearable device. The three-axis accelerometer data is collected from the intelligent wearable device to build a fall detection model. When a dangerous behavior such as a fall is detected, video data is collected from the intelligent monitoring camera at the same time. The collected video frames are style-transferred using a generative adversarial network to achieve non-realistic processing of the full frame, protect the privacy information appearing in the video frame, and then the processed video frames are sent to the server for fall detection, thereby improving the accuracy of fall detection and enhancing the privacy protection of the monitoring video. When the detection model does not detect the occurrence of dangerous behaviors such as falls, the video frames are not sent to the server to protect the normal behavior of the user from being leaked. The present invention distinguishes between normal behavior and fall behavior in the daily life of the user, does not send the normal behavior of the user to the server, and protects the privacy information of the user's daily behavior. After the fall behavior is processed by non-reality, it is sent to the server for subsequent fall detection, verifies the fall behavior and provides help to the user in a timely manner, effectively enhancing the user's privacy protection, improving the fall detection rate, and also reducing bandwidth consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the present invention.

[0037] Figure 2 The flowchart of fall detection based on three-axis accelerometer.

[0038] Figure 3 Flowchart of privacy protection for intelligent monitoring based on CycleGAN. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the embodiments.

[0040] like Figure 1 As shown, a smart monitoring visual privacy protection method based on a three-axis accelerometer includes the following steps:

[0041] S1. Preprocessing the raw data of the three-axis accelerometer collected by the smart wearable device;

[0042] S11. According to the raw signal data (ACC_X, ACC_Y and ACC_Z) collected by the three-axis accelerometer built into the smart wearable device, the data offset is calibrated. First, the coordinate system is determined. Since the theoretical values ​​of ACC_X and ACC_Y should be 0, the offsets of these two readings are calibrated by statistical mean. ACC_Z requires one more step, that is, in the process of statistical offset, the theoretical value must be added to each reading, and then the statistical mean calibration is performed;

[0043] S12. Use Kalman filter to remove noise from the original data.

[0044] S2. Analyze the processed raw data and extract corresponding features;

[0045] S21. Use sliding window to extract features. The sliding window consists of two key variables: sliding window size and sliding step size. The sliding window size refers to the amount of data processed at one time. The sliding window size selected by the present invention is an exponential power of 2. Among them, f is the sampling frequency of the sensor, which ensures the smooth implementation of Fourier transform when extracting frequency domain features;

[0046] S22. Use t to represent the size of a time window, and use signal amplitude area SMA to represent the time domain characteristics. Signal amplitude area refers to the sum of the areas enclosed by the acceleration reading curves of the x, y, and z axes and the three coordinate axes. Generally speaking, the characteristics are more obvious in static and moving states. The calculation formula is:

[0047]

[0048] Where x(t), y(t) and z(t) are the acceleration values ​​of the three axes in time window t.

[0049] S23. For the characteristics of the acceleration value of a single axis, the average acceleration value of the single axis is used to represent it. The calculation formula of the average acceleration value of the y-axis at sampling time n is:

[0050]

[0051] Where y[i] is the acceleration value of the y-axis at sampling time i, and N is the number of samples.

[0052] S3. Figure 2 As shown, the change of acceleration intensity value is used to set a threshold-based fall detection module for the smart wearable device;

[0053] S31. Set threshold T1 according to the strength of the signal amplitude area (SMA) value to perform coarse-grained fall event detection on human activity. Mainly consider static actions (including standing, sitting and lying) and motion actions (including walking, going up and down stairs and falling, etc.). When the user performs motion actions, the SMA value is much higher than the SMA value of static actions;

[0054] S32. A threshold T2 is set according to the intensity of the average acceleration value (y_mean) of the y-axis to perform fine-grained fall event detection, which distinguishes a fall action from actions similar to a fall action (such as lying down and standing up quickly) to improve the accuracy of fall detection.

[0055] S4. Figure 3 As shown in the figure, visual privacy enhancement CycleGAN is used to set up a visual privacy enhancement module for intelligent monitoring to hide the privacy information of the whole frame;

[0056] S41. The edge camera needs to protect the privacy of the frame content before sending the collected video frames containing the fall behavior to the remote server based on the fall signal of the threshold-based fall detection module. The visual privacy enhanced CycleGAN is used to perform unrealistic processing on the video frames where the fall occurs. The training dataset first applies a bilateral filter (an edge-preserving smoothing filter) to reduce the number of colors. Then, a simple image segmentation is performed to quickly locate the contours and edges of possible objects. Finally, the segmented parts are recolored with the average value of the original color of their corresponding areas to achieve unrealistic processing of the entire frame and achieve visual privacy protection;

[0057] S42. During training, a key is assigned to each smart camera. The key is a two-dimensional matrix with the same height and width as the original video frame. Each variable in the matrix is ​​a random number between 0 and 255. During the training phase, the device's key is attached to the training data of the original 3-channel RGB image as a fourth channel watermark. Through visual privacy enhancement, CycleGAN restores the processed video frame to the original video frame, which is only available to authorized users.

[0058] S5. Send the processed fall video to the remote server for subsequent intelligent recognition tasks. If no abnormal behavior such as fall occurs, it will not be sent.

[0059] S51. When the fall detection system detects an abnormal behavior such as a fall, the smart camera sends the processed video frames of the fall time period to the remote server for subsequent intelligent detection to improve the accuracy of fall detection and provide timely assistance to the user;

[0060] S52. When the fall detection system does not detect abnormal behavior such as a fall, the smart camera will not send any video frames to the remote server to protect the user's daily life from being leaked and improve user privacy.

[0061] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for protecting visual privacy in intelligent monitoring based on a three-axis accelerometer, characterized in that: The following steps are involved: S1. Preprocessing the raw data of the three-axis accelerometer collected by the smart wearable device; S2. Analyze the processed raw data and extract corresponding features; S3. Using the change in the acceleration intensity value to set a threshold-based fall detection module for the smart wearable device; S4. Use visual privacy enhancement CycleGAN to set up a visual privacy enhancement module for intelligent monitoring to hide full-frame privacy information; S5. Send the fall video processed by the visual privacy enhancement module to the remote server for subsequent intelligent recognition tasks. If no abnormal behavior such as fall occurs, it will not be sent; The step S1 is specifically as follows: S11. According to the original signal data collected by the three-axis accelerometer built into the smart wearable device, the original signal data includes ACC_X, ACC_Y and ACC_Z, and the data offset is calibrated. First, the coordinate system is determined. Since the theoretical values ​​of ACC_X and ACC_Y should be 0, the ACC_X and ACC_Y reading offsets are calibrated by using the statistical mean method. In the process of statistical offset of ACC_Z, the theoretical value is added to each reading, and then the statistical mean calibration is performed; S12. Use Kalman filter to remove noise from the original data; The step S2 is specifically as follows: S21. Use sliding window to extract features. The sliding window consists of two key variables: sliding window size windowssize and sliding step step. The sliding window size refers to the amount of data processed at one time. The size of the sliding window is selected as an exponential power of 2. Among them, f is the sampling frequency of the sensor, which ensures the smooth execution of Fourier transform when extracting frequency domain features; S22. Use t to represent the size of a time window, and use signal amplitude area SMA to represent the time domain characteristics. Signal amplitude area SMA refers to the sum of the areas enclosed by the acceleration reading curves of the x, y, and z axes and the three coordinate axes. The characteristics are obvious in the static state and the moving state. The calculation formula is: Among them: x(t), y(t) and z(t) are the acceleration values ​​of the x, y and z axes in the time window t respectively; S23. For the characteristics of the acceleration value of a single axis, the average acceleration value of a single axis is used to represent the average acceleration value y of the y axis at sampling time n. mean The calculation formula for [n] is: Where: y[i] is the acceleration value of the y-axis at sampling time i, and N is the number of samples; The step S3 is specifically as follows: S31. According to the strength of the signal amplitude area SMA value, a threshold T1 is set to perform coarse-grained fall event detection on human activity actions, static actions and motion actions. When the user performs motion actions, the SMA value is much higher than the SMA value of static actions; S32. Setting a threshold value T2 according to the intensity of the average acceleration value y_mean of the y-axis to perform fine-grained fall event detection, distinguishing a fall action from actions similar to a fall action, so as to improve the accuracy of fall detection; The step S4 is specifically as follows: S41. The edge camera needs to protect the privacy of the frame content before sending the collected video frames containing the fall behavior to the remote server based on the fall signal of the threshold-based fall detection module; the visual privacy enhancement CycleGAN is used to perform unrealistic processing on the video frames where the fall occurs; the training dataset first applies a bilateral filter to reduce the number of colors, and then performs simple image segmentation to locate the contours and edges of possible objects; finally, the segmented parts are recolored with the average value of the original color of the corresponding area to achieve unrealistic processing of the whole frame and achieve visual privacy protection; S42. During training, a key is assigned to each smart camera. The key is a two-dimensional matrix with the same height and width as the original video frame, and each variable in the matrix is ​​a random number between 0 and 255. During the training phase, the key of the device is attached to the training data of the original 3-channel RGB image as a fourth channel watermark. The processed video frame is restored to the original video frame through visual privacy enhancement CycleGAN, which is only available to authorized users. The step S5 is specifically as follows: S51. When the fall detection system detects an abnormal behavior such as a fall, the smart camera sends the processed video frames of the fall time period to the remote server for subsequent intelligent detection to improve the accuracy of fall detection and provide timely assistance to the user; S52. When the fall detection system does not detect abnormal behavior such as a fall, the smart camera will not send any video frames to the remote server to protect the user's daily life from being leaked and improve user privacy.

Images