Intelligent monitoring system in a privacy environment

Abstract

The application discloses a kind of intelligent monitoring systems under privacy environment, it is characterized in that, including information acquisition module, data preprocessing module, intelligent analysis module: information acquisition module gathers environmental information, and forms stream data based on environmental information;Data preprocessing module is used to form structured data with physical meaning with stream data;Intelligent analysis module obtains human posture estimation result and human behavior recognition result.The application uses touchless sensor as the collection medium of environmental information, and avoids the privacy invasion problem caused by monitoring in home environment;The application uses ToF sensor and infrared sensor to collect environmental information, which can complement the defects of ToF sensor that is not sensitive enough to human perception and infrared sensor cannot obtain the detailed position of objects in three-dimensional space.

Description

TECHNICAL FIELD

[0001] The application relates to an intelligent monitoring system in a privacy environment, and belongs to the technical field of old-age care monitoring and smart home. BACKGROUND

[0002] Population aging has become one of the most severe challenges that China will face in the future. The increasing population dependency ratio makes it difficult for young adults to take care of the elderly, so the situation of the elderly living alone will continue to exist and grow. In this case, real-time monitoring of the elderly living alone and effective monitoring of commercial care services are becoming increasingly important. Traditional monitoring methods infringe on the personal privacy of users, and people are more sensitive to the use of such technical means in application fields that attach great importance to privacy, such as bedrooms and bathrooms.

[0003] Non-contact sensors are increasingly used in the Internet of Things, and artificial intelligence technology can further analyze the data collected by sensors and give abstract signals practical meaning to realize non-intrusive environmental intelligence, which plays an important role in relieving clinical service pressure and improving the quality and effectiveness of medical services. The research in this direction is not deep enough. The application patent with the publication number CN 110428585A proposes an elderly living alone monitoring system based on an infrared sensor, which uses infrared sensors installed at each door in the room to determine the behavior pattern of the elderly living alone. This system can only provide limited monitoring functions and cannot provide effective judgment on the specific behavior pattern in the room. The utility model patent with the authorization announcement number CN 209000182U proposes a human body monitoring system based on wearable devices and infrared sensors. This system needs to compare a large amount of prior information and the behavior of the monitored person to perform behavior monitoring and abnormal alarm, which lacks intelligence and universality. SUMMARY

[0004] The purpose of the present application is to provide an intelligent monitoring system based on ToF sensors and infrared sensors in a privacy environment, which can intelligently monitor the monitored person in a home environment and focus on avoiding the user's privacy content while monitoring to improve the intelligent home system.

[0005] In order to achieve the above purpose, the technical scheme of the present application provides an intelligent monitoring system in a privacy environment, characterized in that it comprises an information collection module, a data preprocessing module and an intelligent analysis module, wherein:

[0006] The information collection module collects environmental information through ToF sensors and infrared sensors, and forms stream data based on the environmental information, and then sends the stream data outward. The information collection module collects and generates depth images through ToF sensors without contact, and the information collection module collects and generates infrared images through infrared sensors without contact.

[0007] The data preprocessing module is configured to form structured data with physical meaning from the stream data obtained by the information collection module.

[0008] The intelligent analysis module is configured to process the structured data output by the data preprocessing module through a pre-trained model encapsulated therein, to obtain a human posture estimation result and a human behavior recognition result, and to transmit the human posture estimation result and the human behavior recognition result to a terminal to realize intelligent real-time monitoring.

[0009] Preferably, the information collection module generates the depth image by the following steps:

[0010] Step 101, the laser emission module in the ToF sensor emits infrared laser of a specific wavelength;

[0011] Step 102, the ToF sensor receives the infrared laser signal reflected by the environment, and filters the environmental noise according to the wavelength;

[0012] Step 103, the information collection module calculates the straight-line distance and the included angle of the reflection point to the sensor according to the time difference between the emission of the laser and the reception of the reflected signal;

[0013] Step 104, the information collection module calculates the horizontal distance of the reflection point to the sensor according to the straight-line distance and the included angle of the reflection point to the sensor;

[0014] Step 105, the information collection module generates the depth image by taking the horizontal distance as the numerical value of each pixel point in the depth image.

[0015] Preferably, the information collection module generates the infrared image by the following steps:

[0016] Step 201, the infrared sensor directly receives infrared light information in the environment;

[0017] Step 202, the information collection module calculates the received infrared light intensity by a signal processing algorithm;

[0018] Step 203, the information collection module generates the infrared image by taking the infrared light intensity as the numerical value of each pixel point in the infrared image.

[0019] Preferably, the data preprocessing module processes the stream data by the following steps:

[0020] Step 301, decode the stream data, and restore the depth image and the infrared image of each frame according to the time stamp;

[0021] Step 302, map the pixel values of the depth image and the infrared image to the range of 0-255, normalize them into a gray image, set all the pixel points with a distance greater than a certain threshold in the depth image as infinite, and set all the pixel points with an infrared light intensity less than a certain threshold in the infrared image as 0 light intensity;

[0022] Step 303, use adaptive histogram equalization to enhance the contrast of the normalized depth image and the infrared image, and improve the recognition degree between the human body close to the background object;

[0023] Step 304, filter the noise points in the depth image and the infrared image;

[0024] Step 305, sharpen the depth image and the infrared image, and further improve the recognition degree between different objects.

[0025] Preferably, the intelligent analysis module comprises a pose estimation submodule and a human behavior recognition submodule, wherein:

[0026] The pose estimation submodule is used to locate the spatial position of the human body in the environment, restore the human body pose, and realize human body reconstruction in the form of a skeleton graph in a three-dimensional environment;

[0027] The human behavior recognition submodule is used to classify and identify the current behavior mode of the human body.

[0028] Preferably, the implementation of the pose estimation submodule comprises the following steps:

[0029] Step 401, use a multi-head convolution structure to extract feature maps from the depth image and the infrared image respectively, superimpose all the feature maps, and perform data fusion through a convolution layer integrated with a self-attention mechanism;

[0030] Step 402, extract one-dimensional features from the fused high-dimensional feature map through a pre-trained deep convolutional neural network;

[0031] Step 403, use a multi-layer perception to regress a preset number of joint coordinates;

[0032] Step 404, construct a human skeleton graph in the form of a graph structure by taking the joints as nodes and the joint connection relationship conforming to the human body as edges.

[0033] Preferably, the implementation of the human behavior recognition submodule comprises the following steps:

[0034] Step 501, calculate the intersection over union of the current frame skeleton graph and the historical frame skeleton graph according to the bounding box of the human skeleton graph, if the intersection over union is greater than a set threshold, assign the current skeleton graph with the corresponding person identity in the historical frame, otherwise, the current skeleton does not exist in the historical frame, and assign the current skeleton graph with a new person identity, thereby realizing human body tracking;

[0035] Step 502, extracting features for each skeleton graph through a pre-trained graph convolutional neural network, and caching into a corresponding feature queue according to the results of human body tracking;

[0036] Step 503, selecting the features extracted from the current frame and the historical frame to form a time sequence feature group for each human body identity, sequentially inputting into a recurrent neural network, and outputting the behavior category of the current target person through a fully connected layer.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] 1) The present application uses a non-contact sensor as an acquisition medium for environmental information, avoiding the privacy invasion problem caused by monitoring in a home environment;

[0039] 2) The present application uses ToF sensors and infrared sensors to collect environmental information, which can complement the defects of the ToF sensor that is not sensitive enough to human body perception and the infrared sensor that cannot obtain the detailed position of the object in three-dimensional space;

[0040] 3) The present application selects a multi-head convolutional neural network and a graph convolutional neural network for modeling analysis according to the data characteristics of depth images, infrared images and human skeleton graphs, ensuring that the extracted features are representative. The recurrent neural network is used to combine context information to ensure the robustness and continuity of behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a system overall flowchart;

[0042] Figure 2 is an information acquisition module flowchart;

[0043] Figure 3 is a data preprocessing module flowchart;

[0044] Figure 4 is a pose estimation module flowchart;

[0045] Figure 5 is a behavior recognition module flowchart;

[0046] Figure 6 is a method schematic diagram of pose estimation and behavior recognition. DETAILED DESCRIPTION

[0047] The present application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the present application after reading the content taught by the present application, and these equivalent forms also fall within the scope defined by the appended claims.

[0048] The intelligent monitoring system in a private environment disclosed in the present application takes a smart box integrated with a ToF sensor and an infrared sensor as a hardware basis, and comprises an information collection module, a data preprocessing module, and an intelligent analysis module.

[0049] Figure 1 The overall flow of the system of the present application is shown, and the module composition logic and data flow are as follows: environmental information is collected by the information collection module, the obtained stream data is formed into structured data with physical meaning by the data processing module, and then the pre-trained model encapsulated in the intelligent analysis module is used for processing to realize human posture estimation and behavior recognition, and the human posture estimation result and the behavior recognition result are transmitted to the terminal to realize intelligent real-time monitoring.

[0050] The information collection module takes a smart box as a hardware basis, and the box is internally integrated with a ToF sensor and an infrared sensor. The sensors used all have the characteristic of non-contact, and the human information and environmental information collected are all ensured to be non-invasive. Compared with the information collected by the existing optical camera monitoring system, the information is more rich in dimension and provides more perfect protection for the privacy of users. The ToF sensor and the infrared sensor are organically combined in a complementary manner, and have stronger adaptability for multi-person scenes or complex scenes. Figure 2 The flow of the information collection module is shown, including the ToF sensor collection flow and the infrared sensor collection flow. The depth image and the infrared image obtained through sensor collection are converted into stream data through data encoding encryption and uploaded to the cloud server.

[0051] The ToF sensor collection flow comprises the following steps:

[0052] Step 1: The laser emission module in the ToF sensor emits infrared laser of a specific wavelength;

[0053] Step 2: The ToF sensor receives the infrared laser signal reflected by the environment, and filters the environmental noise according to the wavelength;

[0054] Step 3: The straight-line distance and the included angle of the reflection point to the sensor are calculated according to the time difference between the emitted laser and the received reflection signal;

[0055] Step 4: The horizontal distance of the reflection point to the sensor is calculated according to the straight-line distance and the included angle of the reflection point to the sensor;

[0056] Step 5: The horizontal distance is taken as the numerical value of each pixel point in the depth image to generate a depth image.

[0057] The infrared sensor collection flow comprises the following steps:

[0058] Step 1: The infrared sensor directly receives the infrared light information in the environment;

[0059] Second step, calculate the received infrared light intensity by signal processing algorithm integrated inside the smart box;

[0060] Third step, generate infrared image with infrared light intensity as the value of each pixel point in the infrared image.

[0061] The data preprocessing module takes the stream data uploaded to the cloud server by the information collection module as input, and outputs the preprocessed depth image and infrared image. The flow chart is as shown in Figure 3 , which includes the following steps:

[0062] First step, decode the stream data, restore the depth image and infrared image of each frame according to the timestamp;

[0063] Second step, map the pixel values of the depth image and the infrared image to the range of 0-255, normalize them to standard 8-bit grayscale images, set all pixel points with a distance greater than a certain threshold in the depth image to infinity, and set all pixel points with an infrared light intensity less than a certain threshold in the infrared image to 0 light intensity;

[0064] Third step, use adaptive histogram equalization to enhance the contrast of the normalized depth image and infrared image, and improve the recognition degree between the human body close to the background object;

[0065] Fourth step, use open-close operation of traditional graphics algorithm to filter noise points in the depth image and infrared image;

[0066] Fifth step, use sharpening algorithm to sharpen the depth image and infrared image, further improve the recognition degree between different objects. In this embodiment, the sharpening algorithm uses high-pass filtering or Laplace operator.

[0067] The intelligent analysis module takes the depth image and infrared image output by the data preprocessing module as input, and outputs the human skeleton graph and human behavior category. It is composed of multiple sub-function modules deployed in the cloud, including a pose estimation submodule and a human behavior recognition submodule. Among them:

[0068] The role of the pose estimation submodule is to locate the spatial position of the human body in the environment, restore the human pose, and realize human reconstruction in the form of skeleton graph in the three-dimensional environment. The module takes the preprocessed depth image and infrared image as input, and outputs the human skeleton graph. The module flow is as shown in Figure 4 , which includes the following steps:

[0069] First step, use a multi-head convolution structure to extract feature maps from the depth image and the infrared image respectively, stack all the feature maps and perform data fusion through a convolution layer integrated with a self-attention mechanism;

[0070] Second step, the high-dimensional feature map after fusion is extracted by a pre-trained deep convolutional neural network to obtain one-dimensional features;

[0071] Third step, the joint coordinates are regressed by using a multi-layer perception network;

[0072] Fourth step, the human skeleton graph is constructed by taking the joints as nodes and the joint connection relationship as edges.

[0073] The human behavior recognition sub-module is used to classify and recognize the current behavior mode of the human body. The input of the module is the human skeleton graph output by the pose estimation module, and the output is the human behavior category. The module process is shown in Figure 5 The following steps are included:

[0074] First step, according to the bounding box of the human skeleton graph, the intersection over union (IOU) of the current frame skeleton graph and the historical frame skeleton graph is calculated. If the intersection over union is greater than a set threshold, the current skeleton graph is assigned the corresponding person identity in the historical frame. Otherwise, the current skeleton does not exist in the historical frame, and the current skeleton graph is assigned a new person identity. In this way, human tracking is achieved.

[0075] Second step, the features of each skeleton graph are extracted by a pre-trained graph convolutional neural network, and the corresponding feature queue is cached according to the human tracking result;

[0076] Third step, for each human identity, the features extracted from the current frame and the historical frame are selected to form a time sequence feature group, which is sequentially input into a recurrent neural network (RNN), and the behavior category of the current target person is output through a fully connected layer.

[0077] The human skeleton graph and human behavior category output by the intelligent monitoring system are transmitted to the terminal display in real time, which is convenient for real-time viewing of the state and abnormal situation alarm of the monitored person.

Claims

1. A smart monitoring system under privacy environment, characterized in that, The system comprises an information collection module, a data preprocessing module and an intelligent analysis module, wherein: The information collection module collects environmental information through a ToF sensor and an infrared sensor, forms stream data based on the environmental information, and sends the stream data out, wherein the information collection module collects and generates a depth image through the ToF sensor without contact, and collects and generates an infrared image through the infrared sensor without contact; The data preprocessing module is used to form structured data with physical meaning based on the stream data obtained by the information collection module; The intelligent analysis module processes the structured data output by the data preprocessing module through a pre-trained model encapsulated therein, obtains human posture estimation results and human behavior recognition results, and transmits the human posture estimation results and human behavior recognition results to a terminal to realize intelligent real-time monitoring; The data preprocessing module processes the stream data by the following steps: Step 301: decoding the stream data, and restoring the depth image and the infrared image of each frame according to the time stamp; Step 302: mapping the pixel values of the depth image and the infrared image to the range of 0-255, normalizing them into grayscale images, setting all the pixel points with a distance greater than a certain threshold in the depth image as infinite, and setting all the pixel points with an infrared light intensity less than a certain threshold in the infrared image as 0 light intensity; Step 303: using adaptive histogram equalization to enhance the contrast of the normalized depth image and the infrared image, and improving the recognition degree between the human body close to the background object; Step 304: filtering the noise points in the depth image and the infrared image; Step 305: performing sharpening processing on the depth image and the infrared image, and further improving the recognition degree between different objects; The intelligent analysis module comprises a posture estimation submodule and a human behavior recognition submodule, wherein: The posture estimation submodule is used to locate the spatial position of the human body in the environment, restore the human posture, and realize human reconstruction in the form of a skeleton graph in a three-dimensional environment; The human behavior recognition submodule is used to classify and identify the current behavior pattern of the human body; The implementation of the posture estimation submodule comprises the following steps: Step 401: using a multi-head convolution structure to extract feature maps from the depth image and the infrared image respectively, stacking all the feature maps and performing data fusion through a convolution layer integrated with a self-attention mechanism; Step 402: extracting one-dimensional features from the fused high-dimensional feature maps through a pre-trained deep convolutional neural network; Step 403: using a multi-layer perception machine to regress a preset number of joint node coordinates; Step 404: taking the joint nodes as nodes and the joint node connection relationship conforming to the human body as edges, constructing a human skeleton graph in the form of a graph structure; The implementation of the human behavior recognition submodule comprises the following steps: Step 501: calculating the intersection-over-union ratio of the current frame skeleton graph and the historical frame skeleton graph according to the bounding box of the human skeleton graph, if the intersection-over-union ratio is greater than a set threshold, assigning the current skeleton graph with the identity of the corresponding person in the historical frame, otherwise, the current skeleton does not exist in the historical frame, and assigning the current skeleton graph with a new person identity, thereby realizing human tracking. Step 502, extracting features of each skeleton graph by a pre-trained graph convolutional neural network, and caching into a corresponding feature queue according to the result of human body tracking; Step 503, selecting the features extracted from the current frame and the historical frame to form a time sequence feature group for each human body identity, sequentially inputting into a recurrent neural network, and outputting the behavior category of the current target human body through a fully connected layer.

2. The intelligent monitoring system under privacy environment as claimed in claim 1, wherein, The information acquisition module generates the depth image by the following steps: Step 101, the laser emission module in the ToF sensor emits infrared laser of a specific wavelength; Step 102, the ToF sensor receives the infrared laser signal reflected by the environment, and filters the environmental noise according to the wavelength; Step 103, the information acquisition module calculates the straight-line distance and the included angle of the reflection point to the sensor according to the time difference between the emission of the laser and the reception of the reflected signal; Step 104, the information acquisition module calculates the horizontal distance from the reflection point to the sensor according to the straight-line distance and the included angle of the reflection point to the sensor; Step 105, the information acquisition module generates a depth image by taking the horizontal distance as the numerical value of each pixel point in the depth image.

3. The intelligent monitoring system under privacy environment as claimed in claim 1, wherein, The information acquisition module generates the infrared image by the following steps: Step 201, the infrared sensor directly receives infrared light information in the environment; Step 202, the information acquisition module calculates the received infrared light intensity by a signal processing algorithm; Step 203, the information acquisition module generates an infrared image by taking the infrared light intensity as the numerical value of each pixel point in the infrared image.

Images