Civil going-out supervision equipment based on infrared camera and deep learning

Abstract

The invention provides a caregiver outgoing supervision device based on an infrared camera and deep learning, and the device combines the infrared camera with a deep learning model by means of an embedded system, and can collect and analyze an infrared image in real time so as to judge whether a caregiver is indoor or outdoor. The working process is as follows: after a caregiver signs in a card, equipment is automatically started, an infrared camera collects environment images according to a fixed interval of one time every 10 seconds, an embedded system carries out preprocessing such as denoising and normalization on the images, then features are extracted and classified by using a convolutional neural network (CNN) model, and finally, the information of the caregiver is obtained. And finally, uploading an analysis result (indoor / outdoor and confidence), a timestamp and a device ID to a server through a wireless network. If the server receives an outdoor result or does not receive data, a video sampling inspection mechanism can be automatically triggered, an AI customer service calls a video call to check the situation, and video content is stored for subsequent inspection.

Description

Technical Field

[0001] The present invention relates to the technical field of caregiver behavior supervision, and particularly to a caregiver going-out supervision device based on an infrared camera and deep learning. Background Art

[0002] With the acceleration of the aging process in China, the demand for the supervision of the service quality of caregivers in the elderly care industry is becoming increasingly urgent. At present, the mainstream supervision methods in the industry mainly focus on the process control before and after the caregiver's service. For example, the arrival time, service duration, etc. of caregivers are recorded through GPS positioning, electronic fences, or intelligent sign-in systems. Taking the patent CN113808293A as an example, this invention realizes the accurate positioning of caregivers arriving at the elderly's homes through an intelligent sign-in system, and automatically calculates the service duration through a timestamp algorithm, effectively solving the problem of supervision at the starting point of the service. However, such technologies have significant limitations.

[0003] The existing technologies can only supervise the "starting point" (such as whether arriving at the post) and "ending point" (such as service duration) of the service process, but lack effective monitoring of the real-time behavior of caregivers during the service. For example, a caregiver may leave the post without permission after signing in, or be away from the work post for a long time during the service, and the existing system cannot detect such abnormal behaviors in time.

[0004] GPS positioning depends on satellite signals, and it is prone to weak signals, positioning deviation, or even failure in indoor environments (such as in the elderly's homes), resulting in the inability to accurately judge whether the caregiver is moving within the service location. In addition, there may be signal shielding areas in some elderly care institutions or home scenarios, further reducing the reliability of the positioning technology.

[0005] Traditional video surveillance means (such as ordinary cameras) can record images in real time, but they involve the privacy of the elderly's lives and are prone to ethical disputes. Especially in sensitive scenarios such as using the toilet and changing clothes, continuous monitoring by ordinary cameras may violate the dignity of the elderly, so it is strictly restricted in practical applications.

[0006] The existing supervision systems usually rely on managers to regularly spot-check surveillance videos or conduct telephone follow-ups. This passive supervision method is inefficient and has a lag in response. According to industry research data, the spot-check coverage rate of a certain elderly care institution is less than 15%, resulting in a large number of abnormal behaviors not being discovered in time, posing major safety hazards.

[0007] The existing systems lack the ability to perform real-time analysis on the collected positioning data, image data, etc., and cannot quickly identify the abnormal behavior patterns of caregivers. For example, in the case where a caregiver goes out briefly multiple times and accumulates overtime, it is difficult for the traditional system to achieve intelligent early warning through simple data comparison.

[0008] In recent years, the rapid development of technologies such as the Internet of Things (IoT) and artificial intelligence (AI) has provided new opportunities for elderly care supervision. For example, behavior recognition technology based on computer vision can judge the activity status of personnel by analyzing video streams. However, existing solutions mostly rely on visible light cameras, and the privacy issue remains unresolved. In addition, the application of edge computing technology can achieve local data processing and reduce network transmission pressure. However, how to deploy efficient deep learning models on low-power devices is still an industry problem. Summary of the Invention

[0009] In view of the above technical gaps, the present invention proposes a real-time supervision solution based on infrared cameras and embedded deep learning, which realizes efficient supervision while protecting privacy.

[0010] The present invention provides a caregiver out-of-site supervision device based on infrared cameras and deep learning, comprising:

[0011] A front-end device for data acquisition and preliminary processing, including:

[0012] A clock-in system for confirming the arrival of the caregiver and starting the supervision process;

[0013] An infrared camera for collecting image data of the environment where the caregiver is located; and

[0014] An embedded system for image preprocessing and deep learning model analysis;

[0015] A data transmission network for transmitting the analysis results, timestamps, and device ID data processed by the embedded system to the server through a wireless network; and

[0016] A back-end management platform for data reception, analysis, storage, and supervision decision execution, including:

[0017] A server for receiving data uploaded by the front-end device, judging and processing the data. If the data is abnormal, the server will trigger an exception handling mechanism and start a video spot-check mechanism; and

[0018] A video spot-check system configured to automatically conduct video spot-checks when the server triggers the exception handling mechanism.

[0019] In an embodiment of the present invention, the infrared camera is configured to collect infrared images of the current environment at fixed time intervals.

[0020] In an embodiment of the present invention, the embedded system uses a deep learning model based on a convolutional neural network (CNN) to extract features and classify images.

[0021] In an embodiment of the present invention, the data anomaly is multiple departures from the post within a specified time or failure to receive data uploaded by the front-end device within a certain period of time.

[0022] The present invention also provides a method for supervising the out-of-duty situation of caregivers based on an infrared camera and deep learning, which is characterized by including:

[0023] The caregiver checks in through the clock-in system on the device. After the server receives the check-in signal, it starts the monitoring program and enters the working state;

[0024] After the monitoring system is started, it begins to initialize the infrared camera and adjust the focal length and exposure parameters to ensure the quality of image acquisition;

[0025] The infrared camera collects infrared images of the current environment at fixed time intervals and preprocesses the images through an embedded system;

[0026] The preprocessed images are input into the deep learning model in the embedded system, and the deep learning model analyzes the images and outputs the analysis results;

[0027] The embedded system uploads the analysis results to the server through a wireless network;

[0028] The server receives the uploaded data and makes a judgment. If the analysis result is "indoors", it records the data and waits for the next upload. If the analysis result is "outdoors" multiple times or no data is received within a certain period of time, it triggers an anomaly handling mechanism;

[0029] The AI customer service makes a video call to the caregiver and stores the content of the video call for the management to check.

[0030] In an embodiment of the present invention, the preprocessing of the images through the embedded system includes:

[0031] Removing noise using Gaussian filtering according to the following formula:

[0032]

[0033] Normalizing the image pixel values to [0,1] according to the following formula:

[0034]

[0035] where is the original image, I min and I max are the minimum and maximum pixel values of the image respectively, G(i, j) is the Gaussian kernel function, σ is a hyperparameter that controls the width of the function, and its default value is 1.5.

[0036] In an embodiment of the present invention, the deep learning model analyzes an image and outputs an analysis result, including:

[0037] Input the processed image into the deep learning model;

[0038] The convolutional layer extracts image features:

[0039] F l =σ(W l *F l-1 +b l ),

[0040] where F l is the feature map of the l-th layer, W l is the convolutional kernel weight, b l is the bias, and σ is the activation function;

[0041] The connection layer outputs the classification result:

[0042] z = W fc ·E final +b fc

[0043] p = softmax(z),

[0044] where z is the output of the fully connected layer, and P is the classification probability distribution.

[0045] Output the final classification result:

[0046]

[0047] where p0 and p1 are the probabilities of "indoor" and "outdoor" respectively.

[0048] The present invention has the following beneficial effects:

[0049] (1) Replace the traditional visible light camera with an infrared camera, only capture thermal radiation information, avoid shooting the facial features and living details of the elderly, and eliminate the risk of privacy leakage from the physical level.

[0050] (2) The embedded system integrates a lightweight deep learning model (such as CNN) to achieve infrared image analysis and classification decision-making within 10 seconds. Compared with the traditional manual sampling inspection (the sampling coverage rate is less than 15%), the system can monitor the status of the caregiver in real time, and the abnormal response time is shortened from the hour level to the minute level, significantly improving the supervision timeliness.

[0051] (3) The server side sets a dual-trigger mechanism. Active trigger: Automatically initiate video sampling inspection when the "outdoor" status is detected continuously 3 times (configurable) within a specified time; Passive trigger: Start emergency verification when the network is interrupted for more than 5 minutes (configurable). Brief Description of the Drawings

[0052] Figure 1 FIG. shows the structural composition diagram of the caregiver going-out supervision device based on an infrared camera and deep learning in an embodiment of the present invention; and

[0053] Figure 2 FIG. shows the flowchart of the caregiver going-out supervision method based on an infrared camera and deep learning in an embodiment of the present invention. Detailed Description of the Embodiments

[0054] In the following description, the present invention is described with reference to the embodiments. However, those skilled in the art will recognize that the embodiments can be implemented without one or more specific details or in combination with other alternative and / or additional methods, materials, or components. In other cases, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the inventive points of the present invention. Similarly, for purposes of explanation, specific quantities, materials, and configurations are set forth to provide a thorough understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details.

[0055] In the present invention, each embodiment is merely intended to illustrate the solution of the present invention and should not be construed as restrictive.

[0056] In this specification, the reference to "an embodiment" or "the embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment of the present invention. The phrase "in an embodiment" appearing throughout this specification does not necessarily refer to the same embodiment.

[0057] In addition, the numbering of the steps of the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps can be executed in different orders.

[0058] The present invention will be further described below with reference to the accompanying drawings in conjunction with the specific embodiments.

[0059] Figure 1 FIG. shows the structural composition diagram of the caregiver going-out supervision device based on an infrared camera and deep learning in an embodiment of the present invention.

[0060] As Figure 1 shown, the caregiver going-out supervision device based on an infrared camera and deep learning in this embodiment includes:

[0061] The front-end device 10, which is used for data acquisition and preliminary processing, includes:

[0062] The clock-in system 11 is used to confirm the arrival of the caregiver and initiate the supervision process. The caregiver checks in through the clock-in system 11 on the device. After successful clock-in, the device starts the monitoring program. This is the starting point of the supervision process. Only after clocking in will the entire supervision device enter the working state.

[0063] The infrared camera 12 is used to collect image data of the environment where the caregiver is located. It is responsible for collecting image data of the environment where the caregiver is located. After the device starts and completes initialization (adjusting focus, exposure parameters, etc.), it collects infrared images at fixed time intervals. Since it uses infrared technology, it can avoid infringing on the privacy of the elderly like ordinary cameras, and at the same time can obtain key image information for analyzing the position of the caregiver (indoors or outdoors). In this embodiment, the fixed interval is 10s, and other interval times can be selected according to different embodiments. The present invention does not limit this here.

[0064] The embedded system 13 undertakes the tasks of image preprocessing and deep learning model analysis. It performs denoising and normalization processing on the original images collected by the infrared camera 12 to improve the image quality and standardization for the analysis of the deep learning model. Then, the preprocessed images are input into a deep learning model based on a convolutional neural network (CNN) for feature extraction and classification judgment, and the environment (indoors / outdoors) where the caregiver is located and the confidence score are output. In addition, it is also responsible for uploading the analysis results (including information such as timestamp, device ID, etc.) to the remote server through the wireless network.

[0065] The data transmission network 20 transmits data such as the analysis results, timestamp, and device ID processed by the embedded system 13 to the remote server through wireless networks such as Wi-Fi or 4G / 5G. It ensures that data can be transmitted from the front-end device to the back-end in real time and stably, providing data support for the server to make subsequent judgments and processing.

[0066] The back-end management platform 30 is used for data reception, analysis, storage, and execution of supervision decisions, including:

[0067] The server 31 receives the data uploaded by the front-end device 10 and makes judgments and processes the data. If the received result shows that the caregiver is "indoors", the data is recorded and waiting for the next upload;

[0068] If the "outdoor" state is continuously detected 3 times within the specified time or the network is interrupted for more than 5 minutes, the emergency verification is initiated. The server 31 will trigger the exception handling mechanism and start the video sampling mechanism to further verify the situation of the caregiver. In this embodiment, the triggering conditions of the exception triggering mechanism are only examples and can be adjusted according to specific situations.

[0069] The video random inspection system 32 comes into play when the server 31 triggers the exception handling mechanism. The AI customer service automatically dials the video call of the caregiver and stores the video content. Managers can check the actual working status of the caregiver when out by viewing these video records, achieving efficient supervision, reducing manual intervention and improving the supervision effect at the same time.

[0070] Figure 2 The flowchart of the caregiver out-of-site supervision method based on an infrared camera and deep learning in an embodiment of the present invention is shown.

[0071] As Figure 2 shown, the process of the caregiver out-of-site supervision method based on an infrared camera and deep learning in this embodiment is as follows:

[0072] Check-in 100. After the caregiver arrives at the service location, they check in through fingerprint, face recognition or RFID. After the device receives the signal, it starts the infrared camera and the embedded system.

[0073] Device startup 200. After the device starts, the infrared camera starts to initialize, adjusting the focal length, exposure parameters, etc. to ensure the quality of image acquisition. And set the acquisition time interval of the infrared camera.

[0074] Infrared acquisition 300. The infrared camera acquires thermal radiation images at a fixed time interval (such as every 10 seconds), covering the caregiver's activity area. And transmit the acquired images to the embedded system, which preprocesses them, including: Image denoising: removing noise interference in the image; Image normalization: adjusting the images to a unified size and format for easy model processing.

[0075] Picture analysis 400. The preprocessed images are input into the deep learning model in the embedded system. The deep learning model (based on the convolutional neural network CNN) extracts features and classifies the images, including: Feature extraction (extracting features such as edges and textures of the images through convolutional layers) and classification judgment (outputting the classification result through the fully connected layer to judge whether the current environment is "indoor" or "outdoor"). The model outputs the result (indoor / outdoor) and the confidence score.

[0076] Result output 500. The embedded system uploads the analysis result (indoor / outdoor) and the confidence score to the remote server through a wireless network (such as Wi-Fi or 4G / 5G). The uploaded content includes: Timestamp (recording the time of image acquisition and analysis), device ID (identifying the specific device) and analysis result (indoor / outdoor classification result).

[0077] For the abnormal random inspection 600, the server receives the uploaded data and makes a judgment. If the result is "indoor", the data is recorded and waiting for the next upload. If the result is "outdoor" multiple times within a short period or no data is received (such as network interruption), the abnormal handling mechanism is triggered. The AI customer service will make a video call to the caregiver and store the video call content for the management to check.

[0078] The following will explain some processes in combination with the algorithms involved in this monitoring system.

[0079] In this process, the inputs include:

[0080] The caregiver's clock-in signal S (S ∈ {0, 1}, 0 means not clocked in, 1 means clocked in);

[0081] The original image I collected by the infrared camera raw ;

[0082] The deep learning model parameters θ (including convolutional kernel weights, biases, etc.).

[0083] The outputs are:

[0084] The classification result C (C ∈ {0, 1}, 0 means outdoor, 1 means indoor);

[0085] The trigger video random inspection flag F (F ∈ {0, 1}, 0 means not triggered, 1 means triggered).

[0086] Clock-in and sign-in 100, if S = 1 (that is, the caregiver has clocked in), the device starts.

[0087] Device startup 200, initialize the infrared camera, and set the time interval as Δt (Δt = 10 seconds) for periodic image acquisition.

[0088] Infrared acquisition 300, at time point t k = t 0 + k·Δt (k = 0, 1, 2,...), acquire the original infrared image and preprocess the image. Preprocess the original image including:

[0089] Denoising: Use Gaussian filtering to remove noise:

[0090]

[0091] where G(i, j) is the Gaussian kernel function:

[0092]

[0093] where σ is the hyperparameter controlling the width of the function, and its default value is 1.5.

[0094] Normalization: Normalize the image pixel values to [0, 1]:

[0095]

[0096] where I min and I max are the minimum and maximum pixel values of the image respectively.

[0097] Image analysis 400, deep learning model analysis, input the pre - processed image into the deep learning model f θ .

[0098] Among them, the convolutional layer is used to extract image features:

[0099] F l = σ(W l *F l-1 + b l ).

[0100] where F l is the feature map of the l - th layer. W l is the convolutional kernel weight, b l is the bias. σ is the activation function, and the activation function adopted in this embodiment is the ReLU function.

[0101] Among them, the fully - connected layer is used to output the classification result:

[0102] z = W fc ·F final + b fc

[0103] p = softmax(z).

[0104] where z is the output of the fully - connected layer. P is the classification probability distribution,

[0105] The final classification result C (k) is as follows:

[0106]

[0107] where p0 and p1 are the probabilities of "indoor" and "outdoor" respectively.

[0108] Although the embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not as a limitation. It will be apparent to those skilled in the relevant art that various combinations, modifications, and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined only in accordance with the appended claims and their equivalents.

Claims

1. A caregiver out-of-office monitoring device based on infrared camera and deep learning, characterized in that: include: Front-end equipment, used for data collection and preliminary processing, including: A clock-in system is used to confirm the arrival of caregivers and start the supervision process; Infrared camera to collect image data of the caregiver's environment; and Embedded systems for image preprocessing and deep learning model analysis; A data transmission network, which transmits the analysis results, timestamp, and device ID data processed by the embedded system to the server via a wireless network; and Backend management platform for data reception, analysis, storage and regulatory decision execution, including: The server is used to receive data uploaded by the front-end device, and judge and process the data. If the data is abnormal, the server will trigger the abnormality handling mechanism and start the video sampling mechanism; and The video sampling system is configured to automatically perform video sampling when the server triggers the exception handling mechanism.

2. The infrared camera and deep learning caregiver outgoing monitoring device according to claim 1 is characterized in that: The infrared camera is configured to collect infrared images of the current environment at fixed time intervals.

3. The infrared camera and deep learning caregiver outgoing monitoring device according to claim 1 is characterized in that: The embedded system uses a deep learning model based on a convolutional neural network (CNN) to extract features and classify images.

4. The infrared camera and deep learning caregiver outgoing monitoring device according to claim 1 is characterized in that: The data anomaly refers to multiple absences from work within a specified period of time or failure to receive data uploaded by the front-end device within a certain period of time.

5. A method for monitoring caregivers' outings based on infrared cameras and deep learning, characterized in that: include: The caregiver signs in through the clock-in system on the device. After receiving the sign-in signal, the server starts the monitoring program and enters the working state; After the monitoring system is started, the infrared camera is initialized and the focus and exposure parameters are adjusted to ensure the quality of image acquisition; The infrared camera collects infrared images of the current environment at fixed time intervals and pre-processes the images through the embedded system; The preprocessed image is input into the deep learning model in the embedded system, and the deep learning model analyzes the image and outputs the analysis results; The embedded system uploads the analysis results to the server via the wireless network; The server receives the uploaded data and makes a judgment. If the analysis result is "indoor", the data is recorded and waits for the next upload. If the analysis result is "outdoor" for many times or no data is received within a certain period of time, the exception handling mechanism is triggered; The AI ​​customer service makes a video call to the caregiver and stores the content of the video call for management personnel to review.

6. The method for monitoring caregivers' outings based on infrared cameras and deep learning according to claim 5 is characterized in that: The preprocessing of the image by the embedded system comprises: Use Gaussian filtering to remove noise according to the following formula: The image pixel values ​​are normalized to [0,1] according to the following formula: in, is the original image, I min and I max are the minimum and maximum pixel values ​​of the image, respectively, G(i, j) is the Gaussian kernel function, σ is a hyperparameter that controls the width of the function, and its default value is 1.

5.

7. The method for monitoring caregivers' outings based on infrared cameras and deep learning according to claim 5 is characterized in that: The deep learning model analyzes the image and outputs the analysis results, including: Feed the processed image into the deep learning model; The convolutional layer extracts image features: F l =σ(W l *F l-1 +b l ), Among them, F l is the feature map of the lth layer, W l is the convolution kernel weight, b l is the bias, σ is the activation function; The connection layer outputs the classification result: z=W fc ·F final +b fc P = softmax(z), Among them, z is the output of the fully connected layer, P is the classification probability distribution, Output the final classification results: Among them, p0 and p1 are the probabilities of "indoor" and "outdoor" respectively.

Images