Human body behavior recognition method and device, equipment and storage medium

Abstract

The invention relates to the technical field of electric data processing, and discloses a human body behavior recognition method, device and equipment and a storage medium, and the method comprises the steps: obtaining a human body heat radiation distribution diagram collected by an infrared sensor for a target object; performing thermal gradient calculation on the human body thermal radiation distribution diagram to generate a direction field diagram; acquiring a thermal gradient direction histogram based on the direction field pattern; and using a human body behavior recognition model to predict the thermal gradient direction histogram to obtain the behavior classification of the target object. According to the method, the problem of low human behavior recognition accuracy caused by insufficient low-resolution thermal imaging data information can be solved, the image quality is improved without super-resolution reconstruction preprocessing, high-complexity calculation is avoided, the human behavior recognition efficiency is high, and the real-time requirement of human behavior recognition is met.

Description

Technical Field

[0001] This invention relates to the field of electrical data processing technology, specifically to a method, apparatus, device, and storage medium for human behavior recognition. Background Technology

[0002] Human behavior classification technology has broad application prospects in fields such as intelligent monitoring, medical care, and human-computer interaction. While traditional behavior recognition methods based on visible light cameras have made significant progress, their applicability is limited in scenarios with insufficient lighting or privacy concerns. Thermal imaging technology, which captures infrared radiation emitted by the human body, offers advantages such as being unaffected by visible light and protecting personal privacy, thus becoming an important alternative in these scenarios.

[0003] However, thermal imaging-based human behavior classification faces a key challenge: due to the hardware cost and technological constraints of thermal imaging sensors, the thermal imaging data acquired in practical applications is often of low resolution. Low-resolution thermal imaging data leads to a significant loss of information on human contours and movement details, making it difficult for traditional behavior classification algorithms (such as convolutional neural network-based methods) to extract sufficient discriminative features, resulting in a significant decrease in classification accuracy. Existing techniques typically improve image quality through super-resolution reconstruction preprocessing, but this method has high computational complexity, poor real-time performance, and may introduce artifacts during reconstruction, further interfering with the performance of the classification model.

[0004] Furthermore, low-resolution thermal imaging data contains a relatively high proportion of noise and background interference, while the proportion of effective behavioral features decreases, leading to increased model sensitivity to noise. Although some studies have attempted to alleviate this problem by improving network structure or adding data augmentation, an effective balance between model lightweighting and classification accuracy has not yet been achieved. Therefore, there is an urgent need for a technical solution that can robustly extract behavioral features directly from low-resolution thermal imaging data and achieve high-precision classification to meet the combined requirements of real-time performance and accuracy in practical applications. Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus, device and storage medium for human behavior recognition, in order to solve the problem that human behavior recognition cannot simultaneously achieve both accuracy and efficiency.

[0006] In a first aspect, the present invention provides a method for human behavior recognition, the method comprising:

[0007] Obtain the human body thermal radiation distribution map collected by the infrared sensor for the target object;

[0008] Thermal gradient calculations are performed on the aforementioned human body thermal radiation distribution map to generate a directional field map;

[0009] Based on the aforementioned orientation field map, a thermal gradient orientation histogram is obtained;

[0010] Using a human behavior recognition model, the thermal gradient direction histogram is predicted to obtain the behavior classification of the target object.

[0011] In one optional implementation, predicting the thermal gradient orientation histogram using a human behavior recognition model includes:

[0012] Obtain the kinetic energy map obtained by superimposing the orientation field maps from multiple consecutive frames;

[0013] The human behavior recognition model is used to predict the thermal gradient direction histogram and the kinetic energy map.

[0014] In one optional implementation, the human behavior recognition model includes a YOLO model, the backbone network of which is a cross-stage Darknet, and the cross-stage Darknet embeds an SE module, which is used to enhance the feature response of key points of the human torso.

[0015] In one optional implementation, the human behavior recognition model includes a backbone network and a spatiotemporal fusion module. The spatiotemporal fusion module is used to aggregate features from the thermal gradient orientation histogram and the kinetic energy map of multiple consecutive frames, and output the behavior classification prediction result.

[0016] In one alternative implementation, the behavior classification includes at least one of the following: falls, smoking, abnormal movements during intravenous infusion, epileptic seizures, getting out of bed after surgery, abnormal positioning of critically ill patients, and theft of newborns;

[0017] And / or,

[0018] The human behavior recognition model is also used to output risk levels.

[0019] In one optional implementation, before performing thermal gradient calculation on the human body thermal radiation distribution map to generate the direction field map, the method further includes:

[0020] Acquire human body point cloud data collected by millimeter-wave radar targeting the target object; fuse the human body point cloud data with the human body thermal radiation distribution map collected simultaneously using Kalman filtering to obtain the fused human body thermal radiation distribution map; and / or,

[0021] An adaptive median filter is used to dynamically adjust the window size based on the local temperature variance, thereby eliminating interference from environmental heat sources.

[0022] In one optional implementation, the step of fusing the human body point cloud data with the simultaneously acquired human body thermal radiation distribution map using Kalman filtering includes:

[0023] Obtain sensor weights determined based on the ambient noise level, the sensor weights including the weights of the infrared sensor and the millimeter-wave radar;

[0024] According to the sensor weights, the human body point cloud data and the human body thermal radiation distribution map collected at the same time are fused by Kalman filtering; wherein, the weight of the infrared sensor is used as the weight of the human body thermal radiation distribution map, and the weight of the millimeter-wave radar is used as the weight of the human body point cloud data.

[0025] In a second aspect, the present invention provides a human behavior recognition device, the device comprising:

[0026] The data acquisition module is used to acquire the human body thermal radiation distribution map collected by the infrared sensor for the target object;

[0027] The orientation field map generation module is used to calculate the thermal gradient of the human body thermal radiation distribution map and generate an orientation field map.

[0028] A thermal gradient direction histogram acquisition module is used to acquire a thermal gradient direction histogram based on the direction field map.

[0029] The prediction module is used to predict the thermal gradient direction histogram using a human behavior recognition model to obtain the behavior classification of the target object.

[0030] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the human behavior recognition method of the first aspect or any corresponding embodiment described above.

[0031] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the human behavior recognition method of the first aspect or any corresponding embodiment described above.

[0032] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the human behavior recognition method of the first aspect or any corresponding embodiment described above.

[0033] The human behavior recognition method, apparatus, device, and storage medium provided in this invention solve the problem of low accuracy in human behavior recognition caused by insufficient information in low-resolution thermal imaging data by quantifying the spatiotemporal features of actions through thermal gradient orientation histogram (HOG-T). Moreover, it eliminates the need for super-resolution reconstruction preprocessing to improve image quality, avoids highly complex calculations, and achieves high efficiency in human behavior recognition, meeting the real-time requirements of human behavior recognition. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a human behavior recognition method according to an embodiment of the present invention;

[0036] Figure 2 This is a structural block diagram of a human behavior recognition device according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] According to an embodiment of the present invention, a method for human behavior recognition is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of executable computer instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] This embodiment provides a human behavior recognition method that can be used in various computer devices, which can be used as edge nodes. Figure 1 This is a flowchart of a human behavior recognition method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0041] Step S101: Obtain the human body thermal radiation distribution map (also known as thermal imaging map) collected by the infrared sensor for the target object.

[0042] Specifically, an infrared sensor, also known as an infrared thermal imaging sensor, has a resolution of 160×120. Infrared sensors can capture the distribution of human body thermal radiation (temperature range -20℃ to 150℃).

[0043] In some optional implementations, while using an infrared sensor to acquire a human body thermal radiation distribution map of the target object, a millimeter-wave radar can be used to acquire human body point cloud data of the target object. Furthermore, in step S102, before performing thermal gradient calculation on the human body thermal radiation distribution map to generate a direction field map, the simultaneously acquired human body point cloud data and the human body thermal radiation distribution map are fused.

[0044] Specifically, the human body point cloud data and the simultaneously acquired human body thermal radiation distribution map can be fused using Kalman filtering to obtain a fused human body thermal radiation distribution map. This fused map includes the heat values ​​of obscured parts of the body (equivalent to penetrating the obstruction to obtain heat information; the heat values ​​of the obscured parts can be inferred using relevant algorithms). Specifically, Kalman filtering fuses a human body thermal radiation distribution map (referred to as a thermal map) captured by an infrared thermal imaging sensor and a point cloud map simultaneously captured by a millimeter-wave radar. The goal of fusion is to cross-reference the two images; these two images are related and can complement each other. Radar can see through obstacles but the image is unclear, while thermal imaging can see the outline but cannot penetrate them. The output of the fusion is more reliable information about the human body's position and outline. Subsequent processing is then performed based on the fused human body thermal radiation distribution map (i.e., thermal gradient calculation to generate a direction field map).

[0045] Millimeter-wave radar can operate in the 77GHz frequency band. It can penetrate obstacles such as clothing and thin walls using a multiple-input multiple-output (MIMO) antenna array to generate point cloud data (approximately 200 points per frame). Infrared sensors and millimeter-wave radar can employ a timestamp synchronization strategy. The infrared sensor captures the distribution of human body thermal radiation, while the millimeter-wave radar analyzes movement velocity using the Doppler effect. This dual-modal data can be transmitted to edge nodes via a USB 3.0 interface at a sampling rate of 30fps. For example, in fall detection, infrared data captures abrupt changes in torso temperature distribution, while the radar detects changes in vertical acceleration, collaboratively filtering out false alarms (such as falling objects).

[0046] In this embodiment of the invention, point cloud data and human thermal radiation distribution map (specifically, thermal imaging contours) are fused using Kalman filtering, which can solve the problem of missed detection by a single sensor in occluded scenarios (such as fall detection in the bathroom).

[0047] In addition, the specific process of fusing the human body point cloud data with the human body thermal radiation distribution map collected at the same time using Kalman filtering includes:

[0048] Step 1: Obtain sensor weights determined based on the environmental noise level. The sensor weights include the weights of the infrared sensor and the millimeter-wave radar.

[0049] Specifically, there are several possible methods to determine the environmental noise level. For example, one can analyze the image contrast and sharpness of infrared thermal imaging (i.e., human body thermal radiation distribution maps) to determine the environmental noise level corresponding to the infrared sensor. Alternatively, one can pre-set rules and look up tables, such as pre-setting sensor weight allocation rules for different weather conditions based on time and / or geographical location. Furthermore, one can adaptively adjust the weights based on fluctuations in the model fusion results; if the fluctuations are large, it indicates that the sensor may be unreliable, requiring dynamic weight adjustment.

[0050] Step 2: According to the sensor weights, perform Kalman filtering fusion on the human body point cloud data and the human body thermal radiation distribution map collected at the same time; wherein, the weight of the infrared sensor is used as the weight of the human body thermal radiation distribution map, and the weight of the millimeter-wave radar is used as the weight of the human body point cloud data.

[0051] The embodiments of the present invention not only improve the robustness of behavior recognition in occluded scenarios through multimodal sensor fusion (cooperative detection by infrared sensors and millimeter-wave radar), but also add dynamic perception fusion, that is, adaptively adjusting the sensor weights based on the environmental noise level (e.g., increasing the weight of infrared sensors to 70% in rainy or foggy weather).

[0052] In addition, during the Kalman filtering fusion of human point cloud data and human thermal radiation distribution map, dual-modal spatial alignment can be achieved through a checkerboard calibration board, with the error controlled within ±2 pixels.

[0053] Step S102: Perform thermal gradient calculation on the human body thermal radiation distribution map to generate an orientation field map to quantify the direction of heat transfer (e.g., heat from the torso diffuses to the ground when falling).

[0054] Specifically, the orientation field map here can also be called a thermal gradient orientation field map, or a thermal gradient map. Each point is represented by two values: the direction of thermal change (represented by an angle value of 0-180 degrees) and the rate of change.

[0055] In addition, after obtaining the human body thermal radiation distribution map (if point cloud data is collected using millimeter-wave radar at the same time as the human body thermal radiation distribution map, then a fused human body thermal radiation distribution map is obtained), before performing thermal gradient calculation on the human body thermal radiation distribution map and generating the direction field map, an adaptive median filter can be used to dynamically adjust the window size according to the local temperature variance (e.g., from 3×3 to 7×7) to eliminate interference from environmental heat sources (such as radiators) and suppress thermal noise.

[0056] Subsequently, thermal gradient calculations can be performed based on the filtered human body thermal radiation distribution map to generate a direction field map.

[0057] Step S103: Based on the orientation field map, obtain the histogram of oriented gradient directions (HOG-T).

[0058] In this embodiment of the invention, the orientation field can be divided into multiple intervals (e.g., 8 intervals), and then the gradient magnitude of each interval can be statistically analyzed to generate a multi-dimensional feature vector (e.g., a 128-dimensional feature vector). Specifically, clustering statistics can be performed on the orientation field map. For example, the 0-180 degree direction can be divided into 9 parts (0, 20, 40, 60, ...), and then the rate of change values ​​of the same range or the same direction can be summed up to obtain a simple array. If a bar chart is drawn using tools such as Excel, it becomes a simple histogram. This histogram can specifically be made for small blocks of a certain size (e.g., 8*8 pixels) in the image, and each small block has a feature histogram. Therefore, each frame of human body thermal radiation distribution map corresponds to many such histograms.

[0059] As can be seen from the above, before using the human behavior recognition model to classify and predict human behavior, the data collected by the sensors (infrared sensors, millimeter-wave radar) can be preprocessed (including Kalman filter fusion, thermal noise suppression, thermal gradient calculation to generate orientation field map and generate thermal gradient orientation histogram).

[0060] Step S104: Using a human behavior recognition model, predict the thermal gradient direction histogram to obtain the behavior classification of the target object.

[0061] The human behavior recognition method provided in this embodiment of the invention achieves posture recognition (i.e., joint point detection) through posture-behavior joint learning, and then achieves behavior classification.

[0062] In some optional embodiments, the behavior classification includes at least one of the following: falls, smoking, abnormal infusion movements (abnormal hand movement trajectory, such as grabbing the infusion tube), epileptic seizures, getting out of bed after surgery, abnormal positioning of critically ill patients, and theft of newborns.

[0063] In addition, the human behavior recognition model is also used to output risk levels (including low, medium and high).

[0064] Based on the human behavior classification results and risk levels output by the human behavior recognition model, structured alarm information can be generated, such as {"behavior type":"fall","location":"bathroom","confidence level":0.91}. It can then be pushed to the platform via the Message Queuing Telemetry Transport (MQTT) protocol, supporting tiered responses (such as triggering audible and visual alarms for high-risk situations).

[0065] The human behavior recognition method provided in this embodiment quantifies the spatiotemporal features of actions using thermal gradient orientation histogram (HOG-T), which solves the problem of low accuracy in human behavior recognition caused by insufficient information in low-resolution thermal imaging data. Moreover, it does not require super-resolution reconstruction preprocessing to improve image quality, avoids high-complexity calculations, and has high efficiency in human behavior recognition, meeting the real-time requirements of human behavior recognition.

[0066] In some optional implementations, step S104, namely, predicting the thermal gradient orientation histogram using a human behavior recognition model, includes:

[0067] Step S1041: Obtain a motion energy image (MEI) based on the superposition of the orientation field maps of multiple consecutive frames. For example, the thermal gradient changes (i.e., orientation field maps) of 10 consecutive frames (approximately 0.33 seconds) can be superimposed to form an energy map to highlight the continuity of motion and reflect areas of dense motion.

[0068] This process of converting the orientation field map into a kinetic energy map is also known as trajectory encoding.

[0069] Step S1042: Use the human behavior recognition model to predict the thermal gradient direction histogram and the kinetic energy map.

[0070] Specifically, the human behavior recognition model can include a two-stream (spatial stream and temporal stream) network and a cross-attention mechanism. The two-stream network processes the thermal gradient orientation histogram of a single frame and the kinetic energy map obtained by superimposing orientation field maps from multiple frames, respectively. Specifically, it extracts features from the thermal gradient orientation histogram and the kinetic energy map, outputting spatial features and temporal features, and then fuses these features through the cross-attention mechanism to obtain fused features, which include spatial pose information and temporal motion trajectory information.

[0071] In other words, in this embodiment of the invention, the human behavior recognition model not only predicts human behavior classification based on the thermal gradient orientation histogram, but also integrates the kinetic energy map to predict human behavior classification.

[0072] In some optional implementations, the human behavior recognition model includes the YOLO (You Only Look Once) model, the backbone network of which is the Cross Stage Partial Darknet (CSPDarknet53), and the Cross Stage Partial Darknet embeds an SE (Squeeze-and-Excitation) module, which is used to enhance the feature response of key points of the human torso (such as the hip and shoulder).

[0073] In this embodiment of the invention, the backbone network of the YOLO model was optimized: the SE attention module was embedded into CSPDarknet53 to enhance the feature response of key points of the human torso (such as the hip and shoulder).

[0074] In some optional implementations, the human behavior recognition model includes a backbone network and a spatiotemporal fusion module. The spatiotemporal fusion module is used to aggregate features of the thermal gradient orientation histogram and the kinetic energy map from multiple consecutive frames and output the behavior classification prediction result.

[0075] Specifically, the spatiotemporal fusion module can aggregate features of the thermal gradient orientation histogram and the kinetic energy map from multiple consecutive frames (e.g., 5 frames) using 3D convolutional kernels (the kernel size could be, for example, 3×3×3). These features can be fused features output from the dual-stream (spatial and temporal) network and cross-attention mechanism described above.

[0076] Specifically, as mentioned above, human behavior recognition models can include the YOLO model, and the backbone network and spatiotemporal fusion module mentioned here can both belong to the YOLO model.

[0077] In summary, in a specific implementation, the human behavior recognition model may include a dual-stream (spatial stream and temporal stream) network, a cross-attention mechanism, and a YOLO model connected in sequence. The YOLO model includes a CSPDarknet53 (as the backbone network) with an embedded SE attention module and a spatiotemporal fusion module.

[0078] In addition, before deployment, the human behavior recognition model can be compressed using a teacher-student knowledge distillation method to adapt it to low-computing-power edge node devices. Furthermore, the model can be quantized using INT8 (the process of converting the weights and activation values ​​in the model from the commonly used 32-bit floating-point numbers (FP32) to 8-bit integers (INT8)) to further improve the model's inference speed.

[0079] This invention uses only infrared sensors and millimeter-wave radar to collect data. Compared to visible light image acquisition sensors, which can more accurately collect personal information, this invention avoids privacy leaks. Furthermore, to further protect privacy, this invention can also perform real-time mosaic processing on the thermal imaging image, i.e., the human body thermal radiation distribution map, such as applying Gaussian blur (e.g., standard deviation σ = 5). Moreover, when performing human behavior recognition based on images acquired by visible light image acquisition sensors, the recognition rate drops by more than 50% at night or in inclement weather.

[0080] In this embodiment of the invention, the combination of Gaussian blurring and edge computing can achieve "detectable but not reversible" anonymization processing, thereby realizing privacy protection.

[0081] Raw thermal imaging data, i.e., the raw human body thermal radiation distribution map, contains a large amount of invalid background information, which leads to low efficiency in human behavior recognition. Therefore, after obtaining the raw human body thermal radiation distribution map, or after fusing the raw human body thermal radiation distribution map with point cloud data collected by millimeter-wave radar to obtain a fused human body thermal radiation distribution map, a background stripping algorithm can be used to segment the static background, reducing invalid data by 70%.

[0082] In summary, the human behavior recognition method provided in this embodiment of the invention is a human behavior recognition method based on thermal imaging and spatiotemporal trajectory modeling, or a human behavior recognition method based on thermodynamic trajectory modeling. It can improve the accuracy of human behavior detection and reduce the false alarm rate; it has strong environmental adaptability, and the recognition rate is basically unaffected even in rainy or foggy weather; it has low deployment cost, with a single edge device node supporting 8 video streams, resulting in low hardware cost; and it has strong predictive ability, with temporal modeling of action sequences, which can provide early warning of potential risks.

[0083] This embodiment also provides a human behavior recognition device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0084] This embodiment provides a human behavior recognition device, such as Figure 2 As shown, it includes:

[0085] Data acquisition module 201 is used to acquire the human body thermal radiation distribution map collected by the infrared sensor for the target object;

[0086] The orientation field map generation module 202 is used to calculate the thermal gradient of the human body thermal radiation distribution map and generate an orientation field map.

[0087] The thermal gradient direction histogram acquisition module 203 is used to acquire the thermal gradient direction histogram based on the direction field map.

[0088] The prediction module 204 is used to predict the thermal gradient direction histogram using a human behavior recognition model to obtain the behavior classification of the target object.

[0089] In some optional implementations, the prediction module 204 includes:

[0090] The kinetic energy map acquisition unit is used to acquire a kinetic energy map obtained by superimposing the orientation field maps based on multiple consecutive frames;

[0091] The prediction unit is used to predict the thermal gradient direction histogram and the kinetic energy map using the human behavior recognition model.

[0092] In some optional implementations, the human behavior recognition model includes a YOLO model, the backbone network of which is a cross-stage Darknet, and the cross-stage Darknet embeds an SE module, which is used to enhance the feature response of key points of the human torso.

[0093] In some optional implementations, the human behavior recognition model includes a backbone network and a spatiotemporal fusion module, wherein the spatiotemporal fusion module is used to aggregate features of the thermal gradient orientation histogram and the kinetic energy map of multiple consecutive frames, and output the behavior classification prediction result.

[0094] In some alternative implementations, the behavior classification includes at least one of the following: falls, smoking, abnormal movements during intravenous infusion, epileptic seizures, getting out of bed after surgery, abnormal positioning of critically ill patients, and theft of newborns;

[0095] And / or,

[0096] The human behavior recognition model is also used to output risk levels.

[0097] In some optional embodiments, the human behavior recognition device further includes:

[0098] The fusion module is used to acquire human point cloud data collected by millimeter-wave radar targeting the target object, and to fuse the human point cloud data with the human thermal radiation distribution map acquired at the same time using Kalman filtering to obtain the fused human thermal radiation distribution map; and / or,

[0099] The thermal noise suppression module is used to eliminate environmental heat source interference by dynamically adjusting the window size based on the local temperature variance using an adaptive median filter.

[0100] In some optional implementations, the fusion module is specifically used to obtain sensor weights determined based on the environmental noise level, the sensor weights including the weights of the infrared sensor and the millimeter-wave radar; according to the sensor weights, the human body point cloud data and the human body thermal radiation distribution map collected at the same time are fused using Kalman filtering; wherein, the weights of the infrared sensor are used as the weights of the human body thermal radiation distribution map, and the weights of the millimeter-wave radar are used as the weights of the human body point cloud data.

[0101] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0102] In this embodiment, the human behavior recognition device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0103] This invention also provides a computer device having the above-described features. Figure 2 The human behavior recognition device shown.

[0104] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention. This computer device can be used as an edge node, such as... Figure 3As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0105] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0106] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0107] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0109] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0110] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0111] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0112] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for recognizing human behavior, characterized in that, The method includes: Obtain the human body thermal radiation distribution map collected by the infrared sensor for the target object; Thermal gradient calculations are performed on the aforementioned human body thermal radiation distribution map to generate a directional field map; Based on the aforementioned orientation field map, a thermal gradient orientation histogram is obtained; Using a human behavior recognition model, the thermal gradient direction histogram is predicted to obtain the behavior classification of the target object.

2. The method according to claim 1, characterized in that, The step of using a human behavior recognition model to predict the thermal gradient direction histogram includes: Obtain the kinetic energy map obtained by superimposing the orientation field maps from multiple consecutive frames; The human behavior recognition model is used to predict the thermal gradient direction histogram and the kinetic energy map.

3. The method according to claim 1 or 2, characterized in that, The human behavior recognition model includes the YOLO model, the backbone network of which is the cross-stage Darknet, and the cross-stage Darknet embeds an SE module, which is used to enhance the feature response of key points of the human torso.

4. The method according to claim 2, characterized in that, The human behavior recognition model includes a backbone network and a spatiotemporal fusion module. The spatiotemporal fusion module is used to aggregate the features of the thermal gradient orientation histogram and the kinetic energy map of multiple consecutive frames, and output the behavior classification prediction result.

5. The method according to claim 1, characterized in that, The behavioral categories include at least one of the following: falls, smoking, abnormal movements during intravenous infusion, epileptic seizures, getting out of bed after surgery, abnormal positioning of critically ill patients, and theft of newborns; And / or, The human behavior recognition model is also used to output risk levels.

6. The method according to claim 1, characterized in that, Before performing thermal gradient calculation on the human body thermal radiation distribution map to generate the directional field map, the method further includes: Acquire human body point cloud data collected by millimeter-wave radar targeting the target object; fuse the human body point cloud data with the human body thermal radiation distribution map collected simultaneously using Kalman filtering to obtain the fused human body thermal radiation distribution map; and / or, An adaptive median filter is used to dynamically adjust the window size based on the local temperature variance, thereby eliminating interference from environmental heat sources.

7. The method according to claim 6, characterized in that, The step of fusing the human body point cloud data with the human body thermal radiation distribution map acquired at the same time using Kalman filtering includes: Obtain sensor weights determined based on the ambient noise level, the sensor weights including the weights of the infrared sensor and the millimeter-wave radar; According to the sensor weights, the human body point cloud data and the human body thermal radiation distribution map collected at the same time are fused by Kalman filtering; wherein, the weight of the infrared sensor is used as the weight of the human body thermal radiation distribution map, and the weight of the millimeter-wave radar is used as the weight of the human body point cloud data.

8. A human behavior recognition device, characterized in that, The device includes: The data acquisition module is used to acquire the human body thermal radiation distribution map collected by the infrared sensor for the target object; The orientation field map generation module is used to calculate the thermal gradient of the human body thermal radiation distribution map and generate an orientation field map. A thermal gradient direction histogram acquisition module is used to acquire a thermal gradient direction histogram based on the direction field map. The prediction module is used to predict the thermal gradient direction histogram using a human behavior recognition model to obtain the behavior classification of the target object.

9. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the human behavior recognition method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the human behavior recognition method according to any one of claims 1 to 7.

Images