A method and device for condition monitoring combining low-light cameras and passive infrared cameras.

Abstract

This application discloses a state monitoring method and apparatus combining a low-light camera and a passive infrared camera, belonging to the field of target recognition technology. The method includes: acquiring images from the low-light camera and the passive infrared camera respectively and preprocessing them to obtain image pairs containing a target number of images, each image pair including one low-light image and one passive infrared image; identifying each image pair based on a deep learning model to obtain target person information in each image pair; performing action evaluation on the target person information in each image pair to obtain action evaluation results; performing body state evaluation on the target person information in each image pair to obtain body state evaluation results; performing facial expression evaluation on the target person information in each image pair to obtain facial expression evaluation results; and obtaining the state monitoring result of the target person based on the action evaluation result, body state evaluation result, and facial expression evaluation result. This method improves the accuracy and efficiency of state monitoring.

Description

Technical Field

[0001] This application belongs to the field of target recognition technology, and in particular relates to a state monitoring method and device that combines a low-light camera and a passive infrared camera. Background Technology

[0002] In power field operations, the complex environment and susceptibility to weather changes often present inherent safety risks. Especially under extreme weather conditions, such as rain, snow, or nighttime, poor lighting and the difficulty of achieving full coverage with existing lighting equipment can create blind spots, making the safety monitoring of personnel particularly crucial. Therefore, effectively ensuring the safety of personnel, especially in complex and unpredictable working environments, has become a critical and urgent issue that needs to be addressed.

[0003] The development of modern sensor technology has provided an effective means to solve this problem. High-precision sensors can detect and monitor the physical condition of a target in real time, allowing for timely assessment of potential safety risks. However, under extreme lighting conditions, traditional monitoring methods often suffer from insufficient sensing or unclear identification.

[0004] In existing technologies, monitoring methods based on infrared sensing, thermal imaging, and visual sensing have solved the problem of human identification in low-light environments to some extent. However, these methods often rely on complex equipment configurations or high energy consumption, which is not conducive to their widespread application in actual operations. Existing technologies suffer from low monitoring accuracy, poor real-time performance, and low efficiency in user status monitoring. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a condition monitoring method combining a low-light camera and a passive infrared camera, which improves the accuracy and efficiency of condition monitoring.

[0006] In a first aspect, this application provides a state monitoring method combining a low-light camera and a passive infrared camera, the method comprising:

[0007] Images acquired by a low-light camera and a passive infrared camera are acquired and preprocessed to obtain a target number of image pairs, each image pair consisting of one low-light image and one passive infrared image.

[0008] Each image pair is identified using a deep learning model to obtain information about the target person in each image pair.

[0009] Action evaluation is performed on the target person information in each image pair to obtain the action evaluation results;

[0010] The physical condition of the target person in each image pair is assessed to obtain the physical condition assessment results.

[0011] Facial expression assessment is performed on the target person information in each image pair to obtain the facial expression assessment results;

[0012] Based on the action assessment results, body condition assessment results, and facial expression assessment results, the state monitoring results of the target person are obtained.

[0013] According to one embodiment of this application, the step of acquiring images from a low-light camera and a passive infrared camera respectively and performing preprocessing to obtain a target number of image pairs includes:

[0014] Acquire low-light video captured by a low-light camera and passive infrared video captured by a passive infrared camera within the same time period;

[0015] The low-light video is decomposed into multiple low-light images. Each low-light image is grayscaled and edge information is extracted using the Canny operator to obtain multiple low-light images after edge information extraction.

[0016] After binarizing the multi-frame low-light images after extracting edge information, the pixel sum of all pixels in each low-light image is calculated;

[0017] The binarized multi-frame low-light images are divided into blocks according to time sequence. The pixels and the largest low-light image are extracted from each block to obtain the target number of low-light images.

[0018] The passive infrared video is decomposed into multiple passive infrared images. The sum of pixels of all pixels in each passive infrared image is calculated. The multiple passive infrared images are divided into blocks according to time sequence. The passive infrared image with the largest sum of pixels in each block is extracted to obtain the target number of passive infrared images.

[0019] Based on the low-light images and passive infrared images of the target number, image pairs of the target number are obtained.

[0020] According to one embodiment of this application, the step of identifying each image pair based on a deep learning model to obtain target person information in each image pair includes:

[0021] Construct a deep learning sample library and a preset deep learning model, wherein the sample library includes multiple image pairs;

[0022] The corner pooling downsampling technique is used to sample multiple image pairs in the sample library to obtain the first dataset;

[0023] The pre-set deep learning model is trained based on the first dataset to obtain the trained deep learning model.

[0024] The target number of image pairs are input into a trained deep learning model, and each image pair is identified to obtain the target person information in each image pair.

[0025] According to one embodiment of this application, the downsampling technique using corner pooling is used to sample multiple image pairs in a sample library to obtain a dataset, including:

[0026] Each image in the sample library is processed into a grid to obtain multiple grids. Each grid includes sub-grid A, sub-grid B, sub-grid C, sub-grid D, sub-grid E, sub-grid F, sub-grid G, sub-grid H and sub-grid I.

[0027] Calculate the differences in grayscale values ​​between sub-mesh B, sub-mesh D, sub-mesh F, sub-mesh H, and sub-mesh E respectively, and then calculate the first average of the four differences. ;

[0028] Calculate the grayscale differences between sub-mesh A, sub-mesh C, sub-mesh G, sub-mesh I, and sub-mesh E respectively, and calculate the second average of the four differences. ;

[0029] The grayscale value of each grid is obtained based on the first average value and the second average value;

[0030] Based on the relationship between the grayscale value of each grid and a preset threshold, multiple image pairs in the sample library are sampled to obtain a dataset.

[0031] According to one embodiment of this application, the step of performing motion evaluation on the target person information in each image pair to obtain motion evaluation results includes:

[0032] The height of the target person's head from the ground is obtained based on the target person information in each image pair;

[0033] Determine the height of the head from the ground. If the height of the head from the ground is less than 0.5 meters, the target person's real-time state is determined to be lying down, and the action assessment result is that the person is abnormal.

[0034] If the height of the head from the ground is greater than or equal to 0.5 meters but less than 1.5 meters, the real-time state of the target person is determined to be bending over or half-squatting. The state judgment result of the previous moment is reviewed. If the duration of the same state is maintained for more than the preset threshold, the action evaluation result is that the person is abnormal; otherwise, the action evaluation result is that the person is normal.

[0035] If the height of the head from the ground is greater than or equal to 1.5 meters, the target person's real-time state is determined to be standing, and the action assessment result is that the person is normal.

[0036] According to one embodiment of this application, the step of assessing the body state of the target person information in each image pair to obtain the body state assessment result includes:

[0037] Convert the infrared image in each image pair to the RGB color space, and calculate the R component value of each pixel in each infrared image in the RGB color space;

[0038] The R component of each infrared image is obtained based on the R component value of each pixel in the RGB color space.

[0039] Calculate the R component difference between two adjacent infrared images to obtain the R component difference set;

[0040] Determine whether each difference in the set of R component differences is within a preset range. If all differences are within the preset range, the physical condition assessment result is determined to be normal. If at least one difference is outside the preset range, the physical condition assessment result is determined to be abnormal.

[0041] According to one embodiment of this application, the step of evaluating facial expressions for the target person information in each image pair to obtain facial expression evaluation results includes:

[0042] Construct a pre-defined generative adversarial network model;

[0043] Obtain multiple images of human facial expressions to obtain the second dataset;

[0044] The pre-defined generative adversarial network model is trained based on the second dataset to obtain the trained generative adversarial network model.

[0045] The RGB values ​​of the low-light image in each image pair are used as the first three dimensions, and the RGB values ​​of the passive infrared image are used as the last three dimensions. These are then input into the trained generative adversarial network model to obtain the facial expression evaluation results.

[0046] Secondly, this application provides a state monitoring device combining a low-light camera and a passive infrared camera, the device comprising:

[0047] The acquisition module is used to acquire images collected by the low-light camera and the passive infrared camera respectively and perform preprocessing to obtain a target number of image pairs, wherein the image pair includes one low-light image and one passive infrared image;

[0048] The first processing module is used to identify each image pair based on a deep learning model and obtain the target person information in each image pair;

[0049] The second processing module is used to perform motion evaluation on the target person information in each image pair to obtain motion evaluation results.

[0050] The third processing module is used to assess the physical condition of the target person in each image pair and obtain the physical condition assessment result.

[0051] The fourth processing module is used to evaluate the facial expressions of the target person in each image pair and obtain the facial expression evaluation results.

[0052] The monitoring module is used to obtain the status monitoring results of the target person based on the action evaluation results, body status evaluation results, and facial expression evaluation results.

[0053] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the state monitoring method combining a low-light camera and a passive infrared camera as described in the first aspect above.

[0054] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the state monitoring method combining a low-light camera and a passive infrared camera as described in the first aspect above.

[0055] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the state monitoring method combining a low-light camera and a passive infrared camera as described in the first aspect.

[0056] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the state monitoring method combining a low-light camera and a passive infrared camera as described in the first aspect above.

[0057] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0058] The present invention provides a state monitoring method combining a low-light camera and a passive infrared camera, which has the following advantages over the prior art:

[0059] (1) This invention acquires images from low-light cameras and passive infrared cameras respectively, preprocesses them to obtain a target number of image pairs, and then identifies each image pair based on a deep learning model. This effectively extracts information about the target person and evaluates their actions, body state, and facial expressions, enabling comprehensive monitoring of the target person's behavior and health status. By integrating the action evaluation results, body state evaluation results, and facial expression evaluation results, the invention can more accurately reflect the target person's real-time state, improve the accuracy and real-time performance of state monitoring, enhance the ability to identify the target person's state, and promptly detect potential abnormal states or health risks, effectively reducing safety hazards and health risks.

[0060] (2) This invention trains a generative adversarial network model by acquiring multiple facial expression images as a dataset, which can more accurately identify and evaluate the facial expressions of the target person. By inputting the RGB values ​​of low-light images and passive infrared images into the trained generative adversarial network model, the features of different image data can be comprehensively considered to obtain more accurate facial expression evaluation results, improve the accuracy and stability of facial expression recognition, and enable more accurate evaluation under different lighting conditions.

[0061] (3) This invention can effectively extract infrared information from images by converting infrared images into RGB color space and calculating the R component value of each pixel. By calculating the difference of the R component between two adjacent infrared images and comparing it with a preset range, it can more accurately determine whether the target person's physical condition is normal, effectively improving the real-time monitoring capability of the target person's physical condition and enabling timely detection of abnormalities. Attached Figure Description

[0062] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0063] Figure 1 This is one of the flowcharts illustrating the state monitoring method combining a low-light camera and a passive infrared camera provided in the embodiments of this application;

[0064] Figure 2 This is one of the schematic diagrams of the sub-mesh provided in the embodiments of this application;

[0065] Figure 3 This is a second schematic diagram of the sub-mesh provided in the embodiments of this application;

[0066] Figure 4 This is a schematic diagram of the structure of the condition monitoring device combining a low-light camera and a passive infrared camera provided in the embodiments of this application;

[0067] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0069] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0070] The following description, in conjunction with the accompanying drawings, details the state monitoring method, state monitoring device, electronic device, and readable storage medium combining a low-light camera and a passive infrared camera provided in this application, through specific embodiments and application scenarios.

[0071] Among them, the status monitoring method combining low-light cameras and passive infrared cameras can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0072] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0073] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0074] The state monitoring method combining a low-light camera and a passive infrared camera provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the state monitoring method combining a low-light camera and a passive infrared camera. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The state monitoring method combining a low-light camera and a passive infrared camera provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0075] Figure 1 This is one of the flowcharts illustrating the state monitoring method combining a low-light camera and a passive infrared camera provided in this application embodiment, such as... Figure 1 As shown, the state monitoring method combining a low-light camera and a passive infrared camera includes steps 110, 120, 130, 140, 150, and 160.

[0076] Step 110: Acquire images from the low-light camera and the passive infrared camera respectively and preprocess them to obtain the target number of image pairs, wherein each image pair includes one low-light image and one passive infrared image;

[0077] In some embodiments, the step of acquiring images from a low-light camera and a passive infrared camera respectively and performing preprocessing to obtain the target number of image pairs includes:

[0078] Acquire low-light video captured by a low-light camera and passive infrared video captured by a passive infrared camera within the same time period;

[0079] The low-light video is decomposed into multiple low-light images. Each low-light image is grayscaled and edge information is extracted using the Canny operator to obtain multiple low-light images after edge information extraction.

[0080] After binarizing the multi-frame low-light images after extracting edge information, the pixel sum of all pixels in each low-light image is calculated;

[0081] The binarized multi-frame low-light images are divided into blocks according to time sequence. The pixels and the largest low-light image are extracted from each block to obtain the target number of low-light images.

[0082] The passive infrared video is decomposed into multiple passive infrared images. The sum of pixels of all pixels in each passive infrared image is calculated. The multiple passive infrared images are divided into blocks according to time sequence. The passive infrared image with the largest sum of pixels in each block is extracted to obtain the target number of passive infrared images.

[0083] Based on the low-light images and passive infrared images of the target number, image pairs of the target number are obtained.

[0084] For example, on the same side of the power operation support vehicle, a low-light camera and a passive infrared camera are deployed at the same height. The distance between the low-light camera and the passive infrared camera is less than 10 centimeters to avoid excessive errors during subsequent image registration. In the case of poor ambient light (such as night operation), the low-light camera and the passive infrared camera are set to the same sampling frequency (same frame rate) to obtain the low-light video captured by the low-light camera and the passive infrared video captured by the passive infrared camera within the same time period.

[0085] Due to poor ambient light, not all frames in the low-light video and passive infrared video can be used for subsequent recognition steps. Therefore, it is necessary to filter the images in the low-light video and passive infrared video to obtain the target number of image pairs. By extracting the edges of the low-light images, images with more edge information are selected, that is, images captured by the low-light camera that are clearer. The corresponding infrared images are then selected as "image pairs". The image pair selection process includes the following steps:

[0086] (1) Select low-light video stream and infrared video stream for the same period of time, for example, 1 minute (60 seconds).

[0087] (2) If the sampling frequency of the two cameras is set to 15 frames / second, then a 1-minute video can be decomposed into 900 low-light frame images and infrared frame images;

[0088] (3) For 900 low-light images, grayscale processing is performed on each image and the edge information is extracted one by one using the Canny operator. For each image with extracted edge information, binarization processing is performed and the pixel value is calculated pixel by pixel: 0 is black and 1 is white.

[0089] (4) Calculate the sum of all pixels in each image, and divide the 900 images into 6 blocks according to time sequence, with each block consisting of 150 images;

[0090] (5) Extract the image pixels and the largest image from each block (i.e., extract one image per minute), and form 6 "image pairs" by combining the low-light image and infrared image corresponding to the 6 selected images.

[0091] It should be noted that the length of the video segment and the selection frequency of key "image pairs" can be adjusted according to actual needs.

[0092] In this embodiment, by acquiring image data from both a low-light camera and a passive infrared camera, and performing preprocessing, edge extraction, grayscale conversion, binarization, and temporal segmentation, the quality of the target image pairs is effectively improved. By calculating the sum of pixels in the passive infrared images and extracting the image with the largest sum of pixels to obtain the passive infrared image of the target, the target person information can be identified more accurately.

[0093] Step 120: Based on the deep learning model, identify each image pair to obtain the target person information in each image pair;

[0094] In some embodiments, the process of identifying each image pair based on a deep learning model to obtain target person information in each image pair includes:

[0095] Construct a deep learning sample library and a preset deep learning model, wherein the sample library includes multiple image pairs;

[0096] The corner pooling downsampling technique is used to sample multiple image pairs in the sample library to obtain the first dataset;

[0097] The pre-set deep learning model is trained based on the first dataset to obtain the trained deep learning model.

[0098] The target number of image pairs are input into a trained deep learning model, and each image pair is identified to obtain the target person information in each image pair.

[0099] It is easy to understand that after obtaining the target number of image pairs, it is necessary to extract the target person information from the image pairs. Due to poor ambient light, neither low-light nor passive infrared images alone can accurately extract person information, so it is necessary to combine two images. First, a deep learning sample library and a preset deep learning model are constructed. The sample library includes passive infrared images of people and "sample image pairs" with the same pose. The sample label is defined as the outline of the person. Traditional image recognition and synthesis often use upsampling methods to obtain or synthesize clearer images, thereby supporting more accurate recognition. The recognition involved in this application only needs to identify the outline information of the person. To avoid underfitting of the model training (due to the ambient light, there may be too much image noise and interference), a corner pooling downsampling technique is used to initially identify the outline region of the person.

[0100] In some embodiments, the downsampling technique using corner pooling is used to sample multiple image pairs in the sample library to obtain a dataset, including:

[0101] Each image in the sample library is processed into a grid to obtain multiple grids. Each grid includes sub-grid A, sub-grid B, sub-grid C, sub-grid D, sub-grid E, sub-grid F, sub-grid G, sub-grid H and sub-grid I.

[0102] Calculate the differences in grayscale values ​​between sub-mesh B, sub-mesh D, sub-mesh F, sub-mesh H, and sub-mesh E respectively, and then calculate the first average of the four differences. ;

[0103] Calculate the grayscale differences between sub-mesh A, sub-mesh C, sub-mesh G, sub-mesh I, and sub-mesh E respectively, and calculate the second average of the four differences. ;

[0104] The grayscale value of each grid is obtained based on the first average value and the second average value;

[0105] Based on the relationship between the grayscale value of each grid and a preset threshold, multiple image pairs in the sample library are sampled to obtain a dataset.

[0106] For example, each image in the sample library is meshed to obtain a 9×9 grid, with each grid containing 9 square sub-grids. Figure 2 This is one of the schematic diagrams of the sub-mesh provided in the embodiments of this application, such as... Figure 2 As shown, each grid includes subgrid A, subgrid B, subgrid C, subgrid D, subgrid E, subgrid F, subgrid G, subgrid H and subgrid I.

[0107] For each grid's nine sub-grids, calculate the difference in grayscale values ​​between sub-grids B, D, F, H, and E, and then calculate the average of these four differences, denoted as . This indicates the grayscale difference between directly adjacent pixels. The grayscale differences of sub-mesh A, sub-mesh C, sub-mesh G, sub-mesh I, and sub-mesh E are calculated, and the average of the four differences is denoted as . This indicates the grayscale difference between the pixels and their indirect neighbors.

[0108] The grayscale value of each grid is obtained based on the first average and the second average. The formula for calculating the grayscale value is as follows:

[0109]

[0110] Where α is an empirical coefficient, which is taken as 0.7 here. The grayscale value of the grid. The grayscale difference between indirectly adjacent pixels. It represents the grayscale difference between directly adjacent pixels.

[0111] Repeat the above steps to downsample all grids in the entire image using the "corner pooling method". Set a threshold β (which can be flexibly determined according to the light and noise environment of the real-time operation environment). Use the β coefficient to filter the downsampled entire image and remove grid points with values ​​less than β to obtain the blurred outline of the person after upsampling.

[0112] It should be noted that a grid consisting of 9 sub-grids can be selected, with all grids connected but not overlapping or repeating. Alternatively, the grid area can be determined by moving one sub-grid step at a time. Figure 3This is a second schematic diagram of the sub-mesh provided in the embodiments of this application, as shown below. Figure 3 As shown, the first grid is selected as ABCDEFGHI, and the second can be selected as BCEFHIJMP.

[0113] In this embodiment, by processing each image in the sample library into a grid and calculating the gray value difference between different sub-grids, the feature information of the image can be extracted more accurately. By calculating the first and second average gray value differences, the gray value of each grid is obtained. Based on the relationship between the gray value of each grid and a preset threshold, image sampling is performed, which can generate training datasets more efficiently and improve the training effect of deep learning models.

[0114] Furthermore, a pre-set deep learning model is trained based on the first dataset to obtain a trained deep learning model. The target number of image pairs are then input into the trained deep learning model to identify each image pair, extract the contour information of the person, and obtain the target person information in each image pair.

[0115] In this embodiment, a deep learning sample library containing multiple image pairs is constructed, and a downsampling technique using corner pooling is employed to sample the sample library, resulting in a first dataset for training. A pre-defined deep learning model is then trained based on this first dataset, yielding a well-trained deep learning model. Inputting the target number of image pairs into the trained model for recognition allows for more accurate extraction of target person information from each image pair, effectively improving the accuracy and efficiency of target person recognition.

[0116] Step 130: Perform motion evaluation on the target person information in each image pair to obtain motion evaluation results;

[0117] In some embodiments, the motion evaluation of the target person information in each image pair to obtain the motion evaluation result includes:

[0118] The height of the target person's head from the ground is obtained based on the target person information in each image pair;

[0119] Determine the height of the head from the ground. If the height of the head from the ground is less than 0.5 meters, the target person's real-time state is determined to be lying down, and the action assessment result is that the person is abnormal.

[0120] If the height of the head from the ground is greater than or equal to 0.5 meters but less than 1.5 meters, the real-time state of the target person is determined to be bending over or half-squatting. The state judgment result of the previous moment is reviewed. If the duration of the same state is maintained for more than the preset threshold, the action evaluation result is that the person is abnormal; otherwise, the action evaluation result is that the person is normal.

[0121] If the height of the head from the ground is greater than or equal to 1.5 meters, the target person's real-time state is determined to be standing, and the action assessment result is that the person is normal.

[0122] It is easy to understand that the height of the target person's head from the ground is obtained based on the target person's information in each image pair. The target person's state is determined by the height of the target person's head from the ground, and the real-time state of the target person is divided into four categories: standing, bending over, half-squatting, and lying down.

[0123] The height of a person's head from the ground is obtained and determined. If the height of the head from the ground is less than 0.5 meters, the real-time state of the target person is determined to be lying down, and the action evaluation result is that the person is abnormal.

[0124] If the height of the head from the ground is greater than or equal to 0.5 meters but less than 1.5 meters, the target person's real-time state is determined to be bending over or half-squatting. The state judgment result of the previous moment is reviewed. If the duration of the same state is maintained for more than the preset threshold (e.g., 10 minutes), the action evaluation result is that the person is abnormal; otherwise, the action evaluation result is that the person is normal.

[0125] If the height of the head from the ground is greater than or equal to 1.5 meters, the target person's real-time state is determined to be standing, and the action assessment result is that the person is normal.

[0126] In some embodiments, the overall or partial movement speed of the target person's body can be determined by adjacent images. Based on the image pair extracted at the current moment, 10 consecutive frames of images are extracted forward. The movement speed of the 10 consecutive frames is judged. If the movement speed is greater than a threshold, the movement evaluation result is that the person is abnormal.

[0127] In this embodiment, by acquiring the target person's information in each image pair and calculating the height of their head above the ground, the real-time status of the target person can be determined more accurately. If the head height is less than 0.5 meters, it is determined to be a lying position, and the person is assessed as abnormal; if the head height is between 0.5 meters and 1.5 meters, the changes in the target person's status are further reviewed, and the duration is used to determine whether it is an abnormal state; if the head height is greater than or equal to 1.5 meters, the person is assessed as normal. This achieves motion assessment of the target person's information, enabling timely detection of abnormal situations and reducing potential risks.

[0128] Step 140: Assess the physical condition of the target person in each image pair to obtain the physical condition assessment results;

[0129] In some embodiments, the step of assessing the body condition of the target person information in each image pair to obtain a body condition assessment result includes:

[0130] Convert the infrared image in each image pair to the RGB color space, and calculate the R component value of each pixel in each infrared image in the RGB color space;

[0131] The R component of each infrared image is obtained based on the R component value of each pixel in the RGB color space.

[0132] Calculate the R component difference between two adjacent infrared images to obtain the R component difference set;

[0133] Determine whether each difference in the set of R component differences is within a preset range. If all differences are within the preset range, the physical condition assessment result is determined to be normal. If at least one difference is outside the preset range, the physical condition assessment result is determined to be abnormal.

[0134] It is easy to understand that the physical condition of the target person in each image pair can be assessed by passive infrared imaging to obtain the physical condition assessment result. Since passive infrared has a large error in temperature perception, it is necessary to use the extreme value relative discrimination method within a time period to collect the areas of rapid changes in body surface temperature and the time periods of sudden changes in heart rate, so as to realize the real-time perception of the target person's body surface temperature and heart rate.

[0135] For example, the infrared image is converted to the RGB color space. For 6 "image pairs" within 1 minute, 6 infrared images are selected. For each image, the R component value (0-255) in the RGB color is calculated pixel by pixel. The sum of the R component values ​​in each image is calculated. If the sum of the R component values ​​between any two of the 6 images is within 10%, the physical condition assessment result is determined to be normal; otherwise, it is considered abnormal.

[0136] In this embodiment, by converting the infrared image to the RGB color space and calculating the R component value of each pixel, infrared information in the image can be effectively extracted. By calculating the difference in the R component values ​​of two adjacent infrared images and comparing them with a preset range, it is possible to more accurately determine whether the target person's physical condition is normal, effectively improving the real-time monitoring capability of the target person's physical condition and enabling timely detection of abnormalities.

[0137] Step 150: Evaluate the facial expressions of the target person in each image pair to obtain the facial expression evaluation results;

[0138] In some embodiments, the facial expression evaluation of the target person information in each image pair to obtain the facial expression evaluation result includes:

[0139] Construct a pre-defined generative adversarial network model;

[0140] Obtain multiple images of human facial expressions to obtain the second dataset;

[0141] The pre-defined generative adversarial network model is trained based on the second dataset to obtain the trained generative adversarial network model.

[0142] The RGB values ​​of the low-light image in each image pair are used as the first three dimensions, and the RGB values ​​of the passive infrared image are used as the last three dimensions. These are then input into the trained generative adversarial network model to obtain the facial expression evaluation results.

[0143] It is easy to understand that by constructing a generative adversarial network model, super-resolution reconstruction of facial features can be achieved, while simultaneously judging facial expression features to assist in monitoring the real-time state of the target person. Specifically, it includes the following steps:

[0144] (1) Construct a pre-defined generative adversarial network model and expand the random noise as input to 6 dimensions: the first 3 dimensions are the RGB values ​​of the low-light image and the last 3 dimensions are the RGB values ​​of the infrared image.

[0145] (2) Introduce a public dataset of facial expressions to construct a second dataset. The second dataset can be collected and produced by human-computer interaction. The pre-set generative adversarial network model is trained based on the second dataset to obtain the trained generative adversarial network model.

[0146] (4) Input the RGB values ​​of the low-light image in each image pair as the first three dimensions and the RGB values ​​of the passive infrared image as the last three dimensions into the trained generative adversarial network model to obtain the facial expression evaluation results.

[0147] In this embodiment, by acquiring multiple facial expression images as a dataset to train the generative adversarial network model, the facial expressions of the target person can be more accurately identified and evaluated. By inputting the RGB values ​​of low-light images and passive infrared images into the trained generative adversarial network model, the features of different image data can be comprehensively considered to obtain more accurate facial expression evaluation results, thereby improving the accuracy and stability of facial expression recognition and enabling more accurate evaluation under different lighting conditions.

[0148] Step 160: Based on the action assessment results, body state assessment results, and facial expression assessment results, obtain the state monitoring results of the target person.

[0149] Finally, weights are assigned to the action assessment results, body condition assessment results, and facial expression assessment results based on the actual situation to obtain the target person's state monitoring results. The calculation formula is shown below:

[0150]

[0151] in, The results of monitoring the target person's condition. For the motion evaluation results, As the weight of the action evaluation results, As a result of the physical condition assessment, Assigning weights to the physical condition assessment results For facial expression assessment results, Weights for facial expression assessment results.

[0152] The state monitoring method combining a low-light camera and a passive infrared camera provided in this application acquires and preprocesses images from both cameras to obtain a target number of image pairs. Then, a deep learning model is used to identify each image pair, effectively extracting information about the target person and evaluating their actions, body state, and facial expressions. This allows for comprehensive monitoring of the target person's behavior and health status. By integrating the action evaluation results, body state evaluation results, and facial expression evaluation results, the method can more accurately reflect the target person's real-time state, improving the accuracy and real-time performance of state monitoring, enhancing the ability to identify the target person's state, and promptly detecting potential abnormal states or health risks, effectively reducing safety hazards and health risks.

[0153] The state monitoring method combining a low-light camera and a passive infrared camera provided in this application can be executed by a state monitoring device combining a low-light camera and a passive infrared camera. This application uses the example of a state monitoring device combining a low-light camera and a passive infrared camera executing the state monitoring method to illustrate the state monitoring device combining a low-light camera and a passive infrared camera provided in this application.

[0154] This application also provides a state monitoring device that combines a low-light camera and a passive infrared camera, such as... Figure 4 As shown, the status monitoring device combining a low-light camera and a passive infrared camera includes: an acquisition module 410, a first processing module 420, a second processing module 430, a third processing module 440, a fourth processing module 450, and a monitoring module 460.

[0155] The acquisition module 410 is used to acquire images collected by the low-light camera and the passive infrared camera respectively and perform preprocessing to obtain a target number of image pairs, wherein the image pair includes a low-light image and a passive infrared image.

[0156] The first processing module 420 is used to identify each image pair based on a deep learning model and obtain the target person information in each image pair;

[0157] The second processing module 430 is used to perform motion evaluation on the target person information in each image pair to obtain motion evaluation results.

[0158] The third processing module 440 is used to evaluate the physical condition of the target person in each image pair and obtain the physical condition evaluation result.

[0159] The fourth processing module 450 is used to evaluate the facial expressions of the target person in each image pair and obtain the facial expression evaluation results.

[0160] The monitoring module 460 is used to obtain the status monitoring results of the target person based on the action evaluation results, body status evaluation results and facial expression evaluation results.

[0161] The state monitoring method combining a low-light camera and a passive infrared camera provided in this application acquires and preprocesses images from both cameras to obtain a target number of image pairs. Then, a deep learning model is used to identify each image pair, effectively extracting information about the target person and evaluating their actions, body state, and facial expressions. This allows for comprehensive monitoring of the target person's behavior and health status. By integrating the action evaluation results, body state evaluation results, and facial expression evaluation results, the method can more accurately reflect the target person's real-time state, improving the accuracy and real-time performance of state monitoring, enhancing the ability to identify the target person's state, and promptly detecting potential abnormal states or health risks, effectively reducing safety hazards and health risks.

[0162] The state monitoring device combining a low-light camera and a passive infrared camera provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the embodiment of the state monitoring method combining low-light cameras and passive infrared cameras will not be described again here to avoid repetition.

[0163] In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described embodiment of the state monitoring method combining a low-light camera and a passive infrared camera, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0164] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0165] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the state monitoring method combining a low-light camera and a passive infrared camera, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0166] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0167] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described state monitoring method combining a low-light camera and a passive infrared camera.

[0168] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0169] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the state monitoring method combining a low-light camera and a passive infrared camera, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0170] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a device-level chip, device chip, chip device, or on-chip device chip, etc.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the status monitoring method combining a low-light camera and a passive infrared camera of the various embodiments of this application.

[0173] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0174] In the description of this application, "multiple" means two or more.

[0175] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0176] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0177] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A state monitoring method combining a low-light camera and a passive infrared camera, characterized in that, The method includes: Images acquired by a low-light camera and a passive infrared camera are acquired and preprocessed to obtain a target number of image pairs, each image pair consisting of one low-light image and one passive infrared image. Each image pair is identified using a deep learning model to obtain information about the target person in each image pair. Action evaluation is performed on the target person information in each image pair to obtain the action evaluation results; The physical condition of the target person in each image pair is assessed to obtain the physical condition assessment results. Facial expression assessment is performed on the target person information in each image pair to obtain the facial expression assessment results; Based on the action assessment results, body condition assessment results, and facial expression assessment results, the state monitoring results of the target person are obtained; The process of identifying each image pair based on a deep learning model to obtain target person information in each image pair includes: Construct a deep learning sample library and a preset deep learning model, wherein the sample library includes multiple image pairs; The corner pooling downsampling technique is used to sample multiple image pairs in the sample library to obtain the first dataset; The pre-set deep learning model is trained based on the first dataset to obtain the trained deep learning model. The target number of image pairs are input into a trained deep learning model, and each image pair is identified to obtain the target person information in each image pair. The downsampling technique using corner pooling is used to sample multiple image pairs in the sample library to obtain a dataset, including: Each image in the sample library is processed into a grid to obtain multiple grids. Each grid includes sub-grid A, sub-grid B, sub-grid C, sub-grid D, sub-grid E, sub-grid F, sub-grid G, sub-grid H and sub-grid I. Calculate the differences in grayscale values ​​between sub-mesh B, sub-mesh D, sub-mesh F, sub-mesh H, and sub-mesh E respectively, and then calculate the first average of the four differences. ; Calculate the grayscale differences between sub-mesh A, sub-mesh C, sub-mesh G, sub-mesh I, and sub-mesh E respectively, and calculate the second average of the four differences. ; The grayscale value of each grid is obtained based on the first average value and the second average value; Based on the relationship between the grayscale value of each grid and a preset threshold, multiple image pairs in the sample library are sampled to obtain a dataset.

2. The state monitoring method combining a low-light camera and a passive infrared camera according to claim 1, characterized in that, The process of acquiring and preprocessing images from a low-light camera and a passive infrared camera to obtain the target number of image pairs includes: Acquire low-light video captured by a low-light camera and passive infrared video captured by a passive infrared camera within the same time period; The low-light video is decomposed into multiple low-light images. Each low-light image is grayscaled and edge information is extracted using the Canny operator to obtain multiple low-light images after edge information extraction. After binarizing the multi-frame low-light images after extracting edge information, the pixel sum of all pixels in each low-light image is calculated; The binarized multi-frame low-light images are divided into blocks according to time sequence. The pixels and the largest low-light image are extracted from each block to obtain the target number of low-light images. The passive infrared video is decomposed into multiple passive infrared images. The sum of pixels of all pixels in each passive infrared image is calculated. The multiple passive infrared images are divided into blocks according to time sequence. The passive infrared image with the largest sum of pixels in each block is extracted to obtain the target number of passive infrared images. Based on the low-light images and passive infrared images of the target number, image pairs of the target number are obtained.

3. The state monitoring method combining a low-light camera and a passive infrared camera according to claim 1, characterized in that, The step of evaluating the action of the target person information in each image pair to obtain the action evaluation result includes: The height of the target person's head from the ground is obtained based on the target person information in each image pair; Determine the height of the head from the ground. If the height of the head from the ground is less than 0.5 meters, the target person's real-time state is determined to be lying down, and the action assessment result is that the person is abnormal. If the height of the head from the ground is greater than or equal to 0.5 meters but less than 1.5 meters, the real-time state of the target person is determined to be bending over or half-squatting. The state judgment result of the previous moment is reviewed. If the duration of the same state is maintained for more than the preset threshold, the action evaluation result is that the person is abnormal; otherwise, the action evaluation result is that the person is normal. If the height of the head from the ground is greater than or equal to 1.5 meters, the target person's real-time state is determined to be standing, and the action assessment result is that the person is normal.

4. The state monitoring method combining a low-light camera and a passive infrared camera according to claim 3, characterized in that, The process of assessing the physical condition of the target person in each image pair to obtain the physical condition assessment result includes: Convert the infrared image in each image pair to the RGB color space, and calculate the R component value of each pixel in each infrared image in the RGB color space; The R component of each infrared image is obtained based on the R component value of each pixel in the RGB color space. Calculate the R component difference between two adjacent infrared images to obtain the R component difference set; Determine whether each difference in the set of R component differences is within a preset range. If all differences are within the preset range, the physical condition assessment result is determined to be normal. If at least one difference is outside the preset range, the physical condition assessment result is determined to be abnormal.

5. The state monitoring method combining a low-light camera and a passive infrared camera according to claim 1, characterized in that, The facial expression evaluation of the target person information in each image pair, to obtain the facial expression evaluation result, includes: Construct a pre-defined generative adversarial network model; Obtain multiple images of human facial expressions to obtain the second dataset; The pre-defined generative adversarial network model is trained based on the second dataset to obtain the trained generative adversarial network model. The RGB values ​​of the low-light image in each image pair are used as the first three dimensions, and the RGB values ​​of the passive infrared image are used as the last three dimensions. These are then input into the trained generative adversarial network model to obtain the facial expression evaluation results.

6. A state monitoring device combining a low-light camera and a passive infrared camera, implemented using the state monitoring method combining a low-light camera and a passive infrared camera as described in any one of claims 1 to 5, characterized in that, The device includes: The acquisition module is used to acquire images collected by the low-light camera and the passive infrared camera respectively and perform preprocessing to obtain a target number of image pairs, wherein the image pair includes one low-light image and one passive infrared image; The first processing module is used to identify each image pair based on a deep learning model and obtain the target person information in each image pair; The second processing module is used to perform motion evaluation on the target person information in each image pair to obtain motion evaluation results. The third processing module is used to assess the physical condition of the target person in each image pair and obtain the physical condition assessment result. The fourth processing module is used to evaluate the facial expressions of the target person in each image pair and obtain the facial expression evaluation results. The monitoring module is used to obtain the status monitoring results of the target person based on the action evaluation results, body status evaluation results, and facial expression evaluation results.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the state monitoring method combining a low-light camera and a passive infrared camera as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the state monitoring method combining a low-light camera and a passive infrared camera as described in any one of claims 1 to 5.

Images