Methods for training static human detection models, persistence detection methods, and storage media

Abstract

This application relates to the field of intelligent monitoring technology and discloses a method for training a static human detection model. The method involves acquiring several static heat source samples, performing differential processing on these samples to obtain differential samples, and then concatenating the infrared image of each frame in the static heat source samples with the corresponding differential image in the differential samples to obtain training samples. Using these training samples, a pre-set neural network is iteratively trained until the neural network converges, resulting in a static human detection model. The neural network includes an encoder and a decoder; the encoder includes a convolutional neural network, and the decoder includes a recurrent neural network. In this embodiment, the differential samples and static heat source samples are concatenated and used as a training set for training. This allows the trained static human detection model to detect static humans by analyzing features such as inter-frame temperature changes and positional changes, resulting in higher detection accuracy.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring technology, and in particular to a method for training a static human body detection model, a method for detecting lingering human presence, and a storage medium. Background Technology

[0002] China has officially entered an aging society, with home-based and institutional care being the mainstream models. However, safety hazards in the daily lives of the elderly are becoming increasingly apparent. For example, seniors are prone to accidental falls in bathrooms and difficulty getting up after prolonged sitting, and caregivers often fail to detect these abnormal behaviors in time, leading to delayed medical attention and serious consequences. With the continuous advancement of information technology, intelligent elderly care services have become a smart choice in the face of the increasingly serious aging problem. Real-time monitoring of abnormal risk behaviors of the elderly in indoor areas and timely notification of relevant alarm information to caregivers is a crucial step towards realizing intelligent elderly care.

[0003] Among these technologies, intelligent sensing can detect in real time whether elderly people are in target areas such as restrooms, and monitor whether they remain on the toilet for extended periods without getting up, are unable to get up, or experience falls. This allows for early detection of abnormal behavior risks and timely dispatch of alerts to caregivers, which is of significant value and importance in improving the intelligence level of elderly care in the new era. However, current monitoring methods mostly rely on sensors, such as wearable motion sensors or millimeter-wave Doppler radar, which are easily affected by environmental factors or usage, leading to inaccurate detection and a tendency for missed or false detections. Summary of the Invention

[0004] The main technical problem solved by the embodiments of this application is to provide a method for training a static human body detection model, a method for detecting lingering human bodies, and a storage medium. The trained static human body detection model can accurately detect static human bodies. Using the static human body detection model for human body lingering detection can accurately detect human body lingering, effectively avoiding the risks of missed detection and false detection caused by lingering human bodies.

[0005] Firstly, some embodiments of this application provide a method for training a static human detection model, including:

[0006] Several static heat source samples are obtained. The static heat source samples include k frames of infrared images with time sequence. Each static heat source sample is labeled with a real label. The real label reflects whether the static heat source in the static heat source sample belongs to a static human body heat source or an interfering heat source. Here, k is an integer greater than 1.

[0007] The static heat source sample is differentially processed to obtain differential samples. Then, the infrared image of each frame in the static heat source sample and the differential image corresponding to the infrared image in the differential sample are channel-stitched to obtain training samples. The differential samples include k differential images corresponding to k frames of infrared images.

[0008] Using several training samples, the pre-set neural network is iteratively trained until the neural network converges, resulting in a static human detection model.

[0009] The neural network includes an encoder and a decoder. The encoder includes a convolutional neural network, which is used to extract the features of static heat sources in the static heat source samples at the single-frame image level. The decoder includes a recurrent neural network, which is used to extract the features of static heat sources in the static heat source samples in the time dimension and output predicted labels.

[0010] In some embodiments, the aforementioned differential processing of the static heat source sample to obtain a differential sample includes:

[0011] The difference between each of the k frames of infrared images in the static heat source sample and the first frame of infrared image is calculated sequentially to obtain the k frames of difference images.

[0012] By setting the negative differences in the k-frame difference images to 0 and retaining the positive differences, a k-frame positive difference image is obtained; and by setting the positive differences in the k-frame difference images to 0 and retaining the negative differences, a k-frame negative difference image is obtained, wherein the negative difference refers to the pixel values ​​that are negative in the difference image, and the positive difference refers to the pixel values ​​that are positive in the difference image.

[0013] The k-frame positive difference images and k-frame negative difference images are stitched together channel by channel to obtain the difference samples.

[0014] In some embodiments, the convolutional neural network includes a first building layer for extracting shape and size features of a static heat source and a second building layer for extracting temperature distribution features of a static heat source.

[0015] The first building layer includes a convolutional layer and at least one first inverse residual module. After the convolutional layer, a batch normalization layer and an activation layer are configured. The first building layer is configured to expand the channels without reducing the resolution.

[0016] The second building layer comprises multiple cross-stacked first and second inverse residual modules. The second building layer is configured to reduce the resolution and expand the channels.

[0017] In some embodiments, the first inverse residual module includes a cascaded first pointwise convolutional layer, a first depthwise convolutional layer, and a second pointwise convolutional layer, each pointwise convolutional layer being followed by a batch normalization layer and an activation layer; wherein, the first pointwise convolutional layer is used for channel expansion, the first depthwise convolutional layer is used for feature extraction, and the second pointwise convolutional layer is used for channel compression.

[0018] In some embodiments, the second inverse residual module includes a cascaded third pointwise convolutional layer, a second depthwise convolutional layer, and a fourth pointwise convolutional layer, each pointwise convolutional layer being followed by a batch normalization layer and an activation layer; the input of the third pointwise convolutional layer is also skip-connected to the output of the last layer in the second inverse residual module; wherein, the third pointwise convolutional layer is used for channel expansion, the second depthwise convolutional layer is used for feature extraction, and the fourth pointwise convolutional layer is used for channel compression.

[0019] In some embodiments, the recurrent neural network includes at least one long short-term memory network and at least one fully connected layer;

[0020] Among them, at least one long short-term memory network is used to learn the temporal differences and correlations of static heat sources in the static heat source samples; at least one fully connected layer is used to map the features to a binary classification space.

[0021] Secondly, some embodiments of this application provide a method for detecting retention, including:

[0022] Infrared video is a time-series sequence of infrared images obtained by using an infrared camera to capture images of a target area.

[0023] Based on infrared video, determine whether there is a heat source in the target area;

[0024] If a heat source is present, determine whether the heat source has moved based on the infrared video.

[0025] If movement occurs, it is determined that a human body exists in the target area, and the duration of the human body's stay is accumulated;

[0026] If no movement occurs, the infrared video is differentially processed to obtain differential samples. Then, each frame of infrared image in the infrared video and the corresponding differential image in the differential samples are stitched together to obtain test samples. The differential samples include the differential images corresponding to each frame of infrared image.

[0027] The test samples are detected using a pre-trained static human detection model. If a human body is detected in the target area, the dwell time of the human body is accumulated. The static human detection model is trained using the method described in the first aspect.

[0028] If the duration of the stay is greater than or equal to the preset duration threshold, it is determined that the human body has been detained.

[0029] In some embodiments, the method further includes:

[0030] If the detection result of the static human body detection model is consistent with the first m detection results, then the detection result of the static human body detection model shall be taken as the final detection result; where the first m detection results are the detection results corresponding to the m infrared videos preceding the infrared videos.

[0031] If the detection result of the static human body detection model is inconsistent with the previous m detection results, then the previous detection result will be taken as the final detection result. The previous detection result is the detection result corresponding to the adjacent infrared video that precedes the infrared video.

[0032] Thirdly, some embodiments of this application provide an electronic device, characterized in that it includes:

[0033] At least one processor; and

[0034] A memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the method as described in the first aspect or the second aspect.

[0035] Fourthly, some embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the methods of the first or second aspect.

[0036] The method for training a static human detection model provided in some embodiments of this application involves acquiring several static heat source samples, performing differential processing on the static heat source samples to obtain differential samples, and then concatenating the infrared image of each frame in the static heat source samples with the corresponding differential image in the differential samples to obtain training samples. Using several training samples, a pre-set neural network is iteratively trained until the neural network converges to obtain the static human detection model. The static heat source samples include k temporally ordered infrared images (k is an integer greater than 1), and each static heat source sample is labeled with a real label reflecting whether the static heat source in the sample belongs to a static human heat source or an interfering heat source. The neural network includes an encoder and a decoder. The encoder includes a convolutional neural network, which is used to extract the features of the static heat source in the static heat source samples at the single-frame image level; the decoder includes a recurrent neural network, which is used to extract the features of the static heat source in the static heat source samples in the time dimension and output predicted labels.

[0037] In this embodiment, the differential samples and static heat source samples are concatenated and merged to form a training set for training. Since the differential samples reflect the inter-frame temperature and positional changes of the heat source, the neural network can learn features such as inter-frame temperature and positional changes. Based on the characteristics that interfering heat sources experience temperature changes but remain stationary in consecutive frames, while static human heat sources have relatively stable temperatures and may exhibit slight swaying in certain body parts (e.g., head and limbs), the network predicts the category of the static heat source by analyzing features such as inter-frame temperature and positional changes. Therefore, the trained static human detection model also possesses the ability to detect static humans by analyzing features such as inter-frame temperature and positional changes, resulting in higher detection accuracy. Furthermore, by setting up convolutional neural networks and recurrent neural networks, the neural network can extract the shape features, temperature distribution features, and inter-frame difference features of the static heat source from both single-frame and temporal dimensions. Classifying static heat sources based on these features helps improve the accuracy of the classification results. This accelerates neural network convergence, resulting in an accurate static human detection model.

[0038] The loitering detection method, electronic device, and storage medium provided in some embodiments of this application acquire infrared video and determine whether a heat source exists within a target area based on the infrared video. If a heat source exists, it is determined whether the heat source has moved based on the infrared video. If it has moved, a human body is determined to exist within the target area, and the loitering time of the human body is accumulated. If no movement has occurred, the infrared video is differentially processed to obtain differential samples, and each frame of the infrared image in the infrared video and the corresponding differential image in the differential samples are stitched together to obtain test samples. A pre-trained static human body detection model is then used to detect the test samples. If a human body is detected within the target area, the loitering time of the human body is accumulated. If the loitering time is greater than or equal to a preset time threshold, it is determined that the human body has loitered.

[0039] In this embodiment, by using infrared video to detect heat source movement and employing a static human body detection model to detect people, not only moving people but also those in static positions can be detected. Dwell time detection can be performed on both moving and static individuals, effectively solving the problem of inaccurate dwell time detection caused by static interference heat sources being easily misjudged as people or by the presence of static people being missed. Furthermore, combining motion detection and static human body detection models to detect and perform dwell time detection offers high robustness to interference from heat sources such as sunlight, hot water, and heated objects, resulting in high detection accuracy. It also effectively avoids user privacy exposure issues. Attached Figure Description

[0040] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0041] Figure 1 This is a schematic diagram of a system for training a static human detection model in some embodiments of this application;

[0042] Figure 2 This is a schematic diagram illustrating the acquisition of infrared video in some embodiments of this application;

[0043] Figure 3 This is a schematic diagram of infrared images in some embodiments of this application;

[0044] Figure 4 This is a schematic diagram of the structure of an electronic device in some embodiments of this application;

[0045] Figure 5 This is a flowchart illustrating the method for training a static human detection model in some embodiments of this application;

[0046] Figure 6 This is a schematic diagram of the neural network structure in some embodiments of this application;

[0047] Figure 7 This is a schematic diagram of the structure of the first inverse residual module in some embodiments of this application;

[0048] Figure 8 This is a schematic diagram of the structure of the second inverse residual module in some embodiments of this application;

[0049] Figure 9 This is a flowchart illustrating the retention detection method in some embodiments of this application. Detailed Implementation

[0050] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0053] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0054] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0055] To facilitate understanding of the methods provided in the embodiments of this application, the terms used in the embodiments of this application will first be introduced:

[0056] (1) Infrared thermal imaging technology

[0057] Infrared thermal imaging technology uses photoelectric technology to detect infrared signals in specific bands of thermal radiation emitted by objects, converting these signals into infrared images that can be discerned by human vision, and further calculating temperature values. The value of each pixel in the infrared image corresponds to the object's temperature in the world coordinate system. Thus, infrared thermal imaging technology allows humans to overcome visual barriers, enabling them to "see" the temperature distribution on an object's surface.

[0058] For example, an infrared camera is installed on the ceiling indoors to capture infrared video of people indoors. It can be understood that the infrared video includes multiple consecutive infrared images.

[0059] (2) Quantiles

[0060] For a string of numbers x1, x2, ..., xn with a quantity of N, n-1 x n The quantiles are defined as 0-100%, where the 0th quantile is the minimum value of the string of numbers, and the 100th quantile is the maximum value. The intermediate quantiles, such as the 50th quantile Q, are also defined. 50 This means that 50% of the numbers in this string are less than or equal to Q. 50Similarly, the 75th percentile Q 75 Then 75% of the numbers in this string are less than or equal to Q. 75 Understandably, Q 50 It doesn't necessarily have to be the middle number in the sequence of numbers from smallest to largest; it could be a number close to the middle. There are existing algorithms for calculating quantiles, which will not be elaborated here.

[0061] QL: Lower quartile, or 25th percentile, indicates that one-quarter of all values ​​are smaller than QL.

[0062] QU: Upper quartile, or 75th quartile, indicates that one-quarter of all values ​​are greater than QU.

[0063] IQR: Interquartile Range, which is QU-QL = 75th quantile - 25th quantile, encompassing half of all values.

[0064] Infrared thermal imaging equipment acquires infrared images at a certain frequency, and can capture and analyze the dynamic behavior of a target human body while ensuring privacy. Each pixel in the infrared image reflects the temperature of an object in the corresponding space. When monitoring the human body, it is necessary to detect the human body from the infrared image for monitoring purposes.

[0065] Before introducing the embodiments of this application, a brief introduction will be given to the relevant methods known to the inventors of this application, so as to facilitate the understanding of the embodiments of this application later.

[0066] In some solutions, the presence of a human body indoors is detected using millimeter-wave Doppler radar technology. First, the millimeter-wave radar emits an electromagnetic signal and acquires the echo signal reflected back from a target object in the space. Second, based on the echo signal, the distance and orientation of the reflected target object relative to the device are determined, and the power spectrum of the echo signal is determined based on this information. Finally, based on the power spectrum, the target echo signal reflected back from the human body is selected from the echo signal, and the presence of the human body in the target space is determined accordingly.

[0067] In this approach, millimeter-wave radar technology primarily relies on wave signals such as target movement and heartbeat to detect human presence. It is highly sensitive to disturbances in the environment, such as moving curtains, fans, or falling objects, which can easily be mistaken for human movement. This can lead to false alarms when a person leaves the area, indicating that the radar is still detecting someone. Furthermore, millimeter-wave radar is significantly affected by multipath effects, making it prone to missing targets outside the beam range. This means that if someone is present in the area, the radar may detect that no one is there, ultimately resulting in missed alarms due to the person remaining in the target area for an extended period.

[0068] In some solutions, wearable devices equipped with motion sensors are used to acquire data and monitor abnormal behavior in the elderly. This solution incorporates motion sensors within the wristband device to collect and transmit real-time movement data of the elderly person's body parts. By analyzing the motion sensor data, it determines whether the elderly person has been inactive for an extended period. If inactivity is detected, an alarm is sent to the caregiver.

[0069] In this proposed solution, firstly, wearable motion sensors are relatively inconvenient for the elderly, and they often forget to wear the devices, resulting in a lack of real-time monitoring. Secondly, monitoring the risk of prolonged stays by wearing motion sensor devices has significant vulnerabilities. For example, if an elderly person stays for an extended period but the body part wearing the device is still moving significantly, the risk of prolonged stays will not be detected. Finally, monitoring prolonged stays by wearing motion sensors cannot target specific areas, meaning it cannot monitor areas such as bathrooms. For instance, when an elderly person is resting in bed, it can easily trigger false alarms for prolonged stays.

[0070] To address the aforementioned issues, this application provides a loitering detection method, electronic device, and storage medium. By acquiring infrared video, the method determines whether a heat source exists within a target area. If a heat source exists, it determines whether the heat source has moved based on the infrared video. If movement occurs, a human body is identified within the target area, and the loitering time of the human body is accumulated. If no movement occurs, the infrared video undergoes differential processing to obtain differential samples. Each frame of the infrared video and its corresponding differential image in the differential samples are then stitched together to obtain test samples. A pre-trained static human body detection model is used to detect the test samples. If a human body is detected within the target area, the loitering time of the human body is accumulated. If the loitering time is greater than or equal to a preset time threshold, the human body is determined to have loitered.

[0071] In this embodiment, by using infrared video to detect heat source movement and employing a static human body detection model to detect people, not only moving people but also those in static positions can be detected. Dwell time detection can be performed on both moving and static individuals, effectively solving the problem of inaccurate dwell time detection caused by static interference heat sources being easily misjudged as people or by the presence of static people being missed. Furthermore, combining motion detection and static human body detection models to detect and perform dwell time detection offers high robustness to interference from heat sources such as sunlight, hot water, and heated objects, resulting in high detection accuracy. It also effectively avoids user privacy exposure issues.

[0072] Some embodiments of this application also provide a method for training a static human detection model. This method involves acquiring several static heat source samples, performing differential processing on the static heat source samples to obtain differential samples, and then concatenating the infrared image of each frame in the static heat source samples with the corresponding differential image in the differential samples to obtain training samples. Using these training samples, a pre-set neural network is iteratively trained until the neural network converges, resulting in a static human detection model. The static heat source samples include k temporally ordered infrared images (k being an integer greater than 1). Each static heat source sample is labeled with a real label, which reflects whether the static heat source in the sample belongs to a static human body heat source or an interfering heat source. The neural network includes an encoder and a decoder. The encoder includes a convolutional neural network (CNN) used to extract features of the static heat source in the static heat source samples at the single-frame image level. The decoder includes a recurrent neural network (RNN) used to extract features of the static heat source in the static heat source samples in the time dimension and output predicted labels.

[0073] In this embodiment, the differential samples and static heat source samples are concatenated and merged to form a training set for training. Since the differential samples reflect the inter-frame temperature and positional changes of the heat source, the neural network can learn features such as inter-frame temperature and positional changes. Based on the characteristics that interfering heat sources experience temperature changes but remain stationary in consecutive frames, while static human heat sources have relatively stable temperatures and may exhibit slight swaying in certain body parts (e.g., head and limbs), the network predicts the category of the static heat source by analyzing features such as inter-frame temperature and positional changes. Therefore, the trained static human detection model also possesses the ability to detect static humans by analyzing features such as inter-frame temperature and positional changes, resulting in higher detection accuracy. Furthermore, by setting up convolutional neural networks and recurrent neural networks, the neural network can extract the shape features, temperature distribution features, and inter-frame difference features of the static heat source from both single-frame and temporal dimensions. Classifying static heat sources based on these features helps improve the accuracy of the classification results. This accelerates neural network convergence, resulting in an accurate static human detection model.

[0074] The following describes exemplary applications of the electronic devices provided in this application for training static human detection models or for loitering detection. The electronic devices provided in this application can be servers, such as servers deployed in the cloud. Some embodiments of this application provide electronic devices that can be various types of terminals such as laptops, desktop computers, or mobile devices.

[0075] As an example, see Figure 1 , Figure 1This is a schematic diagram illustrating an application scenario of the loitering detection system provided in this application embodiment. Terminal 10 connects to server 20 via a network, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0076] Terminal 10 can be used to acquire training data and build neural networks. For example, those skilled in the art can download prepared training data to the terminal and build the network structure of the neural network. The training data includes several static heat source samples. It is understood that terminal 10 can also be used to acquire test samples. For example, an infrared camera can send the collected test samples to terminal 10, thereby allowing terminal 10 to acquire the test samples. In some embodiments, terminal 10 can be integrated with an infrared camera.

[0077] In some embodiments, terminal 10 locally executes the method for training a static human detection model provided in this application to train a pre-designed neural network using training data, determine the final model parameters, and then configure the neural network with the final model parameters to obtain a static human detection model. In some embodiments, terminal 10 may also send training data and a pre-constructed neural network stored on the terminal by those skilled in the art to server 20 via a network. Server 20 receives the training data and the neural network, trains the neural network using the training data, determines the final model parameters, and then sends the final model parameters to terminal 10. Terminal 10 saves the final model parameters, and then configures the neural network with the final model parameters to obtain a static human detection model.

[0078] The following describes the electronic equipment provided in the embodiments of this application for implementing the above-described method for training a static human body detection model and the loitering detection method. It is understood that the electronic equipment has computing power. The electronic equipment in the embodiments of this application can be a monitoring device for on-site monitoring, or a terminal or server that is communicatively connected to the monitoring device.

[0079] In some embodiments, the monitoring device is an infrared camera, such as... Figure 2 As shown, the infrared camera is installed on the ceiling corresponding to the target area indoors, at a height h from the ground, with the camera facing the target area. The infrared camera is communicatively connected to electronic devices.

[0080] Infrared cameras detect infrared energy (heat) non-contactly, convert it into electrical signals, and then generate infrared images that reflect the temperature of various objects within the target area. An infrared image is essentially a temperature data array, where the pixel value of each point represents the temperature of the object in that space.

[0081] Please see Figure 3In some embodiments, the infrared images captured by the infrared camera have a resolution of 24*32, and each acquisition outputs a 24*32 frame of infrared image. Based on the value of each pixel in the infrared image, which corresponds to the temperature value of an object in the world coordinate system, information such as the heat distribution range, approximate shape, and specific temperature value can be obtained from the infrared image. Figure 3 As shown in the image, the highlighted area in the upper left corner is the interfering heat source generated by the heated toilet, and the highlighted area in the lower right corner is the human body heat source generated by the elderly person in the bathroom.

[0082] In some embodiments, the infrared camera acquires images at a frequency of f frames per second (1 ≤ f ≤ 32) and buffers infrared images within a time frame of approximately t seconds (1 ≤ t ≤ 60), resulting in k = f * t frames of infrared images, which are then output as an infrared video. These k frames of infrared images are time-series images. The acquisition frequency f and buffering time t can be dynamically adjusted according to the device's computing power.

[0083] Please see Figure 4 The electronic device 100 includes a processor 101 and a memory 102 connected via a communication link. Here, the communication link can be established via a bus. Figure 4 The bus connection between China and Israel is illustrated by example. It is understood that... Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0084] Based on the communication connection between the electronic device and the infrared camera, when the infrared camera captures infrared video, it sends the infrared video to the electronic device. Thus, the processor is able to acquire the infrared video.

[0085] The processor 101 is configured to support the electronic device 100 in executing corresponding functions in the method for training a static human detection model or the method for detecting loitering. The processor 101 can be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0086] The memory 102, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for training a static human detection model or the loitering detection method in the embodiments of this application. The processor 101, by running the non-transitory software programs, instructions, and modules stored in the memory 102, can implement the method for training a static human detection model or the loitering detection method in any of the following method embodiments.

[0087] Memory 102 may include volatile memory (VM), such as random access memory (RAM); memory 1002 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 102 may also include combinations of the above types of memory.

[0088] It is understood that electronic devices also include other supporting hardware and software. Hardware may include antennas, various sensors, microphones, etc. Software may include operating systems, which are programs that manage and control the hardware and software resources of electronic devices. Software may also include various applications (apps). Other parts of the electronic device not involved in the embodiments of this application will not be described here.

[0089] It is worth noting that in some embodiments, the electronic device can also be integrated with an infrared camera as a monitoring device. In some embodiments, the electronic device can also be a terminal or server, etc. In this application embodiment, no limitation is made on the form of the electronic device, as long as it can acquire infrared image sequences and has computing processing capabilities.

[0090] As can be understood from the above, the loitering detection method provided in this application embodiment can be implemented by various types of electronic devices with processing capabilities, such as being executed by the processor of an electronic device or by other devices with computing capabilities. Other devices with computing capabilities may be smart terminals or servers that are communicatively connected to the electronic device.

[0091] As can be understood from the above, the method for training a static human detection model provided in this application embodiment can be implemented by various types of electronic devices with processing capabilities, such as by the processor of an electronic device or by other devices with computing capabilities. Other devices with computing capabilities may be smart terminals or servers that are communicatively connected to the electronic device.

[0092] The following describes the method for training a static human detection model provided in this application, using exemplary applications and implementations of the electronic device provided in the embodiments of this application. Please refer to... Figure 5 , Figure 5 This is a flowchart illustrating a method for training a static human detection model according to an embodiment of this application. It is understood that the execution entity of this training method can be one or more processors of an electronic device.

[0093] Please refer to it again. Figure 5 The method S100 may specifically include the following steps:

[0094] S10: Obtain several static heat source samples.

[0095] The static heat source samples consist of k-frame infrared images with a time sequence (k is an integer greater than 1). Each static heat source sample is labeled with a real label, which reflects whether the static heat source in the static heat source sample is a static human body heat source or an interfering heat source.

[0096] It is understood that a static heat source sample is a set of k infrared images arranged in chronological order, acquired by an infrared camera over a certain area. In some embodiments, k = f * t, where f is the acquisition frequency of the infrared camera and t is time (seconds). In some embodiments, 1 ≤ t ≤ 60, 1 ≤ f ≤ 32.

[0097] Among these several static heat source samples, some static heat source samples only include interfering heat sources, such as a basin of hot water, a heater, or a heated toilet, and do not include human bodies; other static heat source samples include interfering heat sources and static human bodies, where static human bodies refer to human bodies that have not moved significantly.

[0098] Each static heat source is labeled with a true label reflecting its category. In some embodiments, heat-coded labels can be used. For example, interfering heat sources are represented by 0, and static human body heat sources are represented by 1. Then, static heat source samples that only include interfering heat sources are labeled with 0, and static heat source samples that include static human body heat sources are labeled with 1.

[0099] In some embodiments, an infrared camera is installed at the center of the ceiling corresponding to the target area indoors, facing the target area below, with a height h from the ground of 2.0 to 3.0 meters. The infrared camera collects external video data within the target area. During data collection, personnel may enter the target area independently. Personnel may remain stationary or move within the target area, or they may create static interference heat sources, such as placing a basin of hot water, a hot towel, or a heater in the target area, and then leave after a period of time. Personnel can repeatedly enter and leave the target area during data collection, creating different static interference heat sources within the target area. In some embodiments, data collection can be conducted in real-world application scenarios, such as installing an infrared camera in the center of the bathroom ceiling to collect real-time data about the bathroom scene, with data collection and storage performed on a 24-hour cycle.

[0100] After acquiring and caching infrared video data for a certain duration (e.g., 60 seconds), the infrared camera sends the cached data to an electronic device (e.g., a computer or server), then clears the current cache and continues data acquisition and caching. In some embodiments, the duration of the infrared video data acquired within a single time period is much longer than the t seconds required for static heat source sampling. By observing the data frame numbers corresponding to when people enter and leave the target area in the infrared video data recordings, the presence of a human heat source in a certain segment of the infrared video data can be identified based on these data frame numbers. Based on the recorded data frame numbers corresponding to people entering and leaving the target area, data segments with people and data segments without people can be extracted from the infrared video data acquired within a single time period. These data segments with and without people are then divided into candidate samples of a fixed duration t seconds (i.e., k frames).

[0101] In some embodiments, candidate samples with human movement can be screened out by an algorithm that detects human movement, and candidate samples with human movement can be removed. Thus, the remaining valid candidate samples include static heat source samples (negative samples) containing only interfering heat sources and static heat source samples (positive samples) containing static human heat sources.

[0102] After acquiring several valid candidate samples, the electronic device can label these valid candidate samples manually or using labeling tools to obtain several static heat source samples. Each static heat source sample is labeled with a true label reflecting whether it is an interfering heat source or a static human body heat source.

[0103] It is understandable that in the above-described embodiments of data collection, the static heat source samples are obtained by simulating real-world home scenarios, such as fixed interfering heat sources or when a person stands or sits in one position for a long time, which can result in repetitive samples. To ensure the diversity of the training data as much as possible, in some embodiments, the samples are filtered and repetitive samples are discarded, thereby enabling the trained static human detection model to have better generalization ability.

[0104] Specifically, at least one selected sample is first chosen from the aforementioned static heat source samples, and the rest are used as candidate samples. The intermediate frame `imge2` of the candidate sample and the intermediate frame `imge1` of the selected sample are then compared to obtain a difference frame `diff_imge = imge2 - imge1`. A third temperature threshold is used to segment pixels in the difference frame that are not smaller than this third temperature threshold. If the area of ​​the largest connected region segmented reaches a set third area threshold, the candidate sample is considered a selected sample. If the area of ​​the largest connected region segmented does not reach the third area threshold, the current candidate sample is discarded.

[0105] In some embodiments, data augmentation processing is performed on several static heat source samples after discarding duplicate samples. For example, each frame of the infrared image of the static heat source sample is subjected to vertical mirror flipping, horizontal mirror flipping, or 180° rotation. In some embodiments, in order to increase the number of samples including static human body heat sources, the static heat source samples including static human body heat sources are further subjected to data augmentation processing. For example, with the centroid of the static human body heat source as a reference, the static human body heat source is translated, so that its centroid position will move accordingly.

[0106] In some embodiments, outlier handling is performed before the samples are input into the neural network. Abnormal pixels with null or excessively high or low temperature values ​​are processed. For the case where the temperature value corresponding to a pixel is null, the median of the temperature values ​​of all pixels in the current sample data is used for filling. For the case where the temperature value corresponding to a pixel is excessively high or low, the pixel is truncated by setting upper and lower thresholds for the temperature value.

[0107] Because ambient temperatures vary significantly across seasons, the temperature distribution data of the same heat source acquired by infrared cameras also differs considerably. To facilitate faster convergence during model training and ensure model robustness, the static heat source sample data is normalized, mapping each static heat source sample data to a fixed interval range.

[0108]

[0109] Where imges0 is the static heat source sample currently undergoing normalization, T min With T max These are the maximum and minimum temperatures in the static heat source sample imges0, respectively, while imges1 is the normalized static heat source sample.

[0110] Using several static heat source samples after the above-mentioned processes of discarding duplicate samples, data augmentation, outlier handling, and / or data augmentation as training data for the neural network not only allows the neural network to learn from a wealth of samples, but also accelerates neural network convergence and improves the robustness and generalization ability of the model.

[0111] S20: Perform differential processing on the static heat source sample to obtain differential samples, and stitch the infrared image of each frame in the static heat source sample and the differential image corresponding to the infrared image in the differential sample to obtain training samples. The differential samples include k differential images corresponding to k frames of infrared images.

[0112] It is understood that, in some embodiments, the static heat source sample here may be a static heat source sample after discarding duplicate samples, data augmentation, outlier handling, and / or data augmentation processing.

[0113] Differential processing refers to subtracting the pixel values ​​of two infrared images at their corresponding pixel points. Differential processing can be performed using the following formula:

[0114] D (i,j) =P (i,j) -H (i,j)

[0115] Among them, P (i,j) H is the pixel value of the pixel in the i-th row and j-th column of an infrared image in a static heat source sample. (i,j) D is the pixel value of the pixel in the i-th row and j-th column of another infrared image in the static heat source sample. (i,j) It is the pixel value (i.e., the difference value) of the pixel in the i-th row and j-th column of the difference image.

[0116] The static heat source sample is subjected to differential processing. For example, if the static heat source sample includes k frames of infrared images, each of these k frames of infrared images is differentially calculated with a specific frame of infrared image (such as a specific frame in the k frames of infrared images) to obtain k frames of differential images. Thus, these k frames of differential images constitute the differential sample.

[0117] It is understandable that in two infrared images of a static heat source sample, the pixel regions that do not overlap have larger difference values, while the pixel regions that overlap have difference values ​​close to 0. Therefore, the differential samples can reflect the inter-frame temperature and positional changes of the heat source.

[0118] The training samples are obtained by concatenating the channels of each frame of infrared image in the static heat source sample and the corresponding differential image in the differential sample. For example, k frames of infrared images and k frames of differential images correspond one-to-one according to their frame numbers. Concatenating one frame of infrared image with one frame of differential image results in one frame of 2-channel image. Thus, the training samples include k frames of 2-channel images.

[0119] It is understandable that if the number of channels in the difference samples is greater than or equal to 2, then the image in the training samples is a multi-channel image.

[0120] In some embodiments, the aforementioned step S20 specifically includes:

[0121] S21: Perform differential calculations on all k frames of infrared images in the static heat source sample with the first frame of infrared image to obtain k frames of differential images.

[0122] S22: Set the negative differences in the k-frame difference images to 0 and retain the positive differences to obtain k-frame positive difference images; and set the positive differences in the k-frame difference images to 0 and retain the negative differences to obtain k-frame negative difference images.

[0123] S23: Perform channel stitching on k frames of positive difference images and k frames of negative difference images to obtain difference samples.

[0124] In this embodiment, the static heat source sample is differentially calculated using the following formula:

[0125] diff_i=imge_i-imge_1,i=1,2,...,k-1,k

[0126] Where imge_i is the i-th (1≤i≤k) infrared image in the static heat source sample, and imge_1 is the first infrared image in the static heat source sample. diff_i is the difference image corresponding to the i-th infrared image. If the resolution of the infrared image imge_i is 24*32, then its corresponding difference image diff_i is a difference frame data of size 24*32.

[0127] Understandably, in the k-frame differential images, the difference in the first differential image is 0. Starting from the second differential image, each differential image records the change of the heat source relative to the first frame. This change includes temperature change and / or position change.

[0128] By setting the negative differences in the k-frame difference images to 0 and retaining the positive differences, we obtain the k-frame positive difference image D. 0+ By setting the positive differences in the k-frame difference images to 0 and retaining the negative differences, we obtain the k-frame negative difference image D. 0- Negative difference refers to the negative pixel values ​​(differences) in the difference image, while positive difference refers to the positive pixel values ​​(differences) in the difference image.

[0129] Then, the k positive difference images D 0+ and k negative difference images D 0- Perform channel stitching, such as in positive difference image DP. 0+ The data dimension is k×24×32, and the negative difference image DP 0- The data dimension is k×24×32, and the positive difference image is DP 0+ and negative difference image DP 0- Channel stitching yields a difference sample DP with dimensions k×24×32×2. That is, the difference sample includes k positive difference images and k negative difference images.

[0130] In this embodiment, by stitching the positive and negative difference images together, the difference samples include k positive and k negative difference images. This enables the subsequent neural network to learn features such as inter-frame temperature and position changes of the heat source by combining the positive and negative difference images.

[0131] Based on the characteristics that interfering heat sources undergo temperature changes but remain stationary across consecutive frames, while static human body heat sources exhibit relatively stable temperatures and may exhibit slight swaying in certain body parts (such as the head and limbs), the category of static heat sources is predicted by analyzing features such as inter-frame temperature and positional changes. Therefore, the trained static human body detection model also possesses the ability to detect static humans by analyzing features such as inter-frame temperature and positional changes, resulting in higher detection accuracy.

[0132] S30: Using several training samples, iteratively train the pre-set neural network until the neural network converges to obtain a static human detection model.

[0133] Several training samples are used as training data to train a pre-defined neural network. The network's parameters are continuously adjusted, and under the constraint of the loss function, the predicted labels output by the neural network become increasingly closer to the true labels. When the loss calculated by the loss function fluctuates within a certain range or reaches a certain value, the neural network converges. The parameters at convergence are then used as the model parameters, resulting in a static human detection model.

[0134] In this embodiment, the differential samples and static heat source samples are concatenated and merged to form a training set for training. Since the differential samples can reflect the inter-frame temperature and positional changes of the heat source, the neural network can learn features such as inter-frame temperature and positional changes. Based on the characteristics that interfering heat sources experience temperature changes but remain stationary in consecutive frames, while static human heat sources have relatively stable temperatures and may exhibit slight movement in certain body parts (e.g., head and limbs), the network predicts the category of static heat sources by analyzing features such as inter-frame temperature and positional changes. Therefore, the trained static human detection model also possesses the ability to detect static humans by analyzing features such as inter-frame temperature and positional changes, resulting in higher detection accuracy. Consequently, the trained static human detection model has the ability to accurately distinguish between static humans and interfering heat sources, accurately detecting still human bodies, effectively solving the difficult problems of easily misjudging interfering heat sources as people and missing detections due to the presence of static humans.

[0135] The neural network comprises an encoder and a decoder. The encoder includes a convolutional neural network (CNN), which extracts features of the static heat source in the static heat source samples at the single-frame image level. The decoder includes a recurrent neural network (RNN), which extracts features of the static heat source in the static heat source samples in the time dimension and outputs predicted labels.

[0136] As is understandable, Convolutional Neural Networks (CNNs) are a type of feedforward neural network characterized by local connectivity and weight sharing. The basic structure of a CNN generally includes an input layer, hidden layers, and an output layer. While layers are fully connected, nodes within each layer are unconnected. Therefore, feedforward neural networks do not consider the correlation between data points; the network's output is only related to the network's input at the current moment. Consequently, CNNs can extract features from each frame of image data in the training samples, learning the shape and temperature distribution characteristics of a static heat source within a single frame.

[0137] The basic structure of a Recurrent Neural Network (RNN) generally includes an input layer, hidden layers, and an output layer. In an RNN, the nodes in the hidden layers are connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the hidden layer from the previous time step. Therefore, the RNN memorizes previous information and applies it to the calculation of the current output. This gives the RNN its temporal nature and the ability to memorize temporal data. Thus, the RNN can learn the temporal characteristics and changes of a static heat source in the training samples, that is, the changes in the shape features and temperature distribution features of the static heat source over consecutive frames. Based on these feature changes, it outputs a predicted label. The predicted label is the predicted category of the static heat source in the training samples.

[0138] In this embodiment, by setting up a convolutional neural network and a recurrent neural network, the neural network can extract the shape features, temperature distribution features, and inter-frame difference features of the static heat source from both single-frame and temporal dimensions. Classifying the static heat source based on these features improves the accuracy of the classification results. This accelerates neural network convergence, resulting in an accurate static human detection model.

[0139] In some embodiments, the convolutional neural network includes a first building layer for extracting shape and size features of a static heat source and a second building layer for extracting temperature distribution features of the static heat source.

[0140] The first building layer includes a convolutional layer and at least one first inverse residual module. After the convolutional layer, a batch normalization layer and an activation layer are configured. The first building layer is configured to expand the channels without reducing the resolution.

[0141] Please see Figure 6 The convolutional layer (Conv 3*3, F=8, S=1) is followed by a batch normalization layer and an activation layer (Activation('ReLU6')). After the activation layer, a first residual module is connected. The input training samples are then sequentially processed through the convolutional layer for convolution, the batch normalization layer for normalization, the activation layer for activation, and the first inverse residual module (Bottleneck_1) for feature extraction before being input into the second construction layer. Figure 6 The example only uses one first residual module (Bottleneck_1) for illustration. It is understood that in other embodiments, the first building layer includes two or more first inverse residual modules.

[0142] Please refer to it again. Figure 6The training samples, consisting of k frames of images with a size of 24*32 and 3 channels, are used as input. The dimensions are (k, 24, 32, 3). The first dimension k is the number of frames in the training samples, the second dimension 24 and the third dimension 32 are the size of a single frame image, and the fourth dimension 3 is the number of channels in the image.

[0143] Figure 6 In the configuration parameter F, the number of filters output during convolution (i.e., the value after converting the number of channels in the fourth dimension) is specified. The configuration parameter S is the stride of the convolution kernel during convolution operations, and c is the number of filters output by the inverse residual module (i.e., the value after converting the number of channels in the fourth dimension).

[0144] In this embodiment, based on the principle of using training samples with lower resolution for user privacy, i.e. smaller second and third dimensions, the resolution of the sample data is not reduced in at least one first inverse residual module. Instead, the channels are expanded in the first convolutional layer, i.e., the number of channels is increased. This is beneficial for the neural network to learn basic features such as the shape and size of the heat source.

[0145] In some embodiments, the first inverse residual module includes a cascaded first pointwise convolutional layer, a first depthwise convolutional layer, and a second pointwise convolutional layer, each pointwise convolutional layer being followed by a batch normalization layer and an activation layer; wherein, the first pointwise convolutional layer is used for channel expansion, the first depthwise convolutional layer is used for feature extraction, and the second pointwise convolutional layer is used for channel compression.

[0146] Please see Figure 7 The input data is first expanded through a first pointwise convolutional layer (Conv 1*1), then connected to a first depthwise convolutional layer (DwConv 3*3) to extract features, and finally compressed through a second pointwise convolutional layer (Conv 1*1). This channel expansion followed by feature extraction and then compression significantly reduces the number of parameters and computational cost.

[0147] Each pointwise convolutional layer is followed by a batch normalization layer and an activation layer. The activation function used in the activation layers after the first pointwise convolutional layer and the first depthwise convolutional layer is ReLU6 = min(6, max(0, x)). This ReLU6 activation function sets values ​​less than 0 to 0, values ​​greater than 6 to 6, and values ​​between 0 and 6 remain unchanged. Using ReLU6 makes the model more robust to low-precision computation. The activation function used in the activation layer after the second pointwise convolutional layer is the linear function Linear = a·x, where the coefficient 'a' is typically 1. Using the linear activation function Linear here ensures that features in the low-dimensional space are not corrupted.

[0148] Figure 7 In this configuration, parameter F represents the number of output filters during convolution (i.e., the value converted from the fourth dimension channel number), parameter S represents the stride of the convolution kernel during convolution, c represents the number of output filters from the inverse residual module (i.e., the value converted from the fourth dimension channel number), and m is the expansion factor of the input data channel number c1. The first depthwise convolutional layer performs convolution mapping on each channel of the input data using a separate convolution kernel, with the kernel's stride s taking a value of 1 or 2. The number of input and output channels for the first inverse residual module is inconsistent.

[0149] Compared to residual network modules, which typically perform channel compression first through pointwise convolutional layers (Conv 1*1), then extract features through intermediate convolutional layers (Conv 3*3), and finally expand the number of channels through pointwise convolutional layers (Conv1*1), this embodiment first expands the channels of the input data, keeps the channels unchanged when extracting features using the first depth convolutional layer, and then compresses the channels after feature extraction. This approach can significantly reduce the number of parameters and computational load while effectively extracting features.

[0150] Please refer to it again. Figure 6 The second building layer includes multiple cross-stacked first and second inverse residual modules. The second building layer is configured to reduce the resolution and expand the channels.

[0151] It is understood that the second inverse residual module is a different inverse residual module from the first inverse residual module. In some embodiments, the second inverse residual module includes a cascaded third pointwise convolutional layer, a second depthwise convolutional layer, and a fourth pointwise convolutional layer, each pointwise convolutional layer being followed by a batch normalization layer and an activation layer; the input of the third pointwise convolutional layer is also skip-connected to the output of the last layer in the second inverse residual module; wherein, the third pointwise convolutional layer is used for channel expansion, the second depthwise convolutional layer is used for feature extraction, and the fourth pointwise convolutional layer is used for channel compression.

[0152] like Figure 8 As shown, the input data is first expanded through a third pointwise convolutional layer (Conv 1*1), then connected to a second deep convolutional layer (DwConv 3*3) to extract features, and finally compressed through a fourth pointwise convolutional layer (Conv 1*1). This channel expansion followed by feature extraction and then compression significantly reduces the number of parameters and computational cost. Furthermore, the batch normalization and activation layers configured after each pointwise convolutional layer in the second inverse residual module are the same as those in the first inverse residual module, and will not be repeated here.

[0153] Figure 8In the configuration, parameter F is the number of output filters during convolution (i.e., the value after converting the fourth dimension channel number), parameter S is the stride of the convolution kernel during convolution, c is the number of output filters from the inverse residual module (i.e., the value after converting the fourth dimension channel number), and m is the expansion factor of the input data channel number c. The second depthwise convolutional layer performs convolution mapping on each channel of the input data using a separate convolution kernel, with a stride s of 1. The second inverse residual module has the same number of input and output channels.

[0154] Furthermore, unlike the first inverse residual module, the input of the third pointwise convolutional layer is also skip-connected to the output of the last layer in the second inverse residual module. That is, the input data of the second inverse residual module is added to the output data of the last activation layer to output the final feature map. This allows the feature map output by the second inverse residual module to retain the features of the input data, reducing distortion during feature extraction from the training samples.

[0155] In this embodiment, by cross-stacking multiple first inverse residual modules and second inverse residual modules in the second construction layer, the resolution of the input data is gradually reduced and the number of channels is gradually expanded. This facilitates the neural network in learning the temperature distribution characteristics of static heat sources. In other words, it helps the neural network learn deeper features such as the temperature distribution differences between static human body heat sources and interfering heat sources.

[0156] In some embodiments, the recurrent neural network includes at least one long short-term memory network and at least one fully connected layer.

[0157] Among them, at least one long short-term memory network is used to learn the temporal differences and correlations of static heat sources in the static heat source samples; at least one fully connected layer is used to map the features to a binary classification space.

[0158] Please refer to it again. Figure 6 The recurrent neural network comprises two long short-term memory (LSTM) networks. LSTM networks are existing networks in the field of machine learning, and their structure will not be described in detail here. Those skilled in the art will understand that LSTM networks are a type of neural network used to process sequential data. Compared to general neural networks, they are capable of handling data with sequential variations. Therefore, in this embodiment, the main function of the LSTM network is to aggregate the hidden layer features of k frames in the training samples and learn the inter-frame difference and correlation features between time-series frames, such as the inter-frame difference characteristics and correlation features caused by slight body movements when a human is present, or the heating and cooling of interfering heat sources.

[0159] In some embodiments, please refer again Figure 6The last long short-term memory network is followed by two fully connected layers, with a dropout layer between them. The dropout layer randomly removes some neurons during training, which helps avoid overfitting and thus enhances the model's robustness. A softmax function is configured after the second fully connected layer.

[0160] The fully connected layers flatten the input feature map into a one-dimensional vector. The first fully connected layer generates a one-dimensional vector of length 32, which is used for feature weighting. The second fully connected layer generates a one-dimensional vector of length 2, which maps the training sample feature space to the static heat source binary classification label space. Finally, the softmax function outputs the probability that the static heat source in the target region is classified as a static human body heat source and the probability that it belongs to the interfering heat source.

[0161] In some embodiments, whether a static heat source in the target area is a human body heat source is determined by comparing the output probability value p1 with a preset threshold p0.

[0162]

[0163] Among them, presence is the human presence status value in the target area. Presence of 0 indicates that the static heat source in the target area is an interfering heat source, and presence of 1 indicates that the static heat source in the target area is a human heat source.

[0164] In this embodiment, by setting up a recurrent neural network including at least one long short-term memory network and at least one fully connected layer, on the one hand, it can help the neural network learn the inter-frame difference features and correlation features between time-series frames, and on the other hand, it can perform progressive dimensionality reduction on the data to avoid overfitting and increase the robustness of the model.

[0165] In some embodiments, the loss function used during training includes:

[0166]

[0167] Where Loss is the sum of losses, N is the number of training samples, and w i Let be the percentage of frames in the i-th training sample where a heat source was detected. Let y be the predicted label corresponding to the i-th training sample. i Let y be the true label corresponding to the i-th training sample. i The value can be 0 or 1. A value of 0 indicates that the static heat source in the training sample is an interfering heat source, and a value of 1 indicates that the training sample contains a human heat source.

[0168] In this embodiment, the proportion of frames in each training sample that detect heat sources is used as a weighting coefficient for the difference between the predicted label and the true label. On the one hand, this can effectively avoid the problem that the loss caused by difficult samples that are hard to learn during training is diluted by simple samples, resulting in the difficult samples not being fully learned. On the other hand, it can effectively avoid the interference caused by infrared images where heat sources cannot be segmented.

[0169] As can be understood from the above, the loitering detection method provided in this application embodiment can be implemented by various types of electronic devices with processing capabilities, such as being executed by the processor of an electronic device or by other devices with computing capabilities. Other devices with computing capabilities may be smart terminals or servers that are communicatively connected to the electronic device.

[0170] The following description, in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application, illustrates the loitering detection method provided in the embodiments of this application. Please refer to... Figure 9 , Figure 9 This is a schematic flowchart of the loitering detection method provided in the embodiments of this application. It can be understood that the execution subject of this loitering detection method can be one or more processors of an electronic device.

[0171] like Figure 9 As shown, method S200 may specifically include the following steps:

[0172] S201: Acquire infrared video.

[0173] The infrared video is a time-series sequence of infrared images captured by an infrared camera on a target area. It can be understood that this infrared video is collected by the infrared camera in the above embodiment capturing images of the target area at a certain frequency within t seconds. For example, if t is 5 seconds and the sampling frequency f is 20, then the infrared video contains a sequence of 100 infrared images arranged in chronological order of capture time. The target area is an area in the user's residence that needs to be monitored, such as a bathroom or living room.

[0174] S202: Determine whether there is a heat source in the target area based on infrared video.

[0175] It is understandable that each pixel in an infrared image reflects the temperature value of its corresponding area. If a heat source exists within the target area, the temperature value of the area where the heat source is located will be higher than the temperature value of the area without a heat source in the infrared image. Therefore, the presence of a heat source within the target area can be determined based on the temperature distribution in the infrared image.

[0176] Considering the limited information in a single frame of infrared image, there may be a few frames that fail to clearly reflect the heat source, making single-frame-based detection sporadic and prone to large errors. Therefore, using multiple infrared images from infrared video to determine whether a heat source exists in the target area can make heat source detection more accurate.

[0177] In some embodiments, the aforementioned step S202 specifically includes:

[0178] (1) Traverse the infrared video and determine the first temperature threshold corresponding to each frame of infrared image.

[0179] (2) For the pixels in the infrared image, the pixels with a temperature greater than or equal to the first temperature threshold are selected to form a hot spot region. If the maximum connected area of ​​the hot spot region is greater than or equal to the first area threshold, it is determined that there is a heat source in the infrared image.

[0180] (3) After the infrared video traversal is completed, if the number of infrared images with heat sources is greater than or equal to the first number threshold, then it is determined that there are heat sources in the target area.

[0181] It is understandable that the first temperature threshold is the critical temperature value used to determine whether a pixel belongs to a heat source. Each frame of an infrared image has its own corresponding first temperature threshold. That is, the first temperature threshold used for different infrared images in an infrared video is not exactly the same. It is understandable that the temperature in the target area is affected by the environment, the type of object, and the sensor itself; the temperature distribution in each infrared image is different. For example, the same scene (with the same temperature distribution) may have different temperature distributions in two infrared images. Using a uniform absolute threshold would result in unreasonable thresholds for some infrared images, affecting the extraction of hot spot regions. Therefore, in this embodiment, each frame of an infrared image has its own corresponding first temperature threshold, which is beneficial for subsequent accurate segmentation and extraction of hot spot regions.

[0182] In some embodiments, the temperature of each pixel in the infrared image can be statistically analyzed. For example, the temperatures of each pixel in the infrared image can be sorted from smallest to largest to form a temperature sequence. A first temperature threshold is calculated based on the temperature values ​​located at the upper quartile and the lower quartile of the temperature sequence. In some embodiments, the first temperature threshold can be calculated using the following formula:

[0183] threshold_1 = Q3 + α*(Q3 - Q1)

[0184] Where threshold_1 is the first temperature threshold, α is a parameter factor, for example, α∈[0.5,2.0]. Q1 is the lower quartile of the temperature value corresponding to each frame of infrared image; Q3 is the upper quartile of the temperature value corresponding to each frame of infrared image.

[0185] In this embodiment, a first temperature threshold is determined based on the temperature distribution of each pixel in the infrared image, so that the first temperature threshold can match the infrared image and accurately segment and extract the hot spot region.

[0186] The temperature value corresponding to each pixel in the infrared image is compared with a first temperature threshold. If the temperature value of a pixel is greater than or equal to the first temperature threshold, then that pixel is classified as a hot spot region. In essence, after filtering each pixel in the infrared image, the hot spot region is obtained.

[0187] Then, the maximum connected area of ​​the hotspot region is calculated, which is the number of pixels in the maximum connected area. The first area threshold is a critical value for determining whether the infrared image contains a heat source, and can exclude abnormal pixel blocks. In some embodiments, the first area threshold can be determined based on a small heat source, which can roughly distinguish between heat sources and abnormal pixel blocks.

[0188] Therefore, if the maximum connected area of ​​the hot spot region is greater than or equal to the first area threshold, abnormal pixel blocks can be effectively excluded, and heat sources can be distinguished. Thus, it can be determined that a heat source exists in the infrared image.

[0189] Understandably, considering the occasional errors in heat source detection using single-frame infrared images, infrared video is used for heat source detection here. That is, after the infrared video is traversed, if the number of infrared images containing heat sources in the infrared video is greater than or equal to a first quantity threshold M1, then it is determined that there is a heat source in the infrared video.

[0190] The first quantity threshold M1 can be determined based on the total number of infrared images in the infrared video. In some embodiments, the first quantity threshold M1 = β*k, where β takes the value of [0.4, 0.8] and k is the total number of infrared video frames.

[0191] In this embodiment, the above method can avoid the occasional errors of using single-frame infrared images for heat source detection, thus making the heat source detection accurate.

[0192] S203: If a heat source is present, determine whether the heat source has moved based on the infrared video.

[0193] If a heat source is present, it could be a static interference source or a human body. To further differentiate, infrared video is used to determine whether the heat source has moved. It's understood that if it has moved, it indicates the presence of a human body within the target area. If it hasn't moved, the heat source could be a static interference source or a static human body, meaning a human body may be present within the target area.

[0194] In some embodiments, the aforementioned step S203 specifically includes:

[0195] (1) Perform differential processing on the infrared video to obtain a differential sequence.

[0196] (2) Traverse the difference sequence and determine the second temperature threshold corresponding to each frame of the difference image.

[0197] (3) For the pixels in the differential image, select the pixels with a difference of not less than the second temperature threshold to form a moving region. If the maximum connected area of ​​the moving region is greater than or equal to the second area threshold, it is determined that the heat source in the differential image is moving.

[0198] (4) After the differential sequence is traversed, if the number of differential images in the differential sequence that indicate the movement of the heat source is greater than or equal to the second quantity threshold, then it is determined that the heat source has moved.

[0199] Differential processing involves subtracting the pixel values ​​of two images at their corresponding pixel positions. It's understandable that the difference values ​​for non-overlapping pixel regions in two infrared video images are larger, while the difference values ​​for overlapping pixel regions are close to zero. Since the position of static heat sources in each frame of an infrared video remains almost unchanged, differential calculation can eliminate static heat sources in the infrared video.

[0200] In some embodiments, starting from the second frame, the difference between each infrared image frame and the first infrared image frame is calculated sequentially to obtain a difference sequence including k-1 difference images.

[0201] It is understandable that the second temperature threshold is the critical temperature value used to determine whether a pixel in the difference image belongs to a non-overlapping pixel region. For any given frame of the difference image, there is its own second temperature threshold, threshold_2. In some embodiments, the second temperature threshold, threshold_2, can also be calculated using the formula threshold_2 = Q3′ + β*(Q3′ - Q1′). Here, β∈[0.5,2.0] is a parameter factor, Q1′ is the lower quartile of the temperature value corresponding to each frame of the difference image, and Q3′ is the upper quartile of the temperature value corresponding to each frame of the difference image.

[0202] Pixels with a difference of not less than the temperature threshold are selected to form a moving region. If the maximum connected area of ​​the moving region is greater than or equal to the second area threshold, it is determined that the heat source in the difference image has moved.

[0203] The temperature value corresponding to each pixel in the difference image is compared with a second temperature threshold, threshold_2. If the temperature value of a pixel is greater than or equal to the second temperature threshold_2, then that pixel is classified into the moving region. In essence, after filtering each pixel in the difference image, the moving region is obtained.

[0204] Then, the maximum connected area of ​​the moving region is calculated, which is the number of pixels in the maximum connected region. The second area threshold is a critical value for judging whether the heat source has moved in the difference image, and can exclude abnormal pixels.

[0205] Therefore, if the maximum connected area of ​​the moving region is greater than or equal to the second area threshold, it indicates that the heat source has moved. If the maximum connected area of ​​the moving region is less than the second area threshold, it indicates that the heat source has not moved.

[0206] Understandably, considering the temporal continuity of motion and the occasional errors in motion detection using single-frame difference images, a difference sequence including k-1 frames of difference images is used for motion detection. That is, after traversal, if the number of difference images in the difference sequence showing heat source movement is greater than or equal to a second threshold, it indicates that the motion is continuous, and the heat source is determined to have moved; otherwise, the heat source is determined not to have moved. The second threshold can be determined based on the motion characteristics; for example, in some embodiments, the second threshold can be 1 or 2.

[0207] In this embodiment, by performing differential calculation on the infrared video, the area of ​​the moving region in each differential image is calculated, and differential images with the maximum connected area of ​​the moving region being greater than or equal to a second area threshold are selected. Based on the number of selected differential images being greater than or equal to a second number threshold, it is possible to accurately determine whether the heat source has moved.

[0208] If the heat source moves, it indicates that the heat source is a human body, and a human body is present in the target area. If the heat source does not move, it may be a static interfering heat source or a static human body heat source, requiring further investigation.

[0209] S204: If movement occurs, determine that a human body exists in the target area and accumulate the duration of the human body's stay.

[0210] Understandably, if the heat source moves, it indicates the presence of a human body within the target area. After confirming the presence of a human body within the target area, the next infrared video is acquired for human body detection, and the dwell time of the human body is accumulated. In some embodiments, if a human body is detected in the current infrared video, the corresponding time t is accumulated into the dwell time; if no human body is detected in the next infrared video, the dwell time is reset to zero; if a human body is detected in the next infrared video, the time t is accumulated into the dwell time.

[0211] S205: If no movement occurs, perform differential processing on the infrared video to obtain differential samples, and stitch the infrared image of each frame in the infrared video and the differential image corresponding to the infrared image in the differential samples together to obtain test samples. The differential samples include the differential images corresponding to each frame of the infrared image.

[0212] The specific implementation process of step S205 is described in step S20 and will not be repeated here. Differential samples can reflect the inter-frame temperature and positional changes of the heat source. The test sample is obtained by stitching together the infrared image of each frame in the infrared video with its corresponding differential image in the differential samples.

[0213] S206: The test sample is detected using a pre-trained static human detection model. If a human body is detected in the target area, the dwell time of the human body is accumulated. The static human detection model is trained by the method of training a static human detection model in any of the above embodiments.

[0214] If the heat source does not move, it could be a static interference heat source or a static human body heat source. In this case, a pre-trained static human body detection model is used to detect the infrared video. If a human body is detected within the target area, the dwell time of the human body is accumulated. In some embodiments, if a human body is detected in the current infrared video, the corresponding time t is accumulated into the dwell time. If no human body is detected in the next infrared video, the dwell time is reset to zero. If a human body is detected in the next infrared video, the time t is accumulated into the dwell time.

[0215] The static human detection model is obtained by training a neural network with a large number of training samples. In other words, by training the neural network with a large number of rich training samples, the neural network can learn the single-frame features and time-domain features of static human heat sources and interfering heat sources. As a result, the trained static human detection model has the ability to accurately distinguish between static human bodies and interfering heat sources, and accurately detect human bodies that have not moved.

[0216] S207: If the duration of stay is greater than or equal to the preset duration threshold, then it is determined that the human body has been detained.

[0217] By comparing the duration of human stay in the target area (stay_time) with the pre-set time threshold (stay_time_threshold), the risk of abnormal human stay behavior can be accurately determined, and an alarm message for excessive human stay in the target area can be reported at the current time, thus meeting the care needs.

[0218] In some embodiments, the following formula is used to determine whether abnormal behavior of lingering has occurred;

[0219]

[0220] The alarm value indicates whether a person is currently loitering in the target area, i.e., whether a loitering alarm should be reported to the end user. If the alarm value is 1, it means that a person is currently loitering in the target area and an alarm should be triggered; if the alarm value is 0, it means that a person is currently not loitering in the target area and no alarm should be triggered.

[0221] As can be seen from the above, in some embodiments, by performing heat source movement detection on infrared video and using a static human body detection model to detect human bodies, not only moving human bodies can be detected, but also those in static positions. Dwell time detection can be performed on both moving and static human bodies, thus effectively solving the problem of inaccurate dwell time detection caused by the easy misjudgment of static heat sources as people or the missed detection of static human bodies. Furthermore, combining movement detection and static human body detection models to detect human bodies and perform dwell time detection has high robustness to interference caused by static heat sources such as sunlight, hot water, and heated objects, resulting in high detection accuracy. On the other hand, it effectively avoids user privacy exposure issues.

[0222] In some embodiments, to reduce the impact of the detection error of the static human detection model on subsequent loitering detection, the detection results output by the static human detection model are validated. Specifically, the detection results corresponding to m infrared videos located between the current infrared videos are taken as reference objects to determine the final detection result of the static human detection model.

[0223] It is understandable that static heat sources exhibit continuity over a short period. By using m detection results from the current moment, the final detection result of the static human body detection model is determined, resulting in more stable and accurate detection results.

[0224] In some embodiments, the method S200 further includes:

[0225] S208: If the detection result of the static human body detection model is consistent with the first m detection results, then the detection result of the static human body detection model shall be taken as the final detection result; wherein, the first m detection results are the detection results corresponding to the m infrared videos preceding the infrared videos.

[0226] S209: If the detection result of the static human body detection model is inconsistent with the previous m detection results, the previous detection result shall be taken as the final detection result. The previous detection result is the detection result of the adjacent infrared video that precedes the infrared video.

[0227] It is understandable that at least one of the first m detection results could be determined by detecting whether the heat source has moved. If the first m detection results show no human body, the static human body detection model's result is consistent with the first m results, indicating that the detection result is stable and reflects reality. If the first m detection results show a human body, the static human body detection model's result is consistent with the first m results, indicating that the human body is indeed stationary, and the detection result is stable and accurate. Therefore, if the static human body detection model's result is consistent with the first m results, then the static human body detection model's result is taken as the final detection result.

[0228] If the first m detection results show no human body, the static human body detection model's results are inconsistent with the first m results, indicating that the static human body detection module may misidentify static interference heat sources as human bodies, resulting in false detections. Therefore, the previous detection result is used as the final detection result, meaning the final detection result is a continuation of the previous detection results, making the final detection result more accurate.

[0229] If the static human detection model detects the presence of a human body in the first m detection results, the results are inconsistent with those of the first m detection results. This indicates that the static human detection module may misidentify a static human body as a static heat source, resulting in false detections. Therefore, the previous detection result is used as the final detection result, meaning the final detection result is a continuation of the previous detection results, making the final detection result more accurate.

[0230] In this embodiment, the above-mentioned calibration method makes the final detection results more accurate, avoids frequent fluctuations in the detection results, and reduces the possibility of missed detections or underreporting due to the human body lingering in the target area.

[0231] In summary, the loitering detection method provided in some embodiments of this application, by detecting heat source movement in infrared video and using a static human body detection model to detect human bodies, can detect not only moving human bodies but also those in static positions. It can perform loitering time detection on both moving and static human bodies, thus effectively solving the problem of inaccurate loitering detection caused by the easy misjudgment of static interference heat sources as people or the missed detection of static human bodies. Furthermore, combining movement detection and static human body detection models to detect human bodies and perform loitering detection has high robustness to interference caused by static heat sources such as sunlight, hot water, and heated objects, resulting in high detection accuracy. It also effectively avoids user privacy exposure issues.

[0232] This application also provides a computer-readable storage medium, such as a memory including program code, which can be executed by a processor to complete the method for training a static human detection model or the persistence detection method in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0233] This application also provides a computer program product comprising one or more lines of program code stored in a computer-readable storage medium. A processor of an electronic device reads the program code from the computer-readable storage medium and executes the program code to complete the steps of the method for training a static human detection model or the steps of the dwell detection method provided in the above embodiments.

[0234] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0235] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a static human detection model, characterized in that, include: Acquire several static heat source samples, each of which includes k frames of infrared images with temporal sequence. Each static heat source sample is labeled with a real label, which reflects whether the static heat source in the static heat source sample is a static human body heat source or an interfering heat source, where k is an integer greater than 1. The static heat source sample is subjected to differential processing to obtain differential samples. Then, each frame of infrared image in the static heat source sample and the corresponding differential image in the differential sample are channel-stitched to obtain training samples. The differential samples include k frames of differential images corresponding to k frames of infrared images. Using several training samples, the pre-set neural network is iteratively trained until the neural network converges to obtain the static human detection model. The neural network includes an encoder and a decoder. The encoder includes a convolutional neural network, which is used to extract the features of the static heat source in the static heat source sample at the single-frame image level. The decoder includes a recurrent neural network, which is used to extract the features of the static heat source in the static heat source sample in the time dimension and output a predicted label. The differential processing of the static heat source sample to obtain a differential sample includes: The k frames of infrared images in the static heat source sample are successively compared with the first frame of infrared image to obtain k frames of difference images; The negative differences in the k-frame difference images are set to 0, and the positive differences are retained to obtain k-frame positive difference images; and the positive differences in the k-frame difference images are set to 0, and the negative differences are retained to obtain k-frame negative difference images, wherein the negative differences are the pixel values ​​that are negative in the difference images, and the positive differences are the pixel values ​​that are positive in the difference images. The k-frame positive difference image and the k-frame negative difference image are respectively stitched together by channel to obtain the difference sample.

2. The method according to claim 1, characterized in that, The convolutional neural network includes a first construction layer for extracting the shape and size features of the static heat source and a second construction layer for extracting the temperature distribution features of the static heat source; The first building layer includes a convolutional layer and at least one first inverse residual module. The convolutional layer is followed by a batch normalization layer and an activation layer. The first building layer is configured to expand the channels without reducing the resolution. The second building layer includes multiple first inverse residual modules and second inverse residual modules arranged in a cross-stack configuration. The second building layer is configured to reduce the resolution and expand the channels.

3. The method according to claim 2, characterized in that, The first inverse residual module includes a cascaded first pointwise convolutional layer, a first depthwise convolutional layer, and a second pointwise convolutional layer. Each pointwise convolutional layer is followed by a batch normalization layer and an activation layer. The first pointwise convolutional layer is used for channel expansion, the first depthwise convolutional layer is used for feature extraction, and the second pointwise convolutional layer is used for channel compression.

4. The method according to claim 3, characterized in that, The second inverse residual module includes a cascaded third pointwise convolutional layer, a second depthwise convolutional layer, and a fourth pointwise convolutional layer. Each pointwise convolutional layer is followed by a batch normalization layer and an activation layer. The input of the third pointwise convolutional layer is also skipped to the output of the last layer in the second inverse residual module. The third pointwise convolutional layer is used for channel expansion, the second depthwise convolutional layer is used for feature extraction, and the fourth pointwise convolutional layer is used for channel compression.

5. The method according to claim 1, characterized in that, The recurrent neural network includes at least one long short-term memory network and at least one fully connected layer; The at least one long short-term memory network is used to learn the temporal differences and correlations of static heat sources in the static heat source samples; the at least one fully connected layer is used to map the features to a binary classification space.

6. A method for detecting retention, characterized in that, include: Acquire infrared video, which is a time-series sequence of infrared images obtained by using an infrared camera to capture images of a target area; Based on the infrared video, determine whether there is a heat source in the target area; If a heat source is present, determine whether the heat source has moved based on the infrared video. If movement occurs, it is determined that a human body exists in the target area, and the duration of the human body's stay is accumulated; If no movement occurs, the infrared video is differentially processed to obtain differential samples, and each frame of infrared image in the infrared video and the corresponding differential image in the differential samples are channel-stitched to obtain test samples, wherein the differential samples include differential images corresponding to each frame of infrared image. The test sample is detected using a pre-trained static human detection model. If a human body is detected in the target area, the dwell time of the human body is accumulated. The static human detection model is trained using the method described in any one of claims 1-5. If the duration of the stay is greater than or equal to a preset duration threshold, then it is determined that the human body has been detained.

7. The method according to claim 6, characterized in that, The method further includes: If the detection result of the static human body detection model is consistent with the first m detection results, then the detection result of the static human body detection model shall be taken as the final detection result; wherein, the first m detection results are the detection results corresponding to the m infrared videos preceding the infrared video. If the detection result of the static human body detection model is inconsistent with the previous m detection results, then the previous detection result shall be taken as the final detection result, wherein the previous detection result is the detection result corresponding to the adjacent infrared video located before the infrared video.

8. An electronic device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the method as described in any one of claims 1-7.

Images