Method and system for detecting, counting, and tracking using low resolution infrared devices

Abstract

Methods and systems for detecting, counting, and tracking organisms using low resolution infrared devices are provided. The detection method provided by the invention comprises the following steps: S1, carrying out background removal on each obtained low-resolution infrared frame according to a preset rule, and obtaining an infrared frame in which pixels of a target object are reserved; s2, carrying out averaging processing on the infrared frame; s3, improving the resolution of the infrared frame; s4, extracting a foreground object; and S5, comparing the pixel number or proportion of the foreground object with a preset value, and determining that the target object is detected. According to the method, the low-resolution infrared equipment is used, and the infrared frame is processed, so that the low-resolution infrared detection equipment with lower cost is used, the problems of low precision and low accuracy caused by the low-resolution infrared equipment are solved by utilizing a proper algorithm, the detection effects of low cost and high precision can be achieved, and the method is suitable for popularization and application. And the problem of privacy exposure of the user is avoided.

Description

Technical Field

[0001] The present invention relates generally to detection devices and, more particularly, to methods and systems for detecting, counting and tracking using low resolution infrared devices. Background Art

[0002] Technologies for detecting, counting and / or tracking living things, especially humans, can provide valuable information for a variety of applications. Examples include energy conservation in commercial buildings (activating different levels of cooling, ventilation, etc. depending on the number of people in the building), crowd flow control (such as preventing trampling caused by excessive traffic), and physical health monitoring (such as bedside monitoring, fall detection). There are currently a variety of methods to achieve this, which usually use cameras to record videos or images. However, due to privacy issues, camera-based methods are unacceptable in some applications, such as safety monitoring of the elderly at home or in senior centers. Therefore, it is necessary to study devices and methods that do not rely on cameras to achieve accurate biological detection, counting and tracking functions. Summary of the invention

[0003] The present invention aims to detect target objects (such as living things) using low-resolution infrared devices, and in particular to provide accurate detection, counting and tracking functions of living things (such as humans), especially in situations involving privacy issues (such as homes and healthcare centers).

[0004] The detection method using a low-resolution infrared device provided by the present invention comprises the following steps:

[0005] S1: For each acquired low-resolution infrared frame, background removal is performed according to a preset rule to obtain an infrared frame that retains the pixels of the target object;

[0006] S2: average the infrared frames after background removal;

[0007] S3: Improve the resolution of the infrared frame after averaging;

[0008] S4: extract foreground objects;

[0009] S5: Compare the number of pixels or the proportion of the foreground object with a preset value. If the number of pixels or the proportion of the foreground object exceeds the preset value, it is confirmed that the target object is detected.

[0010] The present invention uses a low-resolution infrared device and performs the above-mentioned processing on the infrared frame, thereby using a low-cost low-resolution infrared detection device and overcoming the problems of insufficient precision and low accuracy brought by the low-resolution infrared device by using a suitable algorithm, so that the system can have a low-cost, high-precision detection effect and will not cause the problem of user privacy exposure.

[0011] Based on the above method, the present invention further provides a method for counting and tracking target objects using a low-resolution infrared device. The counting method includes the above detection method, and further includes:

[0012] S6: Find the seed points using fast radial symmetry in the infrared frame and obtain the number of target objects.

[0013] Tracking methods include the counting methods mentioned above, and also include:

[0014] S7: Analyze the position, direction and speed of the seed points of the infrared frames recorded by different devices, and regard the seed points with similar directions and speeds as the same target object for tracking.

[0015] The present invention also provides a system for detecting, counting and tracking using the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the arrangement of infrared devices used in the embodiments provided by the present invention.

[0017] Figure 2 It is the result of the infrared frame in the embodiment provided by the present invention being processed through different steps.

[0018] Figure 3 It shows the influence of different values ​​of four parameters on the calculation of seed points for fast radial symmetry in the embodiment provided by the present invention.

[0019] Figure 4 The result after multiple corrosion iterations are performed on the infrared frame in the embodiment provided by the present invention is shown.

[0020] Figure 5 An example of an original image of an infrared frame after preprocessing (left side) and after processing in step S6 (right side) is shown.

[0021] Figure 6 Another example of an original image of an infrared frame after preprocessing (left side) and after processing in step S6 (right side) is shown.

[0022] Figure 7 An example of an original image corresponding to the infrared frame (upper side) and an infrared frame processed in step S6 (lower side) is shown.

[0023] Figure 8 A schematic diagram of the infrared frame of the target object tracking system is shown. DETAILED DESCRIPTION

[0024] Example 1

[0025] Embodiment 1 of the present invention provides a system for detecting, counting and tracking using low-resolution infrared devices. The system can be used to detect, count and track target objects. Since the system uses infrared devices, the target object should have a certain difference from the environment in infrared radiation. In this embodiment, the target object is a living thing, such as a human being. The system can provide accurate biological detection, counting and tracking, especially in situations where there are privacy issues (such as at home and in healthcare centers). Therefore, the system can be applied to homes and healthcare centers and commercial buildings.

[0026] For example, in homes and healthcare centers, the system can monitor the bedside to detect the occupancy of the elderly's beds, preventing them (especially those with dementia who cannot return to bed after leaving home) from leaving their beds without the knowledge of the caregivers; conduct safety monitoring such as human falls, and promptly notify caregivers of falls; and alarm when abnormally high temperatures appear in the kitchen or other areas of the home. The system can also be used in buildings, such as adjusting the air conditioning power in the building according to the number of people in the room; and better utilizing facilities (including conference rooms) by understanding the occupancy of the room (whether there are people in it).

[0027] In this embodiment, the system includes an infrared device and a processor. The infrared device and the processor are connected via wired or wireless communication, and the image acquired by the infrared device is transmitted to the processor.

[0028] In this embodiment, the system uses multiple low-resolution infrared devices, such as infrared sensors. The infrared sensor can obtain infrared rays in the environment and output them in the form of image frames according to the intensity of the infrared rays. Compared with camera-based devices, the use of low-resolution infrared devices can protect privacy and reduce costs. Compared with high-resolution infrared devices, low-resolution infrared devices are less expensive.

[0029] The low-resolution infrared devices used in this system usually refer to those with a resolution lower than 50x50=2500 pixels. Each infrared device can cover a specific viewing angle. Figure 1 As shown, the infrared devices of this system (such as infrared sensors, Figure 1 The infrared sensors (using a circle as an example) can be arranged on the ceiling or wall, with the detection direction facing the room. The detection ranges of the infrared sensors can overlap to a certain extent to ensure complete coverage of the detection area. In this embodiment, each infrared sensor is 8×8 infrared pixels (64 resolution) and detects an area of ​​3.6 meters × 3.6 meters. Each infrared pixel represents the infrared intensity of an area of ​​approximately 45 centimeters × 45 centimeters. In other applications, infrared sensors with different resolutions can be used according to the detection environment and purpose.

[0030] like Figure 2 As shown, the system first preprocesses the image obtained by the low-resolution infrared device. The preprocessing steps include:

[0031] S1: Background removal

[0032] This step is to obtain the infrared frame ( Figure 2 (a)) in the figure is processed, and some of the pixels are set as background according to the preset rules to preliminarily distinguish them from the pixels of the target object.

[0033] This step includes: collecting infrared frames from each infrared device and averaging the pixel values ​​in each infrared frame. In a single infrared frame, if the pixel value of a certain pixel is within a preset range around the average value, the pixel is set as the background, and the pixel value of the pixel used as the background is removed to 0, and the non-background pixel value is retained. The processed image is shown in Figure 2 As shown in (b) in .

[0034] If there are X pixels in a certain infrared frame, the pixel values ​​of each pixel are A 1 , A 2 , A 3 ,...,A X, First, the system calculates the average value A a The system can preset the first threshold B, and a ±B is set as the range of background pixels. That is, the system sets the pixel value within A a The pixels within ±B are set to background pixel values, and the pixel values ​​of background pixels are set to 0, and they are not processed in the subsequent processing. a The value in the range of ±B is set as the value to be retained. In other embodiments, two thresholds may also be set, such as B 1 and B 2 , as the upper and lower limits of the background pixel value, respectively. At this time, the range of background pixels is A a +B 1 To A a -B 2 .

[0035] Generally speaking, the pixel value of the area where the target object is located will be higher than the pixel value of the area where the surrounding environment is located. Therefore, through the above steps, the system can preliminarily distinguish the pixels of the target object from the pixels of the surrounding environment, and the above method is simple and easy to implement, and consumes little calculation. The low-resolution infrared device used in this embodiment has lower fluctuation stability and accuracy of the detection value during detection than the high-resolution infrared device. However, in the infrared frame acquired by the low-resolution infrared device at a single time, the fluctuation range and accuracy of the pixel value of each pixel point are consistent. Therefore, this method can avoid the above defects of the low-resolution infrared device by preprocessing a single infrared frame using the above method, and can better distinguish the target object from the environment. The above method can also preset different thresholds according to different application environments and target objects to improve the integrity and accuracy of the target object separation.

[0036] In other embodiments, in addition to using the above method for calculating the average value, other methods may be used, such as the standard deviation method. By calculating the average value and standard deviation of the pixel values, the values ​​that are more than a multiple of the preset standard deviation from the average value are set as non-background pixel values ​​and retained; and the values ​​within the preset standard deviation range are set as background pixel values ​​and removed.

[0037] S2: Minimize temperature detection error

[0038] This step includes: continuing the average processing of the infrared frames from which the background has been removed in step S1. The image after the average processing is as follows Figure 2 As shown in (c) in .

[0039] Specifically, the pixel values ​​of the infrared frame from which the background has been removed (ie, the non-background pixel values ​​after the S1 process) are averaged again.

[0040] S3: Improved resolution

[0041] Since this system uses low-resolution infrared equipment, in order to improve accuracy, this system further processes the value obtained by S2. The processed image is as follows Figure 2 The specific steps include:

[0042] S3.1: Interpolate the infrared frames processed by S2 to obtain an image with higher resolution and accuracy, so as to obtain a more accurate heat source location. For example, after the resolution is increased, the pixel value of the newly added pixel is taken as the average value of the pixel values ​​of the surrounding pixels. In other embodiments, various types of interpolation methods that are currently more mature can also be applied, such as Nearest Neighbor Interpolation, Bilinear Interpolation, Bicubic Interpolation, Lanczos Interpolation, Gaussian Interpolation, etc.

[0043] S3.2: filtering the interpolated infrared frame to make the image smoother.

[0044] In this embodiment, Gaussian filtering is used to process the interpolated infrared frame to make the image smoother. In other embodiments, other filtering methods can also be used for processing, such as median filtering, mean filtering (MeanFilter), bilateral filtering (BilateralFilter), non-local mean filtering (Non-local MeansFilter), etc.

[0045] Different interpolation and filtering methods can be selected according to different application environments and characteristics of target objects to obtain a smoother image.

[0046] S4: Extract foreground objects

[0047] Through this step, the system can extract the target object (i.e. the foreground object) and complete the image preprocessing. Figure 2 As shown in (e) in .

[0048] In the present embodiment, threshold processing is performed by Otsu binarization algorithm to extract foreground. Since Otsu binarization algorithm is maturely applied in the field of image processing, its specific application mode is not described in detail here. By adopting Otsu binarization algorithm to perform threshold processing, the best global threshold can be found by maximizing the inter-class variance, thereby realizing the binarization segmentation of the image. Specifically, the algorithm divides the gray value of the image into two parts, the foreground and the background, by calculating the inter-class variance under different thresholds, and selecting the threshold that makes the inter-class variance maximum as the segmentation point. This method is simple to calculate and is not affected by the brightness and contrast of the image. In addition to adopting Otsu binarization algorithm, in other embodiments, other threshold setting methods can also be used for processing, such as the average method or the bimodal method, etc. Or by artificially setting the threshold in advance for further processing.

[0049] like Figure 2 As shown, the above steps S1-S4 can be called preprocessing. Through preprocessing, the system can obtain a foreground extraction image with a smoother image and higher precision from the infrared frame obtained from the low-resolution infrared device.

[0050] After preprocessing, the system can further perform subsequent processing on the foreground extracted image, including target object detection, counting and tracking.

[0051] S5: Object Detection

[0052] This step mainly uses the image processed by S4 to detect the foreground extraction object.

[0053] In this embodiment, the target object is a living thing, such as a human being. The biological detection is mainly performed by analyzing the number of pixels of the foreground extraction object. If the number of pixels of the foreground extraction object is greater than the predefined number of pixels, it means that the foreground object is large enough to be considered as the target object, and the target object can be considered to be detected. If the number of pixels of the foreground extraction object is less than the predefined number of pixels, it means that the foreground extraction object is not the target object. In this embodiment, the number of pixels of the foreground extraction object can usually be set to 20% of the image area.

[0054] In other embodiments, the target object may also be changed according to different requirements, such as detecting a fire source, etc. If the target object is different, the preset parameters in this step also need to be adjusted accordingly.

[0055] The target object can be detected by processing a single pre-processed infrared frame obtained by a single infrared device to obtain the target object within the range corresponding to the single infrared device.

[0056] S6: Target object count

[0057] Based on S5, the system can confirm the number of target objects in the target area by counting, and then make corresponding settings. For example, the number of people in the target area in the building is confirmed, and the air conditioning power is adjusted. In this embodiment, the target objects are organisms, so the main thing is the number of organisms in the target area.

[0058] Specifically, in this example, the seed point can be found using Fast Radial Symmetry (FRS), and each seed point in the foreground area represents a biological object.

[0059] Fast Radial Symmetry (FRS) is an image processing technique that detects highly radially symmetrical points by analyzing the neighborhood around each pixel in the image. It is used to detect symmetric areas in an image and is particularly suitable for identifying objects or patterns with radial symmetry features, such as faces, flowers, and the heads of certain animals. Because it only focuses on information on radial paths rather than the entire image, this makes FRS computationally efficient and very useful in real-time applications, such as real-time detection and tracking of faces in video streams. The basic idea of ​​FRS is to exploit the symmetry of objects in an image and identify symmetry by calculating the change in pixel values ​​on radial lines (or called radial paths) from the center of the image to the edge. The factors that affect the results of FRS mainly include:

[0060] Radius value: The size of the radius directly affects the detection range. A smaller radius may not capture the complete symmetry, while a larger radius may contain more noise or irrelevant information.

[0061] Gaussian kernel size: The Gaussian kernel is used to smooth the image and avoid noise interference during the derivation process. The size of the kernel affects the degree of smoothing and thus the accuracy of symmetry detection.

[0062] Radial strictness parameter: This parameter determines how strict the symmetry check is. A higher strictness value may exclude some areas that actually have symmetry, while a lower value may misinterpret noise as symmetry.

[0063] Normalization factor: Normalization is used to balance the brightness differences in different areas. Different settings of the normalization factor will affect the final symmetry detection results.

[0064] refer to Figure 3 ,In actual applications, the applicant found that among all parameters, the radius value has a greater ,influence on the accuracy of finding the seed point, while the other ,parameters basically do not change the position of the seed point and basically do not ,affect the accuracy of the result. Figure 3 The effect of different values ​​of the four parameters on the fast radial symmetry on the calculated seed point is shown. (a) is the original result. The subsequent upper and lower images represent the image processing results when the following parameters take smaller and larger values, respectively: (b) radius value; (c) Gaussian kernel size; (d) radial strictness value; (e) normalization factor value.

[0065] like Figure 4 As shown, in order to confirm the appropriate radius value, in this embodiment, the system uses erosion iteration in mathematical morphology to process the image.

[0066] In Mathematical Morphology, Erosion Iteration is a basic image processing operation used to analyze and modify structural elements in an image. Erosion is a mathematical morphology operation that reduces the size of objects in an image or removes small parts of an object by comparing each pixel in the image with its structural element. The structural element is usually a simple geometric shape such as a rectangle, circle, or a more complex shape. Erosion Iteration is the process of repeating the erosion operation. Each iteration erodes the image using the same structural element, which may cause further reduction of objects in the image or further loss of details. The number of iterations can be adjusted as needed to achieve the desired effect.

[0067] The steps of the corrosion iteration are:

[0068] (1) Select a structural element S.

[0069] (2) Apply the erosion operation to image B to obtain the eroded image B′.

[0070] (3) If further iteration is required, take B′ as the new input image and repeat step (2).

[0071] In this embodiment, Figure 4 (a) in the figure shows an image of the foreground when two people with a height difference of about 90 cm stand close to each other. As can be seen from the figure, the image regions of the two people are of different sizes. In order to find the optimal radius value, an iterative erosion operation is performed using a 3x3 square structure element. Each erosion operation removes a layer of pixels at the edge of the foreground region. The iteration stops when all foreground pixels are removed by the erosion operation. Figure 4 The pixels of different colors in (a) represent the pixels removed by each erosion operation. As can be seen from the figure, after 3 iterations, the foreground area becomes very small. In the 4th iteration, all pixels are removed and the erosion operation stops. The optimal radius value is defined as the number of iterations of the last iteration (i.e. 4 in this case). After the optimal radius value is defined, the set of radii is from 2 to 6. That is, there is a tolerance of two pixels starting from the optimal radius value to cope with the size difference between the two areas. In this 32x32 resolution image, the tolerance of two pixels is appropriate. In other embodiments, for higher resolution images, more pixel tolerances can be set.

[0072] After defining the radius set, it can be obtained by fast radial symmetry Figure 4 The seed point shown in (b).

[0073] refer to Figures 5 to 7,This system shows more results using the above methods.

[0074] exist Figure 5 In the figure, the original image is two people separated from each other, the image after preprocessing the original image of the infrared device is the image on the left, and the schematic diagram of the seed point after processing in step S6 is the image on the right.

[0075] Figure 6 In the figure, the original image is of two people relatively close to each other and there is a small heat source on the left. The image after preprocessing the original image of the infrared device is the image on the left, and the schematic diagram of the seed point after processing in step S6 is the image on the right.

[0076] Figure 7 In the figure, the original image is of three people in different states, the original image corresponding to the infrared frame of the infrared device is the image above, and the schematic diagram of the seed point after processing in step S6 is the corresponding image below. Figure 7 (a) in the figure is an image of three people standing separately. Figure 7 (b) shows three people standing apart, one of them holding a hot water cup (heat source); Figure 7 In (c), three people are standing close to each other, and a hot water cup (heat source) is placed at the lower right. It can be seen from the corresponding figure below them that after the processing in step S6, the obtained seed point is relatively accurate.

[0077] S7: Tracking the target object

[0078] Based on S6, the system can track the target object by matching the horizontal and vertical directions and speeds of seed points in infrared frames from multiple infrared devices. Generally speaking, seed points with similar directions and speeds in multiple infrared frames from multiple infrared devices can be regarded as the same target object, so their position changes can be tracked.

[0079] refer to Figure 8 There are two target objects in the figure. They move to different positions within a period of time (four moments are recorded in the figure). With the help of S7's tracking method, this system can continuously track and record the positions of the target objects.

[0080] In general, this system uses low-resolution infrared detection equipment to detect, count and track target objects. It not only uses low-cost low-resolution infrared detection equipment, but also uses appropriate algorithms to overcome the problems of low precision and low accuracy brought by low-resolution infrared equipment, so that this system can have low-cost and high-precision detection effects.

[0081] Although the preferred arrangements of the present invention have been disclosed to meet the requirements of the specification, it should be understood that various changes, modifications and substitutions may be incorporated without departing from the spirit of the invention as defined by the appended claims.

Claims

1. A detection method using low-resolution infrared equipment, characterized in that: The following steps are involved: S1: For each acquired low-resolution infrared frame, background removal is performed according to a preset rule to obtain an infrared frame that retains the pixels of the target object; S2: average the infrared frames after background removal; S3: Improve the resolution of the infrared frame after averaging; S4: extract foreground objects; S5: Compare the number of pixels or the proportion of the foreground object with a preset value. If the number of pixels or the proportion of the foreground object exceeds the preset value, it is confirmed that the target object is detected.

2. The method according to claim 1, characterized in that In step S1, the preset rule includes: taking an average value of each pixel value in each infrared frame; In a single infrared frame, the pixel values ​​of pixels within a preset range around the average value are set as background pixels, the pixel values ​​of pixels that are background pixels are removed to 0, and the pixel values ​​of non-background pixels are retained.

3. The method according to claim 2, characterized in that Step S3 includes: S3.1 interpolates the infrared frames processed by S2 to obtain images with higher resolution and accuracy; S3.2 filters the interpolated infrared frame to make the image smoother.

4. The method according to claim 3, characterized in that The foreground object is extracted in the infrared frame by using Otsu binarization algorithm, average method or bimodal method.

5. A counting method using a low-resolution infrared device, characterized in that: The detection method comprising 1 to 4 above, further comprising: S6: Find the seed points using fast radial symmetry in the infrared frame and obtain the number of target objects.

6. A tracking method using a low-resolution infrared device, characterized in that: The counting method according to 5 above further comprises: S7: Analyze the position, direction and speed of the seed points of the infrared frames recorded by different devices, and regard the seed points with similar directions and speeds as the same target object for tracking.

7. A detection system using a low-resolution infrared device, characterized in that: include: A low-resolution infrared device for obtaining a low-resolution infrared frame of a target area; A processor, used to process the infrared frame, the processor comprising: A background removal unit is used to remove the background of each acquired low-resolution infrared frame according to a preset rule to obtain an infrared frame that retains the pixels of the target object; An averaging processing unit, used for performing averaging processing on the infrared frames after background removal; A resolution enhancement unit, used to enhance the resolution of the infrared frame after averaging processing; A foreground extraction unit, used for extracting a foreground object; The target detection unit is used to compare the number of pixels or the proportion of the foreground object with a preset value, and if the number of pixels or the proportion of the foreground object exceeds the preset value, it is confirmed that the target object is detected.

8. The detection system according to claim 7, characterized in that: include: In the background removal unit, the preset rule includes: taking an average value of each pixel value in each infrared frame; In a single infrared frame, the pixel values ​​of pixels within a preset range around the average value are set as background pixels, the pixel values ​​of pixels that are background pixels are removed to 0, and the pixel values ​​of non-background pixels are retained.

9. A counting system using low-resolution infrared equipment, characterized in that: The detection system according to claim 7 or 8, further comprising: The counting unit is used to find the seed points in the infrared frame using fast radial symmetry to obtain the number of target objects.

10. A tracking system using a low-resolution infrared device, characterized in that: The counting system of claim 9, further comprising: The tracking unit is used to analyze the position, direction and speed of the seed points of the infrared frames recorded by different devices, and to track the seed points with similar directions and speeds as the same target object.

Images