Method for removing front targets at intersections based on infrared thermal imaging

Abstract

The present invention belongs to the field of intelligent traffic control technology and proposes a method for removing pre-targets at intersections based on infrared thermal imaging. Specifically, moving and stationary motor vehicles, non-motor vehicles, pedestrians, and the like are referred to as pre-targets. The pre-targets are removed from the image based on the temperature difference presented by the infrared thermal imaging of the pre-targets and fixed targets at the intersection, thereby achieving a clean image free of vehicles and pedestrians for later control purposes such as scene synthesis. The removal method comprises the following steps: S1: using an infrared thermal imager to collect temperature and shape information for the entire intersection; S2: dividing the entire intersection into temperature regions; S3: calculating the percentages of the divided regions to obtain the temperature region with the largest percentage; S4: image processing to obtain a coverage area; S5: comparing the overlap of the coverage areas obtained from the two samples to see if it is greater than 90%. If not, return to S1; if so, retain the overlapping region and remove the remaining regions; S6: completing the integration of static objects at the intersection using a deep learning algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent traffic control technology and relates to a method and device for removing pre-targets at intersections based on infrared thermal imaging. That is, motor vehicles, non-motor vehicles, pedestrians, etc. in motion and at rest are referred to as pre-targets. By using the difference in infrared presentation between the pre-targets and fixed targets at the intersection, the pre-targets are removed from the image, thereby achieving a clean image without various vehicles and pedestrians, which is beneficial for the later control of scene synthesis in intelligent traffic. Background Art

[0002] Urban intersections are the most complex traffic scenes on urban roads, with the largest number of participants and the most frequent problems. They are nodes and hubs in the road traffic system, carrying a large amount of traffic. Intelligent traffic management and control at urban intersections requires a critically important approach, where the central control platform and regional traffic control units in the command center can display a real-world intersection scene without pre-processed images of vehicles, non-motorized vehicles, and pedestrians at the intersection. This allows for the arbitrary insertion of virtual vehicles and pedestrians pre-configured by the intelligent traffic system into the intersection for simulated command and control, ultimately achieving intelligent traffic management.

[0003] Smart traffic control systems must rely on intersection scene visualization as a prerequisite for implementation. First, a three-dimensional static real scene of the intersection must be created without various vehicles, pedestrians, and shadows from roadside buildings, trees, and traffic facilities. Traditional video acquisition methods mostly use satellites, drones, and camera systems. The real-time intersection videos collected by the above methods have complex pedestrian and vehicle traffic flows and cannot generate a pure three-dimensional static real scene intersection without vehicles and people. Therefore, real-time simulated images must be used to replace real-time video images for laying out, thus giving rise to the intersection front target foreground removal technology.

[0004] Traditional high-precision maps present intersections from a two-dimensional, simulated, one-way perspective. Vehicles can only access lane information while driving, unable to perceive the actual status of vehicles and pedestrians within the intersection, making it impossible for drivers to make effective driving predictions. With the development of smart transportation, optimizing intersection management and improving road capacity, and achieving efficient and highly perceptive traffic, are becoming the current trends in transportation development. This has led to the emergence of holographic intersections that can understand traffic trends and monitor intersection behavior. Currently, holographic intersections use edge computing to convert video from all four directions of an intersection into a digital holographic perspective, transforming a one-way, two-dimensional, static intersection into an omnidirectional, three-dimensional, real-world static intersection. This comprehensively depicts the traffic conditions at the intersection and outputs traffic metadata such as lane-level license plates, attributes, vehicle speed, location, and driving posture.

[0005] A holographic intersection stitches together four real-time, single-directional front views of the intersection into a panoramic bird's-eye view. Image processing removes the foreground to create a clean, real-time panoramic view. This clean panoramic view then serves as an image base, stitching videos captured from four directions at the current moment into the bird's-eye view to create a holographic view of the intersection. However, existing technologies for removing the foreground from the panoramic view to create the clean base present several challenges. First, using object detection methods, which first detect the foreground and then replace it with a clean image, this approach leaves a noticeable edge after foreground removal due to lighting differences caused by temporal variations in the foreground. Second, dynamic foreground removal methods based on background difference remove the dynamic foreground by comparing historical frames in the video stream. While this approach can avoid the edge effects of foreground replacement compared to object detection methods, it requires adjusting the parameters of the historical frames that affect background difference based on the duration of traffic lights at the intersection, increasing operator workload. Furthermore, for objects that remain stationary or move slowly for a long time, this can cause afterimages, or even cause the system to interpret the scene as static and not remove it if the foreground remains stationary for too long. Third, existing methods for processing clean versions of panoramic bird's-eye view intersections fail to remove shadows cast by buildings, trees, and traffic facilities. These issues significantly impact the ability to obtain a clear, clean version of the panoramic bird's-eye view, and directly affect the realistic presentation of the holographic intersection using this as a base for video stitching. Therefore, obtaining a clear, clean version of the panoramic bird's-eye view base is a prerequisite for achieving a clear and realistic holographic intersection. Summary of the Invention

[0006] The method for removing pre-targets at intersections based on infrared thermal imaging proposed in the present invention collects temperature and shape information of the entire intersection through infrared thermal imaging, divides the area into regions using temperature differences to calculate regional proportions, retains the area with the maximum temperature proportion, removes other temperature proportions, and then uses a deep learning algorithm to compare and verify the collected shape information to complete the static objects in the intersection to form a clean panoramic bird's-eye view of the intersection without shadows, afterimages, or retained targets.

[0007] The technical solution of the present invention is achieved as follows:

[0008] S1: Use an infrared thermal imager to collect temperature and shape information of various vehicles, non-motorized vehicles, and pedestrians in all four directions of the intersection;

[0009] S2: Divide the physical state and structural characteristics of the entire intersection into temperature zones;

[0010] Furthermore, step S2 includes

[0011] S21: The lowest temperature value collected in the entire intersection is used as the base temperature, which is recorded as T;

[0012] S22: Set the amplitude variation range to X℃, and get m temperature zones, which are recorded as T m , where Tm represents the temperature between T+(m-1)X and T+mX, (m=1,2......K);

[0013] Furthermore, the temperature region T m The maximum value is the highest temperature value collected within the entire intersection range.

[0014] S3: Calculate the temperature area ratio of each divided temperature area to obtain the temperature area P with the largest ratio. Determine whether the number of sampling times is less than 2. If so, return to S1. Otherwise, proceed to the next step.

[0015] Furthermore, step S3 includes

[0016] S31: The minimum pixel of the infrared thermal imager to collect temperature is , the temperature of each pixel is T m Within the range, the set of pixels corresponding to the temperature t℃ is recorded as ;

[0017] S32: Temperature in region T m The set of pixels within the range is ;

[0018] S33: Calculate and sort the proportions of the m temperature regions to obtain the temperature region P1 with the largest proportion;

[0019] S34: Return to S1 to perform a second sampling, calculate and sort the temperature area proportions according to the temperature areas previously divided, and obtain the temperature area P2 with the largest proportion;

[0020] S4: Comparative sampling obtains the temperature areas P1 and P2 with the largest proportion, and image processing obtains the coverage area M of P1 and P2;

[0021] S5: Compare the coverage areas of M1 and M2 to see if the overlap is greater than 90%. If not, return to S1. If so, retain the overlapped area of ​​M1 and M2 and remove the rest.

[0022] S6: Compare and verify the collected shape information, and complete the static objects at the intersection through deep learning algorithms;

[0023] The working principle and beneficial effects of the present invention are:

[0024] Taking advantage of the fact that any object has a different temperature range, infrared thermal imaging is used to collect temperature information of various vehicles, non-motor vehicles and pedestrians in the four directions of the intersection. By showing different infrared heat sources of any object, it is possible to effectively remove the front scenes of motor vehicles, non-motor vehicles and pedestrians at the intersection, as well as shadows formed by buildings, trees, traffic facilities, etc., to achieve the averaging of the temperature of the clean intersection scene.

[0025] Through the dual acquisition function of infrared thermal imaging temperature and shape of different objects, the static base part that was mistakenly deleted can be supplemented, and the foreground targets that were mistakenly retained and have a temperature similar to that of the static base object can be removed again, thereby obtaining the final static base image of the intersection without the foreground targets.

[0026] By comparing and verifying the deep learning algorithm with the real-life video, the static objects at the intersection are completed to form a clean panoramic bird's-eye view of the intersection with no shadows, afterimages, or retained targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Figure 1 This is a flow chart of the method for removing front targets at intersections based on infrared thermal imaging;

[0029] Figure 2 This is a diagram showing an embodiment of a method for removing a preceding target at an intersection based on infrared thermal imaging;

[0030] Figure 3 This is a real-life image of the method for removing pre-targets at intersections based on infrared thermal imaging;

[0031] Figure 4 This is an infrared thermal imaging diagram of the method for removing front targets at intersections based on infrared thermal imaging;

[0032] Figure 5 This is the final effect of the intersection front target removal method based on infrared thermal imaging;

[0033] 01. Motor vehicle at rest; 02. Light pole shadow; 03. Pedestrian; 04. Motor vehicle in motion; 05. Motor vehicle shadow; 11. Infrared imaging temperature of motor vehicle at rest: 40.5°C; 12. Infrared imaging temperature of light pole shadow: 22°C; 13. Infrared imaging temperature of pedestrian: 36.3°C; 14. Infrared imaging temperature of motor vehicle in motion: 45.7°C; 15. Infrared imaging temperature of motor vehicle shadow: 22.5°C. Implementation Method

[0034] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0035] The method of removing front targets at intersections based on infrared thermal imaging includes:

[0036] S1: Use an infrared thermal imager to collect temperature and shape information of various vehicles, non-motorized vehicles, and pedestrians in all four directions of the intersection;

[0037] S2: Divide the physical state and structural characteristics of the entire intersection into temperature zones;

[0038] Step S2 includes

[0039] S21: The lowest temperature value collected in the entire intersection is used as the base temperature, which is recorded as T;

[0040] S22: Set the amplitude variation range to X℃, and get m temperature zones, which are recorded as T m , where Tm represents the temperature between T+(m-1)X and T+mX, (m=1,2......K);

[0041] S3: Calculate the temperature area ratio of each divided temperature area to obtain the temperature area P with the largest ratio. Determine whether the number of sampling times is less than 2. If so, return to S1. Otherwise, proceed to the next step.

[0042] Step S3 includes

[0043] S31: The minimum pixel of the infrared thermal imager to collect temperature is , the temperature of each pixel is T m Within the range, the set of pixels corresponding to the temperature t℃ is recorded as ;

[0044] S32: Temperature in region T m The set of pixels within the range is ;

[0045] S33: Calculate and sort the proportions of the m temperature regions to obtain the temperature region P1 with the largest proportion;

[0046] S34: Return to S1 to perform a second sampling, calculate and sort the temperature area proportions according to the temperature areas previously divided, and obtain the temperature area P2 with the largest proportion;

[0047] S4, comparing and sampling to obtain the temperature areas P1 and P2 with the largest proportion, and performing image processing to obtain the coverage area M of P1 and P2;

[0048] S5. Compare the coverage areas of M1 and M2 to see if the overlap is greater than 90%. If not, return to S1. If so, retain the overlapped area of ​​M1 and M2 and remove the rest.

[0049] S6: Compare and verify with the collected shape information, and complete the static objects at the intersection through deep learning algorithms.

[0050] Since the actual image temperature pixel set is a relatively large calculation process, this embodiment is a simple example to illustrate the implementation process of this method. Figure 2 As shown, this embodiment is described by taking 10*10 temperature pixels as an example.

[0051] The infrared thermal imager is used to collect the temperature and shape information of the four directions of the entire intersection for the first time. Figure 2-1 As shown in the figure, the temperature range collected is 25°C to 80°C (assuming that the intersection in this example includes the intersection ground, tree shadows, moving vehicles, and pedestrians. The collected intersection ground temperature is around 45°C, the tree shadow temperature is around 25°C, the moving vehicle temperature is around 80°C, and the pedestrian temperature is around 36°C).

[0052] like Figure 2-1 In the example, the minimum pixel Mij (i=1, 2...10, j=1, 2...10), the temperature values ​​t collected by the infrared thermal imager include 25℃, 26℃, 36℃, 36.5℃, 43℃, 44℃, 79℃, and 80℃. Calculate the set of all temperature values ​​within the temperature collection range of this intersection, that is, the set of pixels corresponding to the temperature t=25℃. ; The set of pixels corresponding to temperature t=26℃ ; The set of pixels corresponding to temperature t=36℃ ; The set of pixels corresponding to temperature t=36.5℃ ; The set of pixels corresponding to temperature t=43℃ ; The set of pixels corresponding to temperature t=44℃ ; The set of pixels corresponding to temperature t=79℃ ; The set of pixels corresponding to temperature t=80℃ ;

[0053] Set the amplitude change range X=5℃, and get 11 temperature zones: T1=25℃~30℃, T2=30℃~35℃, T3=35℃~40℃, T4=40℃~45℃, T5=45℃~50℃, T6=50℃~55℃, T7=55℃~60℃, T8=60℃~65℃, T9=65℃~70℃, T10=70℃~75℃, T11=75℃~80℃;

[0054] The set of pixels whose temperature is within the range of area T1 is The pixel set whose temperature is within the range of area T2 is The pixel set whose temperature is within the range of area T3 is The pixel set whose temperature is within the range of area T4 is The pixel set whose temperature is within the range of area T5 is The pixel set whose temperature is within the range of area T6 is The pixel set whose temperature is within the range of area T7 is The pixel set whose temperature is within the range of area T8 is The pixel set whose temperature is within the range of area T9 is The pixel set whose temperature is within the range of area T10 is The pixel set whose temperature is within the range of area T11 is ; Calculate the proportion of the 11 temperature zones and obtain the temperature zone with the largest proportion P1=V4;

[0055] Use infrared thermal imager to collect temperature and shape information of all four directions of the intersection for the second time. Figure 2-2 As shown, the minimum pixel The temperature values ​​t collected by the infrared thermal imager include 25℃, 26℃, 36℃, 36.3℃, 36.5℃, 43℃, 44℃, 79℃, and 80℃. Calculate the set of all temperature values ​​within the temperature collection range of this intersection, that is, the set of pixels corresponding to temperature t=25℃. ; The set of pixels corresponding to temperature t=26℃ ; The set of pixels corresponding to temperature t=36℃ ; The set of pixels corresponding to temperature t=36.3℃ ; The set of pixels corresponding to temperature t=36.5℃ ; The set of pixels corresponding to temperature t=43℃ ; The set of pixels corresponding to temperature t=44℃ ; The set of pixels corresponding to temperature t=79℃ ; The set of pixels corresponding to temperature t=80℃ ;

[0056] Calculation is still performed according to the 11 temperature zones divided for the first time: T1=25℃~30℃, T2=30℃~35℃, T3=35℃~40℃, T4=40℃~45℃, T5=45℃~50℃, T6=50℃~55℃, T7=55℃~60℃, T8=60℃~65℃, T9=65℃~70℃, T10=70℃~75℃, T11=75℃~80℃;

[0057] The set of pixels whose temperature is within the range of area T1 is The pixel set whose temperature is within the range of area T2 is The pixel set whose temperature is within the range of area T3 is The pixel set whose temperature is within the range of area T4 is

[0058] The pixel set whose temperature is within the range of area T5 is The pixel set whose temperature is within the range of area T6 is The pixel set whose temperature is within the range of area T7 is The pixel set whose temperature is within the range of area T8 is The pixel set whose temperature is within the range of area T9 is The pixel set whose temperature is within the range of area T10 is The pixel set whose temperature is within the range of area T11 is ; Calculate the proportion of the 11 temperature zones and find that the temperature zone with the largest proportion is P2=V4 ’ ;

[0059] Comparative sampling yields the largest temperature regions P1 and P2. Image processing yields the coverage region M1 of P1 and the coverage region M2 of P2. The calculated overlap of the coverage regions of M1 and M2 is 91.4%. The overlapping regions of M1 and M2 are retained, and the rest are removed. The region retained in this embodiment is P2. Comparison and verification are performed with the collected shape information, and the integrated static objects at the intersection are completed using a deep learning algorithm based on historical data. The shape data collected through infrared thermal imaging is then compared and learned with historically collected data, and the incorrectly deleted static base information is completed based on historical data. Finally, the completed image, obtained after the foreground target removal, is compared and verified with the real-life video to obtain the final static base image of the intersection with the foreground target removed.

[0060] It is worth noting that, as shown in the figure, this embodiment only shows a simple implementation method of the method for removing pre-targets at intersections based on infrared thermal imaging. It is only for the purpose of demonstrating the principle of this method. The actual pixel accuracy, temperature zone division, calculation accuracy, etc. are much higher than those exemplified in the embodiment. The specific number, position, angle, etc. of settings can be set according to the needs of actual applications.

[0061] In the description of the present invention, it should be understood that the terms "up", "down", "front", "back", "left", "right", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0062] The embodiments described above are merely a description of one possibility of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A method for removing front-end targets at intersections based on infrared thermal imaging, characterized in that: include S1: Use an infrared thermal imager to collect temperature and shape information of various vehicles, non-motorized vehicles, and pedestrians in all four directions of the intersection; S2: The lowest temperature value collected in the entire intersection is used as the base temperature, recorded as T; the amplitude variation range is set to X℃, and m temperature zones are obtained, recorded as T m , where Tm represents the temperature between T+(m-1)X and T+mX, (m=1,2......K); S3: The minimum pixel for temperature acquisition is denoted as Mij (i = 1, 2......n, j = 1, 2......n), and the temperature of each pixel is within T m range. The set of pixels corresponding to the temperature of t °C is denoted as f(t) = {Mij丨Mij = t}; the set of pixels with temperatures within the region T m range is ; Calculate and sort the proportion of m temperature regions to obtain the temperature region P1 with the largest proportion; Determine whether the number of samplings is less than 2. If so, return to S1 for the second sampling, calculate and sort the proportion of the temperature region according to the temperature region divided in the previous time to obtain the temperature region P2 with the largest proportion; If not, proceed to the next step; S4, comparing and sampling to obtain the temperature areas P1 and P2 with the largest proportion, and performing image processing to obtain the coverage area M of P1 and P2; S5. Compare the coverage areas of M1 and M2 to see if the overlap is greater than 90%. If not, return to S1. If so, retain the overlapped area of ​​M1 and M2 and remove the rest. S6: Compare and verify with the collected shape information, and complete the static objects at the intersection through deep learning algorithms.

2. The method for removing pre-positioned targets at intersections based on infrared thermal imaging according to claim 1, characterized in that: Temperature zone T m The maximum value is the highest temperature value collected within the entire intersection range.

3. The method for removing pre-targets at intersections based on infrared thermal imaging according to claim 1, characterized in that: The shape data collected by infrared thermal imaging is compared with the historical collected data, and the accidentally deleted static base information is supplemented according to the historical data.

4. The method for removing pre-targets at intersections based on infrared thermal imaging according to claim 1, characterized in that: The completed image without the foreground target is compared with the real scene video to obtain the final static base image of the intersection without the foreground target.

Images