Multi-sensor combined tunnel early warning method, system and device and storage medium

Abstract

The invention discloses a multi-sensor combined tunnel early warning method, system and device and a storage medium, a thermal imaging camera and a laser radar are arranged on a vehicle, and the method comprises the following steps: judging whether the vehicle enters a tunnel; if the vehicle enters the tunnel, the tunnel is identified through a thermal imaging camera, and whether a sudden change area exists or not is judged based on thermal imaging data of the thermal imaging camera; a tunnel is identified through a laser radar, and a sensing area of the tunnel is obtained based on point cloud data of the laser radar; if the sensing area has the mutation area, a target area is divided in the sensing area, and the target area comprises the mutation area; and fusing the point cloud data on the target area and the thermal imaging data on the target area, performing identification based on a fusion result, and giving out an early warning based on an identification result. According to the invention, through combined identification of the laser radar and the thermal imaging camera, the moving obstacle on the road shoulder in the tunnel can be effectively identified.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-sensor combined tunnel early warning method, system, device and storage medium. Background Art

[0002] Some existing tunnels have no lighting equipment (or only a few lighting equipment), resulting in low brightness in the tunnel. When a vehicle is traveling, the human eye and visible light camera cannot accurately detect whether there are active obstacles on the shoulder. At the same time, since the active obstacles may be very close to the tunnel wall, the laser radar cannot accurately identify them, causing serious safety impacts on the vehicle's driving in the tunnel. Therefore, how to provide a more accurate trial wearing effect has become a technical problem that technicians in this field need to solve. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a multi-sensor combined tunnel early warning method, system, device and storage medium, which can accurately identify active obstacles at the shoulder position of the tunnel.

[0004] In a first aspect, the present application discloses a multi-sensor combined tunnel warning method, which is provided on a vehicle by a thermal imaging camera and a laser radar. The method includes: determining whether the vehicle enters a tunnel; If the vehicle enters the tunnel, the tunnel is identified by the thermal imaging camera, and whether there is a mutation area is determined based on the thermal imaging data of the thermal imaging camera; The tunnel is identified by the laser radar, and the perception area of ​​the tunnel is obtained based on the point cloud data of the laser radar; If the mutation region exists in the sensing region, dividing a target region in the sensing region, wherein the target region includes the mutation region; The point cloud data on the target area and the thermal imaging data on the target area are fused, recognition is performed based on the fusion result, and an early warning is issued based on the recognition result.

[0005] In an optional implementation, fusing the point cloud data on the target area with the thermal imaging data on the target area, and performing identification based on the fusion result includes: Based on whether there is a change in the thermal imaging data of the mutation area within a unit time, determining whether there is a target object in the mutation area; If there is a target object in the mutation area, obtaining point cloud data of the mutation area based on the point cloud data on the target area and the thermal imaging data on the target area; The target object is identified based on the point cloud data of the mutation area.

[0006] In an optional embodiment, obtaining the point cloud data of the mutation area based on the point cloud data of the target area and the thermal imaging data of the target area includes: According to the activity traces of the thermal imaging of the mutation area within a unit time period, the thermal imaging data changes of the mutation area are obtained; Based on the thermal imaging change of the mutation area and the point cloud data of the target area within a unit time period, point cloud data about the mutation area is obtained.

[0007] In an optional implementation, the identifying the target object based on the point cloud data of the mutation area includes: Based on the thermal imaging data of the target area, a thermal imaging contour in the mutation area based on the thermal imaging data is obtained; Based on the point cloud data of the target area and the thermal imaging contour at the same time, obtaining a point cloud contour in the mutation area based on the point cloud data; The target object is identified based on the point cloud contour.

[0008] In an optional embodiment, the vehicle is further provided with a first marker, and the first marker is provided in the acquisition area of ​​the thermal imaging camera; if the mutation area exists in the sensing area, the detection includes: According to the preset distance between the first marker and the thermal imaging camera, combined with the thermal imaging image data, the actual distance between the mutation area and the thermal imaging camera is obtained; According to the actual distance between the mutation area and the thermal imaging camera, it is repeatedly determined whether the mutation area is within the scanning area of ​​the laser radar.

[0009] In an optional embodiment, obtaining the actual distance between the mutation area and the thermal imaging camera based on the preset distance between the first marker and the thermal imaging camera in combination with the thermal imaging image data comprises: According to the preset distance between the first marker and the thermal imaging camera, an actual distance represented by a unit pixel in the current frame of thermal imaging data is obtained, and the actual distance represented by a unit pixel in the current frame of thermal imaging data is defined as a first reference distance; Based on the first reference distance and the number of pixels of the mutation area and the first marker in the current frame of thermal imaging data, the actual distance between the mutation area and the thermal imaging camera is obtained.

[0010] In an optional implementation, the repeatedly determining whether the mutation area is within the scanning area of ​​the laser radar according to the actual distance between the mutation area and the thermal imaging camera includes: According to the actual distance between the mutation area and the thermal imaging camera and the scanning distance of the laser radar, confirm whether the mutation area is within the scanning area of ​​the laser radar; If the mutation area is not within the scanning area of ​​the LiDAR, the actual distance between the mutation area and the visible light camera in each subsequent frame of thermal imaging data is continuously calculated until the mutation area is within the scanning area of ​​the LiDAR.

[0011] In an optional implementation, the vehicle is further provided with a visible light camera, and the determining whether the vehicle enters a tunnel includes: Determine, based on the image data collected by the visible light camera, whether there is an area in the current frame image where the brightness contrast exceeds a preset range, where the area is defined as a first area; Determining whether the first area is a tunnel entrance; If the first area is a tunnel entrance, it is determined whether the vehicle has entered the tunnel according to the distance between the vehicle and the first area.

[0012] In an optional embodiment, the vehicle is further provided with a second marker, which is provided in a collection area of ​​the visible light camera; the determining, based on the image data collected by the visible light camera, whether there is an area in the current frame image where the brightness contrast exceeds a preset range, wherein the area is defined as the first area including: According to the preset distance between the second marker and the visible light camera, an actual distance represented by a unit pixel in the current frame image is obtained, and the actual distance represented by the unit pixel in the current frame image is defined as a second reference distance; Based on the second reference distance and the number of pixels of the first area and the second marker in the current frame image, the actual distance between the first area and the visible light camera is obtained.

[0013] In an optional implementation manner, before determining whether the first area is a tunnel entrance, the step further includes: According to the actual distance between the first area in the current frame image and the visible light camera and the scanning distance of the laser radar, confirm whether the first area is within the scanning area of ​​the laser radar; If the first area is not within the scanning area of ​​the laser radar, the actual distance between the first area and the visible light camera in each subsequent frame of image data is continuously calculated until the first area is within the scanning area of ​​the laser radar.

[0014] In an optional implementation manner, the determining whether the vehicle enters a tunnel includes: Determine, based on the point cloud data collected by the laser radar, whether there is an area in the current frame point cloud image where the position information continuity change contrast exceeds a preset range, and the area is defined as a first area; Determining whether the first area is a tunnel entrance; If the first area is a tunnel entrance, it is determined whether the vehicle has entered the tunnel according to the distance between the vehicle and the first area.

[0015] In an optional implementation, the perception area is divided into at least two sub-areas, including at least one road sub-area and at least one shoulder sub-area, and each of the sub-areas corresponds to a weight coefficient; and issuing an early warning based on the recognition result includes: Based on the weight coefficient corresponding to the sub-area of ​​the target area in the perception area and the recognition result of the target area, different types of warning signals are issued.

[0016] In a second aspect, the embodiment of the present application discloses a multi-sensor combined tunnel warning system, which is set on a vehicle through a thermal imaging camera and a laser radar, including: Judgment module: used to judge whether the vehicle enters the tunnel; Thermal imaging perception module: after the vehicle enters the tunnel, the tunnel is identified by the thermal imaging camera, and whether there is a mutation area is determined based on the thermal imaging data of the thermal imaging camera; Point cloud perception module: identifies the tunnel through the laser radar and obtains the perception area of ​​the tunnel based on the point cloud data of the laser radar; Region division module: if the mutation region exists in the perception region, a target region is divided in the perception region, wherein the target region includes the mutation region; Early warning module: fuses the point cloud data on the target area with the thermal imaging data on the target area, performs recognition based on the fusion result, and issues an early warning based on the recognition result.

[0017] In a third aspect, an embodiment of the present application discloses a computer storage medium storing a computer program, which implements the above-mentioned method steps when executed by a processor.

[0018] In a fourth aspect, the embodiment of the present application discloses a multi-sensor combined tunnel warning device, including a thermal imaging camera, a laser radar and an industrial computer; The thermal imaging camera, laser radar and industrial computer are arranged on the vehicle; The industrial computer is used to execute the above steps.

[0019] The embodiments of the present application have the following beneficial effects: The multi-sensor combined tunnel warning method of the present application obtains a target area that may include active obstacles by utilizing the thermal imaging image data of a thermal imaging camera, and then identifies the target area through the point cloud data of the laser radar in the above target area and the thermal imaging image data of the thermal imaging camera in the target area, and issues a warning based on the identification result. The present application effectively avoids the problem of ineffective and accurate identification of active obstacles on the shoulder of the road by visible light cameras and laser radars in tunnels without lighting equipment through the temperature difference characteristics of active obstacles, thereby effectively ensuring the safety of vehicles in tunnels.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flow chart of Embodiment 2 of the present application is shown; Figure 2 The structural entity diagram of Embodiment 1 of the present application is shown; Figure 3 An environmental diagram of an embodiment of the present application is shown; Figure 4 A schematic diagram of an implementation of S100 in Example 2 of the present application is shown; Figure 5 A schematic diagram of an implementation of S110 in Example 2 of the present application is shown; Figure 6 Another schematic diagram of an implementation of S100 in Example 2 of the present application is shown; Figure 7 Another schematic diagram of implementation of S100 in Example 2 of the present application is shown; Figure 8 A schematic diagram of the brightness comparison outside the tunnel entrance in an embodiment of the present application is shown.

[0023] Fig. 9 A schematic diagram of an implementation of S500 in Example 2 of the present application is shown; Fig.10 A schematic diagram of an implementation of S520 in Example 2 of the present application is shown; Fig.11 A schematic diagram of an implementation of S530 in Example 2 of the present application is shown; Fig.12 A schematic diagram of an implementation of S400 in Example 2 of the present application is shown; Fig.13 A schematic diagram of an implementation of S410 in Example 2 of the present application is shown; Fig.14 A schematic diagram of an implementation of S420 in Example 2 of the present application is shown; Fig.15 A structural schematic diagram of Example 3 of the present application is shown.

[0024] Explanation of main component symbols: 10. Thermal imaging camera; 20. Industrial computer; 30. LiDAR. Implementation

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0026] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0027] Hereinafter, the term "comprising" and its cognates that can be used in various embodiments of the present application are intended only to indicate a specific feature, number, step, operation, element, component, or a combination of the aforementioned items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or a combination of the aforementioned items or increasing the possibility of one or more features, numbers, steps, operations, elements, components, or a combination of the aforementioned items. In addition, the terms "first", "second", etc. are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0028] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0029] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0030] Overview To facilitate understanding, the background technology and specific terms are now described in detail.

[0031] Among the existing tunnels, some tunnels have no lighting equipment (or have very few lighting equipment), resulting in very low brightness in the tunnel (the above situation is more common in railway tunnels). When a vehicle (motor vehicle or train) enters the tunnel, since there is no lighting equipment in the tunnel, the visibility of the tunnel is very low, and moving obstacles cannot be discovered in time by the human eye or visible light camera, which is extremely detrimental to the driving safety of the vehicle. Some vehicles may be equipped with lighting equipment, but the lighting range of ordinary lighting equipment is limited and may not cover the shortest safe distance of the vehicle at high speeds. If long-range lighting equipment (such as high beams) is used, it may seriously affect the driving safety of oncoming vehicles and may even violate relevant laws and regulations. In addition, if the movable obstacle is on the shoulder of the road, since the distance between the movable obstacle and the tunnel wall is very close (such as Figure 3 ), laser radar alone cannot detect active obstacles on the road shoulder in time.

[0032] In the embodiment of the present application, vehicles are not limited to motor vehicles, but also include trains, trains, etc. The thermal imaging camera detects infrared energy (heat) non-contactly and converts it into electrical signals to generate thermal imaging image data. The laser radar collects the target area through laser. In the embodiment of the present application, it is not limited to mechanical laser radar, but also includes hybrid solid-state laser radar and solid-state laser radar. In the embodiment of the present application, active obstacles refer to animals such as people and wild animals.

[0033] Based on this, this embodiment provides a multi-sensor combined tunnel warning method, system, device and storage medium to identify active obstacles to avoid the situation where the vehicle cannot identify the absence of active obstacles when the tunnel brightness is low. Example

[0034] See also Figure 2 The embodiment of the present application provides a multi-sensor combined tunnel warning device, including a thermal imaging camera 10, a laser radar 30 and an industrial computer 20, wherein the thermal imaging camera 10, the laser radar 20 and the industrial computer 30 are arranged on a vehicle; the thermal imaging camera 10 is arranged on the vehicle to collect thermal imaging image data in front of the vehicle; the laser radar 30 is arranged on the vehicle to collect point cloud data in front of the vehicle; the industrial computer 20 is used to perform corresponding processing on the thermal imaging image data collected by the thermal imaging camera 10 and the point cloud data collected by the laser radar 30 to identify whether there are active obstacles in the tunnel.

[0035] In one embodiment, the number of thermal imaging cameras 10 and laser radars 20 provided in the tunnel warning device is not limited.

[0036] In one embodiment, there is a clear preset distance between the thermal imaging camera 10 and the laser radar 20, which is used to calculate the distance between the future target area and the scanning area of ​​the laser radar 20, and the time when the target area enters the scanning area of ​​the laser radar 20. Example

[0037] This embodiment describes in detail the multi-sensor combined tunnel warning method in combination with the above-mentioned tunnel warning device.

[0038] S100: Determine whether the vehicle enters a tunnel; In one embodiment, see Figure 4 The vehicle is also provided with a visible light camera, and step S100 specifically includes the following steps: S110: Determine, based on the image data collected by the visible light camera, whether there is an area in the current frame image where the brightness contrast exceeds a preset range, where the area is defined as a first area; Since the visible light camera can adjust the depth of field by itself, if there are no obstructions, the visible light camera can adjust a higher depth of field. Therefore, when taking images, the visible light camera can identify a larger detection range than the lidar, that is, some areas captured by the visible light camera may not be in the scanning area of ​​the lidar. Therefore, it is possible to perceive in advance whether the first area is about to enter the scanning area of ​​the lidar through the image data collected by the visible light camera. In addition, since it is difficult for light outside the tunnel to enter the tunnel, there will be a significant difference in brightness between the inside and outside of the tunnel entrance when viewed from the outside (see Figure 8 Based on the above characteristics, it is possible to determine whether there is a first area (i.e., an area that may be a tunnel entrance) in front of the vehicle based on the visible light camera, so as to prepare for the vehicle to enter the tunnel later.

[0039] In one embodiment, see Figure 5 , a second marker is provided on the vehicle, and the second marker is provided within the collection area of ​​the visible light camera, and step S110 specifically includes the following steps: S111: obtaining an actual distance represented by a unit pixel in a current frame image according to a preset distance between the second marker and the visible light camera, wherein the actual distance represented by the unit pixel in the current frame image is defined as a second reference distance; Exemplarily, the second marker is an item that can be displayed in the image data collected by the visible light camera, and the present application does not impose any restrictions on this. It is preferably a laser radar on the vehicle. Since the preset distance between the second marker and the visible light camera is pre-set. Therefore, the actual distance represented by each pixel of the current frame image (i.e., the second reference distance) can be obtained by combining the number of pixels between the second marker and the visible light camera in the current frame image and the preset distance between the second marker and the visible light camera.

[0040] S112: Based on the second reference distance and the number of pixels between the first area and the second marker in the current frame image, obtain the actual distance between the first area and the visible light camera.

[0041] Based on the actual distance between the first area and the visible light camera, the driver / industrial computer can know the actual distance between the vehicle and the first area, and prepare for the subsequent lidar scanning to identify the first area.

[0042] In one embodiment, see Figure 6 , before S140 also included: S120: confirming whether the first area is within the scanning area of ​​the laser radar according to the actual distance between the first area and the visible light camera in the current frame image and the scanning distance of the laser radar; S130: If the first area is not within the scanning area of ​​the laser radar, continuously calculating the actual distance between the first area and the visible light camera in each subsequent frame of image data until the first area is within the scanning area of ​​the laser radar; As mentioned above, since the visible light camera can identify a higher detection range than the laser radar when taking images, when the visible light camera identifies the first area, the first area may not have entered the scanning area of ​​the laser radar. Therefore, it is necessary to continuously calculate the actual distance between the first area and the visible light camera to confirm whether the first area is in the scanning area of ​​the laser radar and how long it will take for the first area to enter the scanning area of ​​the laser radar.

[0043] S140: Determine whether the first area is a tunnel entrance; Since the point cloud data of the laser radar carries the location information of each point cloud, by analyzing the contour of the first area, including analyzing parameter information such as the length, height, and shape of the contour, it can be determined whether the first area is a tunnel entrance.

[0044] S150: If the first area is a tunnel entrance, determine whether the vehicle has entered the tunnel based on the distance between the vehicle and the first area.

[0045] Since the distance information obtained through point cloud data is more accurate than the distance information obtained through the second reference distance, when the first area is confirmed to be the tunnel entrance, the actual distance between the tunnel entrance and the vehicle can be determined through point cloud data, and the time when the vehicle enters the tunnel entrance can be obtained by combining the real-time speed of the vehicle and the driving path of the vehicle.

[0046] In another embodiment, see Figure 7 Step S100 may further include the following steps: S110': determining, based on the point cloud data collected by the laser radar, whether there is an area in the current frame point cloud image where the continuous change contrast of the position information exceeds a preset range, and the above area is defined as a first area; S120': Determine whether the first area is a tunnel entrance; S130': If the first area is a tunnel entrance, determine whether the vehicle enters the tunnel according to the distance between the vehicle and the first area.

[0047] In this implementation, the regional information of the tunnel entrance is obtained by another method. Based on the principle of laser radar, the emitted light beam will be reflected back to the receiving module of the laser radar when it reaches the obstacle. Therefore, when the laser radar scans the tunnel entrance and the surrounding area, the surrounding area is reflected back to the receiving module of the laser radar due to the presence of occlusion. In reality, the surrounding area is a natural shape such as a mountain, and the change in the position information of the overall point cloud data reflects continuity. Since there is no obstacle blocking the tunnel entrance, the emitted light beam will be projected into the tunnel. When encountering occlusion and reflection, the receiving module may be unable to be recognized because the intensity of the reflected signal is too small (mainly due to distance), which is reflected as the loss of the above point cloud data. Therefore, on the point cloud image of the laser radar, there are two situations for the point cloud data in the tunnel entrance area. One is that the intensity of the reflected signal is too small, resulting in the loss of point cloud data, and the other is that the reflection is received by the receiving module of the laser radar due to occlusion in the tunnel, but the position information of the above point cloud data will be greatly different from the position information of the point cloud data in the surrounding area of ​​the tunnel entrance. Therefore, no matter which of the above situations, the point cloud data of the tunnel entrance will be significantly different from the point cloud data around the tunnel entrance. Based on the above characteristics, by analyzing whether the continuous change of the position information in the point cloud data of the scanning area exceeds the preset range, it is analyzed whether there is a tunnel entrance in the scanning area.

[0048] S200: After the vehicle enters the tunnel, the tunnel is identified by a thermal imaging camera, and whether there is a mutation area is determined based on the thermal imaging data of the thermal imaging camera; As mentioned above, in a tunnel without lighting equipment, visible light cameras / human eyes cannot accurately identify whether there are active obstacles in the tunnel. In addition, a simple laser radar cannot effectively identify the situation of a long-distance active obstacle on the shoulder. However, since it is an active obstacle, there must be a large temperature difference between its temperature and the surrounding temperature, so it can be identified by a thermal imaging camera. Thermal imaging cameras are very sensitive to temperature and can effectively identify and discover even distant active obstacles. Although, due to the long distance, there may not be many pixels of suspected active obstacles, due to the temperature difference, they will appear in completely different colors in the thermal imaging image data, so even if they occupy a small number of pixels, they are easy to identify. In addition, since tunnels are generally set up in the mountains, light cannot penetrate in, and the outside is covered by the mountains, heat conduction is relatively difficult. The temperature inside the tunnel is generally lower than that outside the tunnel, and the overall temperature change inside the tunnel is also more uniform. Therefore, when there is an active obstacle, the degree of its temperature difference will be more obvious than that of the active obstacle outside the tunnel, and it is more conducive to identification. On the other hand, during the laser radar transmission process, the longer the transmission distance, the farther the distance between the point clouds. At the same time, because the active obstacles are near the tunnel wall, the number of point clouds on the active obstacles is very small, and it is easy to mix with the point clouds on the tunnel wall and cannot be identified. Therefore, when taking thermal imaging images, thermal imaging cameras can identify a larger detection range than laser radars, that is, some areas captured by thermal imaging cameras may not be in the scanning area of ​​the laser radar.

[0049] S300: Identify the tunnel through the laser radar and obtain the sensing area of ​​the tunnel based on the point cloud data of the laser radar; As mentioned above, due to the high sensitivity of thermal imaging cameras to temperature, there may be a situation where the mutation area identified by the thermal imaging camera is not in the scanning area of ​​the LiDAR. Therefore, it is necessary to obtain the LiDAR's perception area of ​​the tunnel through the scanning of the LiDAR, and then match the LiDAR's perception area of ​​the tunnel with the mutation area identified by the thermal imaging camera to determine whether the mutation area is in the perception area.

[0050] S400: if there is a mutation region in the perception region, dividing a target region in the perception region, where the target region includes the mutation region; In one embodiment, see Fig.12 The vehicle is also provided with a first marker, which is provided in the acquisition area of ​​the thermal imaging camera; if there is a mutation area in the sensing area in step S400, the following steps are included: S410: Obtaining an actual distance between the mutation area and the thermal imaging camera according to a preset distance between the first marker and the thermal imaging camera and in combination with the thermal imaging image data; In one embodiment, see Fig.13 , step S410 includes the following steps: S411: obtaining an actual distance represented by a unit pixel in the current frame of thermal imaging data according to a preset distance between the first marker and the thermal imaging camera, wherein the actual distance represented by a unit pixel in the current frame of thermal imaging data is defined as a first reference distance; Exemplarily, the first marker is an item that can be displayed in the image data collected by the thermal imaging camera. The present application does not impose any restrictions on this. It is preferably a laser radar on the vehicle. Because the transmitting module inside the laser radar will generate a higher temperature when the laser radar is working, it is an easily distinguishable item for thermal imaging data. And it can also be recognized by the optical camera. If the first marker and the second marker are unified as the laser radar on the vehicle, it will be possible to effectively avoid setting too many markers on the vehicle, and it will also be beneficial to simplify the overall method. Since the preset distance between the first marker and the thermal imaging camera is pre-set. Therefore, the actual distance represented by each pixel of the current frame thermal imaging data (i.e., the first reference distance) can be obtained by combining the preset distance between the first marker and the thermal imaging camera in the current frame thermal imaging data with the number of pixels between the first marker and the thermal imaging camera.

[0051] S412: Based on the first reference distance and the number of pixels between the mutation area and the first marker in the thermal imaging data of the current frame, the actual distance between the mutation area and the thermal imaging camera is obtained.

[0052] S420: Repeatedly determine whether the mutation area is within the scanning area of ​​the laser radar according to the actual distance between the mutation area and the thermal imaging camera.

[0053] In one embodiment, see Fig.14 , step S420 includes the following steps: S421: According to the actual distance between the mutation area and the thermal imaging camera and the scanning distance of the laser radar, it is determined whether the mutation area is within the scanning area of ​​the laser radar. S422: If the mutation area is not within the scanning area of ​​the laser radar, the actual distance between the mutation area and the visible light camera in each subsequent frame of thermal imaging data is continuously calculated until the mutation area is within the scanning area of ​​the laser radar.

[0054] As mentioned above, thermal imaging cameras are more sensitive to temperature. By utilizing this feature, they can detect suspected active obstacles farther away than LiDAR. When the thermal imaging camera identifies a mutation area, the mutation area may not have entered the scanning area of ​​the LiDAR. Therefore, it is necessary to continuously calculate the actual distance between the mutation area and the thermal imaging camera to confirm whether the mutation area is within the scanning area of ​​the LiDAR and how long it will take for the mutation area to enter the scanning area of ​​the LiDAR.

[0055] S500: Fusing the point cloud data on the target area and the thermal imaging data on the target area, performing recognition based on the fusion result, and issuing an early warning based on the recognition result.

[0056] In one embodiment, see Fig. 9 In step S500, the point cloud data on the target area and the thermal imaging data on the target area are fused, and the identification is performed based on the fusion result, which specifically includes the following steps: S510: judging whether there is a target object in the mutation area based on whether there is a change in the thermal imaging data of the mutation area within a unit time; For example, multiple frames of thermal imaging images within a unit time (e.g., one second or several seconds) can be compared to determine whether the mutation area has changed. Generally speaking, under normal circumstances, active obstacles will be in motion in the tunnel, so whether there is a target object (i.e., active obstacle) in the mutation area can be determined by whether the shape of the mutation area has changed.

[0057] S520: If there is a target object in the mutation area, obtaining point cloud data of the mutation area based on the point cloud data on the target area and the thermal imaging data on the target area; In one embodiment, see Fig.10 , step S520 includes the following steps: S521: obtaining a change in thermal imaging data of the mutation area according to the activity traces of the thermal imaging of the mutation area within a unit time period; S522: Based on the thermal imaging change of the mutation area and the point cloud data of the target area within a unit time period, point cloud data about the mutation area is obtained.

[0058] Since the target area has been banned from the laser radar's perception area of ​​the tunnel at this time, the mutation area cannot be accurately identified due to distance and other reasons. Therefore, the division of the mutation area in the perception area needs to rely on the change of thermal imaging data, and the mutation area in the perception area is outlined in reverse from the change of thermal imaging data. Therefore, the change of thermal imaging data obtained in a unit time period can be applied to the perception area of ​​the tunnel to obtain the point cloud data of the mutation area.

[0059] S530: Identify the target object based on the point cloud data of the mutation area.

[0060] In one embodiment, see Fig.11 , step S530 includes the following steps: S531: Based on the thermal imaging data of the target area, a thermal imaging contour in the mutation area based on the thermal imaging data is obtained; Since the thermal imaging images obtained by thermal imaging cameras are not accurate, the data itself does not have location information and cannot be used directly for identification. However, thermal imaging cameras can be used to obtain the mutation area including active obstacles and the outline of the mutation area.

[0061] S532: Based on the point cloud data and the thermal imaging contour of the target area at the same time, obtain a point cloud contour based on the point cloud data in the mutation area; In one embodiment, step S532 includes the following contents: The thermal imaging contour of the mutation area is defined as the first contour; the variance of the first contour in a unit time period is calculated, and it is determined whether the variance of the first contour is within a preset range; if the variance of the first contour is within the preset range, the first contour is defined as the thermal imaging contour; based on the point cloud data and the thermal imaging contour of the target area at the same time, a preliminary contour based on the point cloud data in the mutation area is obtained and defined as the second contour; the variance of the second contour in a unit time period is calculated, and it is determined whether the change of the second contour at different times in the unit time period is within the preset range; if the variance of the second contour is within the preset range, the parameters of the second contour and the parameters of the thermal imaging contour at the same time are compared to determine whether the difference is within the preset range; if the difference between the parameters of the second contour and the parameters of the thermal imaging contour is within the preset range, the second contour is defined as the point cloud contour; In the above embodiment, due to the low accuracy of the thermal imaging image obtained by the thermal imaging camera, the contour information of the active obstacle cannot be accurately obtained, and only the first contour of the mutation area containing the active obstacle can be obtained. However, based on the above first contour and the point cloud data of the target area, the second contour of the mutation area can still be obtained (at this time, it still belongs to the preliminary contour of the mutation area, not the final contour). However, since the active obstacle is already within the range of the thermal imaging camera and the laser radar at the same time, the change characteristics of the active obstacle in the active state are consistent in the thermal imaging data and the point cloud data. In addition, generally speaking, as long as the projection of the limbs does not overlap, whether it is a thermal imaging camera or a laser radar, the length of the contour under different actions is unchanged, only the shape has changed. Based on the above two characteristics, it can be compared whether the length of the first contour has changed / whether the change exceeds expectations based on the movement of the active obstacle in the same time period. If not, it can be temporarily determined that there is no problem with the selection of the above first contour; the second contour also needs to be processed in the same way. If it is temporarily determined that there is no problem with the selection of the second contour. It is necessary to compare whether the change trends of the first contour and the second contour are the same at the same time point. If they are basically the same, the second contour can be defined as a point cloud contour. In the above implementation, the variance is used to reflect the variation of the length of the first contour / the length of the second contour in the same time period. Therefore, the variation may also be reflected in other ways, such as mean difference, etc. This application does not impose any limitation on this.

[0062] S533: Identify the target object based on the point cloud contour.

[0063] After obtaining the point cloud contour of the mutation area, since it matches the motion characteristics of the moving obstacle, the point cloud contour itself can represent the contour information of the moving obstacle. Since the point cloud data itself has position information, the shape covered by the point cloud contour, the length of the contour, etc. can be accurately known through the point cloud contour, and the target object can be identified, specifically, it can be a person, wild boar, pheasant or mouse, etc.

[0064] In one embodiment, the sensing area is divided into at least two sub-areas, including at least one road sub-area and at least one shoulder sub-area, and each sub-area corresponds to a weight coefficient; issuing an early warning based on the recognition result in step S500 specifically includes the following contents: Based on the weight coefficient corresponding to the sub-area of ​​the target area in the perception area and the recognition result of the target area, different types of warning signals are issued.

[0065] The specific information of the target object is obtained, and different warning signals are generated according to the specific information of the target object (such as the type). In addition, the perception area of ​​the tunnel can be divided into at least one road sub-area (the area for vehicles to walk) and at least one shoulder sub-area (the area on the road surface other than the area for vehicles to walk, such as the shoulder). Similarly, it can be divided into two left and right shoulder sub-areas based on the left and right directions, and different weight coefficients can be set. In combination with the type information of the target object, warning signals of different priorities can be issued.

[0066] Exemplarily, if a mouse is detected on the shoulder of the road, the lowest level of warning signal can be issued. Even if a mouse strays into the road, it can pass directly and basically will not cause too much danger. However, if a wild boar is detected on the shoulder of the road, an intermediate warning signal can be issued to remind the driver to pay attention to avoid the wild boar running onto the road due to fright or other reasons. The wild boar is large in size and a collision between the two may cause an accident. In addition, if a person is detected on the shoulder of the road, a high-level warning signal needs to be issued to remind the driver to pay attention to the person on the shoulder of the road ahead. If a person is detected on the road, and according to the speed of the vehicle, the person is about to enter or has entered the danger range, the brake command is directly activated and the driver is notified immediately. Example

[0067] See also Fig.15 The present application embodiment discloses a multi-sensor combined tunnel warning system, which is set on a vehicle through a thermal imaging camera and a laser radar, including: Judgment module: used to judge whether the vehicle enters the tunnel; Thermal imaging perception module: When a vehicle enters a tunnel, the thermal imaging camera is used to identify the tunnel and determine whether there is a sudden change area based on the thermal imaging data of the thermal imaging camera; Point cloud perception module: identifies the tunnel through the laser radar and obtains the perception area of ​​the tunnel based on the point cloud data of the laser radar; Region division module: if the mutation region exists in the perception region, a target region is divided in the perception region, where the target region includes the mutation region; Early warning module: fuses the point cloud data on the target area and the thermal imaging data on the target area, performs recognition based on the fusion results, and issues an early warning based on the recognition results.

[0068] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method of the above embodiment are implemented.

[0069] It should be noted that the embodiments of the present application may involve the collection and use of the geographical environment. In actual applications, the collected data of the geographical environment can be used in the scheme described in this article within the scope permitted by applicable laws and regulations in accordance with the applicable laws and regulations of the country where the application is located (for example, obtaining permission from government departments, etc.).

[0070] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0071] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0072] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0073] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A multi-sensor combined tunnel early warning method, characterized in that: By setting a thermal imaging camera and a laser radar on a vehicle, the method includes: determining whether the vehicle enters a tunnel; If the vehicle enters the tunnel, the tunnel is identified by the thermal imaging camera, and whether there is a mutation area is determined based on the thermal imaging data of the thermal imaging camera; The tunnel is identified by the laser radar, and the perception area of ​​the tunnel is obtained based on the point cloud data of the laser radar; If the mutation region exists in the sensing region, dividing a target region in the sensing region, wherein the target region includes the mutation region; The point cloud data on the target area and the thermal imaging data on the target area are fused, recognition is performed based on the fusion result, and an early warning is issued based on the recognition result.

2. The multi-sensor combined tunnel early warning method according to claim 1 is characterized in that: The step of fusing the point cloud data on the target area with the thermal imaging data on the target area and performing identification based on the fusion result includes: Based on whether there is a change in the thermal imaging data of the mutation area within a unit time, determining whether there is a target object in the mutation area; If there is a target object in the mutation area, obtaining point cloud data of the mutation area based on the point cloud data on the target area and the thermal imaging data on the target area; The target object is identified based on the point cloud data of the mutation area.

3. The multi-sensor combined tunnel early warning method according to claim 2 is characterized in that: The step of obtaining the point cloud data of the mutation area based on the point cloud data of the target area and the thermal imaging data of the target area includes: According to the activity traces of the thermal imaging of the mutation area within a unit time period, the thermal imaging data changes of the mutation area are obtained; Based on the thermal imaging change of the mutation area and the point cloud data of the target area within a unit time period, point cloud data about the mutation area is obtained.

4. The multi-sensor combined tunnel early warning method according to claim 3 is characterized in that: The step of identifying the target object based on the point cloud data of the mutation area includes: Based on the thermal imaging data of the target area, obtaining a thermal imaging contour in a mutation area based on the thermal imaging data; Based on the point cloud data of the target area and the thermal imaging contour at the same time, obtaining a point cloud contour based on the point cloud data in the mutation area; The target object is identified based on the point cloud contour.

5. The multi-sensor combined tunnel early warning method according to any one of claims 1 to 4, characterized in that: The vehicle is also provided with a visible light camera, and the step of determining whether the vehicle has entered a tunnel includes: Determine, based on the image data collected by the visible light camera, whether there is an area in the current frame image where the brightness contrast exceeds a preset range, where the area is defined as a first area; Determining whether the first area is a tunnel entrance; If the first area is a tunnel entrance, whether the vehicle enters the tunnel is determined based on the distance between the vehicle and the first area.

6. The multi-sensor combined tunnel early warning method according to claim 5 is characterized in that: The vehicle is also provided with a second marker, which is provided in a collection area of ​​the visible light camera; and according to the image data collected by the visible light camera, determining whether there is an area in the current frame image where the brightness contrast exceeds a preset range, wherein the area is defined as the first area, comprises: According to the preset distance between the second marker and the visible light camera, an actual distance represented by a unit pixel in the current frame image is obtained, and the actual distance represented by the unit pixel in the current frame image is defined as a second reference distance; Based on the second reference distance and the number of pixels of the first area and the second marker in the current frame image, the actual distance between the first area and the visible light camera is obtained.

7. The multi-sensor combined tunnel early warning method according to claim 5 is characterized in that: Before the step of determining whether the first area is a tunnel entrance, the method further includes: According to the actual distance between the first area in the current frame image and the visible light camera and the scanning distance of the laser radar, confirm whether the first area is within the scanning area of ​​the laser radar; If the first area is not within the scanning area of ​​the laser radar, the actual distance between the first area and the visible light camera in each subsequent frame of image data is continuously calculated until the first area is within the scanning area of ​​the laser radar.

8. The multi-sensor combined tunnel early warning method according to claim 1, 2, 3, 4, 6 or 7, characterized in that: The step of determining whether the vehicle has entered a tunnel comprises: Determine, based on the point cloud data collected by the laser radar, whether there is an area in the current frame point cloud image where the continuous change contrast of the position information exceeds a preset range, and the area is defined as a first area; Determining whether the first area is a tunnel entrance; If the first area is a tunnel entrance, whether the vehicle enters the tunnel is determined according to the distance between the vehicle and the first area.

9. The multi-sensor combined tunnel early warning method according to claim 1, 2, 3, 4, 6 or 7, characterized in that: The sensing area is divided into at least two sub-areas, including at least one road sub-area and at least one shoulder sub-area, and each of the sub-areas corresponds to a weight coefficient; The step of issuing an early warning based on the identification result comprises: Based on the weight coefficient corresponding to the sub-area of ​​the sensing area where the target area is located, combined with the recognition result of the target area, different types of warning signals are issued.

10. The multi-sensor combined tunnel early warning system is characterized by: The thermal imaging camera and lidar are set up on the vehicle, including: Judgment module: used to judge whether the vehicle enters the tunnel; Thermal imaging perception module: after the vehicle enters the tunnel, the tunnel is identified by the thermal imaging camera, and whether there is a mutation area is determined based on the thermal imaging data of the thermal imaging camera; Point cloud perception module: identifies the tunnel through the laser radar and obtains the perception area of ​​the tunnel based on the point cloud data of the laser radar; Region division module: if the mutation region exists in the perception region, a target region is divided in the perception region, wherein the target region includes the mutation region; Early warning module: fuses the point cloud data on the target area with the thermal imaging data on the target area, performs recognition based on the fusion result, and issues an early warning based on the recognition result.

11. A multi-sensor combined tunnel early warning device, characterized in that: Including thermal imaging cameras, LiDAR and industrial computers; The thermal imaging camera, laser radar and industrial computer are arranged on the vehicle; The industrial computer is used to execute the steps of the tunnel early warning method described in any one of claims 1 to 9.

12. A computer storage medium, characterized in that The device stores a computer program, which, when executed, implements the tunnel early warning method according to any one of claims 1 to 9.

Images