A laser-visual integrated machine for holographic intersection traffic management

Abstract

The invention discloses a radar-vision integrated machine for holographic intersection traffic management, which relates to the field of intelligent intersection traffic management. The system comprises: a camera group, a main control machine, and a radar sensor. Through a combination configuration mode of a global exposure camera, a near-view camera, and a far-view camera, a configuration method for near-view and far-view image acquisition is adopted. The target in the acquired camera picture changes from large to small, and target detection and recognition are performed on the near-view and far-view images respectively, and then the target is subjected to multi-camera fusion processing, which can improve the accuracy of target detection and facilitate the tracking processing of the target. At the same time, a radar sensor is used to obtain the speed information and position information of each moving target from the stop line of the intersection to the end of the road, and the speed information and position information are fused with the target data detected and recognized by the camera, which can further improve the accuracy of target recognition and tracking.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic management, and in particular to a radar-vision integrated machine for holographic intersection traffic management. Background Art

[0002] Vehicle-road collaboration is an inevitable trend in the development of traffic control, and urban intersections are important nodes for vehicle-road collaboration. With the rise of the concept of holographic intersections, major software and hardware manufacturers have made plans. Holographic intersections can provide high-precision, real-time traffic data, and are an important means to promote the refined implementation of intelligent traffic management and fully empower urban traffic management. It uses a variety of sensors to perceive road conditions from multiple perspectives and modalities, accurately perceive traffic conditions, and finally digitally restore the complex state of intersections to achieve on-demand scheduling and intelligent management of road rights. In each module of the holographic intersection, perception is the first step, and the configuration of the sensor directly affects the subsequent data processing and algorithm flow.

[0003] At present, the main method of traffic management at holographic intersections is a multi-camera pure vision solution, which uses cameras of different focal lengths and resolutions and relevant algorithms in the field of computer vision to complete tasks such as target detection, classification, tracking, and re-identification. However, the existing multi-camera pure vision solution is strongly affected by lighting when the camera collects images. It is affected by factors such as night, strong light, rain, snow, and smoke. The visibility is poor when the environmental conditions are poor, and it does not have the characteristics of all-day and all-weather. As a result, the efficiency and accuracy of traffic management at existing holographic intersections are low. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a radar-vision integrated machine for traffic management at holographic intersections, so as to solve the problems of low efficiency and accuracy of traffic management at existing holographic intersections.

[0005] To achieve the above-mentioned purpose, the present invention provides a radar-visual integrated machine for holographic intersection traffic management, the radar-visual integrated machine comprising: a camera group, a main control machine, and a radar sensor;

[0006] The camera group includes: a global exposure camera, a close-up camera, and a long-range camera;

[0007] The global exposure camera is used to collect images of the entire intersection area corresponding to the electric police pole or the traffic light pole;

[0008] The close-range camera is used to collect images of the entire intersection area corresponding to the electric police pole or the traffic light pole and the vehicle waiting area on the opposite road;

[0009] The long-range camera is used to collect images of the vehicle waiting area on the opposite road corresponding to the electric alarm pole or the traffic light pole and images of the road area extending to the far end;

[0010] The radar sensor is used to collect speed information and position information of each moving target from the stop line at the intersection to the end of the far road;

[0011] The main control machine is used to perform target recognition on the images of different areas collected by the global exposure camera, the near-view camera, and the far-view camera to obtain identification information, feature information, and pixel position information of each target in the image;

[0012] The main control machine is further used to match the target recognition results corresponding to the global exposure camera, the near-view camera, and the far-view camera to establish the target's track information, wherein the track information includes the target's identification information, feature information, and the target's real position information, wherein the target's real position information is obtained based on the pixel position information of the target in the picture;

[0013] The main control machine is also used to fuse the track information of each target obtained by the camera group and the speed information and position information of each target obtained by the radar sensor, and perform traffic management based on the fused data.

[0014] Furthermore, the main control machine is specifically used to perform target recognition on the image of the entire intersection area corresponding to the electric police pole or the traffic light pole collected by the global exposure camera, and obtain the identification information, feature information and pixel position information of the target in the image;

[0015] The main control machine is further used to perform target recognition on the images of the entire intersection area and the vehicle waiting area of ​​the opposite road corresponding to the electric police pole or the traffic light pole collected by the close-range camera, and match the target detected by the global exposure camera through a preset image stitching and re-recognition algorithm according to the recognition result to establish track information, wherein the track information includes the identification information, feature information and real position information of the target, and the real position information of the target is obtained according to the pixel position information of the target in the image;

[0016] The main control machine is also specifically used to detect and identify distant targets based on the vehicle waiting area images and road area images of the opposite road corresponding to the electric police poles or traffic light poles collected by the long-range camera, and match the recognition results with the target recognition results corresponding to the close-range camera through a preset image stitching algorithm to update the track information.

[0017] Furthermore, the main control machine is specifically used to obtain the distortion coefficient of each camera according to the compensation parameters of each camera in the camera group.

[0018] Furthermore, the main control machine is specifically used to obtain the perspective transformation coefficient of each camera by selecting four non-collinear pixel points in the picture collected by each camera and the corresponding physical coordinates in the world coordinate system.

[0019] Furthermore, the main control machine is specifically used to obtain the real position information of the target according to the distortion coefficient, the perspective transformation coefficient and the pixel position information of the target in the picture.

[0020] Furthermore, the integrated radar and video machine further comprises: a configuration unit;

[0021] The configuration unit is used to time-calibrate the image data collected by each camera and synchronize the timestamps corresponding to the image data collected by different cameras; spatially calibrate the installation position of the radar sensor and align it with the lane information of the monitoring area according to the correction position error coefficient.

[0022] Furthermore, the main control machine is also used to perform point cloud segmentation and clustering on the data of each moving target from the stop line at the intersection to the end of the road collected by the radar sensor to obtain speed information and position information of each target.

[0023] Furthermore, the installation sight angle of the integrated radar and vision device is 3-15 degrees downward from the horizontal direction.

[0024] Furthermore, the angle between the center of the field of view of the global exposure camera and the close-range camera and the horizontal line is in the range of 10-50 degrees.

[0025] Furthermore, the angle between the center of the vision camera's field of view and the horizontal line ranges from about 5 to 45 degrees.

[0026] The present invention provides a radar-vision integrated machine for holographic intersection traffic management. Through a combination configuration of multiple cameras including a global exposure camera, a near-view camera, and a far-view camera, a configuration method for capturing near- and far-view images is adopted. The targets in the captured camera images change from large to small, and target detection and recognition are performed on the near-view and far-view images respectively. The targets are then subjected to multi-camera fusion processing, which can improve the accuracy of target detection and facilitate target tracking. At the same time, a radar sensor is used to obtain the speed information and position information of each moving target from the stop line of the intersection to the end of the road, and the information is fused with the target data detected and recognized by the camera, which can further improve the accuracy of target recognition and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the scene acquisition of the integrated radar and vision device provided by the present invention;

[0028] Figure 2 It is a structural schematic diagram of a radar-vision integrated machine for holographic intersection traffic management provided by the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0030] The embodiment of the present invention provides a laser-visual integrated device for holographic intersection traffic management, such as Figure 2 As shown, the system includes: a camera group 21, a main control machine 22, and a radar sensor 23;

[0031] The camera group includes: a global exposure camera 2101, a near-view camera 2102, and a far-view camera 2103; Figure 1 As shown, the radar-vision integrated machine is usually installed on an electric police pole or traffic light pole at a height of about 5-7m. The electric police pole is a pole where electronic police or electronic eyes are installed, which is generally opposite to the traffic light pole. The angle between the line of sight of the radar-vision integrated machine and the horizontal plane is usually 3-15°, at which the intersection area can be observed well; the mainstream frequency of the radar sensor is 77G-80G, which meets the distance and distance measurement of motor vehicles in a scene with 8 lanes in both directions and a longitudinal distance of 350m. The focal length of the lens of the global exposure camera, the long-range camera and the close-range camera is generally 6-20mm. The angle between the center of the field of view of the global exposure camera and the close-range camera and the horizontal line is about 10-50 degrees; the angle between the center of the field of view of the long-range camera and the horizontal line is about 5-45 degrees. In terms of azimuth, the camera group can meet the coverage of 8 lanes in both directions. In terms of longitudinal distance, the visible range of the global exposure camera is 20-200m from the installation pole; the visible range of the near-view camera is about 20-300m from the installation pole; the visible range of the far-view camera is about 100-350m from the installation pole. In order to improve the accuracy of detection, multiple cameras are used for different tasks such as motor vehicle information recognition and near-view and far-view target detection. Among them: the global exposure camera is responsible for collecting targets in the intersection area, 15m-50m away from the installation position, the near-view camera is responsible for collecting targets in the vehicle waiting area of ​​the intersection and the opposite road, 20m-150m away from the installation position, and the far-view camera mainly collects vehicles from the vehicle waiting area to the end of the road, 150m-250m away from the installation position, so as to achieve full-range detection, recognition and tracking of targets from near to far.

[0032] The global exposure camera 2101 is used to collect images of the entire intersection area corresponding to the electric police pole or the traffic light pole.

[0033] The close-range camera 2102 is used to collect images of the entire intersection area corresponding to the electric police pole or the traffic light pole and the vehicle waiting area on the opposite road.

[0034] The long-range camera 2103 is used to collect images of the vehicle waiting area on the opposite road corresponding to the electric alarm pole or the traffic light pole and images of the road area extending to the far end.

[0035] The radar sensor 23 is used to collect information such as speed, acceleration, position, RCS, etc. of each moving target from the stop line at the intersection to the end of the far road.

[0036] The main control computer 22 is used to perform target recognition on the images of different areas captured by the global exposure camera, the near-view camera, and the far-view camera to obtain identification information, feature information, and pixel position information of each target in the image.

[0037] The main control computer 22 is also used to match the target recognition results corresponding to the global exposure camera, the near-view camera, and the far-view camera to establish the target's track information, wherein the track information includes the target's identification information, feature information, and the target's real position information, and the target's real position information is obtained based on the target's pixel position information in the image.

[0038] The main control machine 22 is also used to fuse the track information of each target obtained by the camera group and the speed information and position information of each target obtained by the radar sensor, and perform traffic management based on the fused data.

[0039] Specifically, after completing the preprocessing of radar and camera data, the timestamps of the two are aligned, and the radar data are associated using the global nearest neighbor, probability association algorithm, Kuhn-Munkras algorithm, etc., to match the data of multiple sensors. After matching the sensor data at a single intersection, multi-intersection fusion and global track management can also be performed.

[0040] Furthermore, the main control machine 22 is specifically used to perform target recognition on the image of the entire intersection area corresponding to the electric police pole or the traffic light pole collected by the global exposure camera, and obtain the identification information, feature information and pixel position information of the target in the image; the main control machine 22 is also specifically used to perform target recognition on the image of the entire intersection area and the vehicle waiting area of ​​the opposite road corresponding to the electric police pole or the traffic light pole collected by the close-range camera, and match the recognition result with the target detected by the global exposure camera through a preset image stitching and re-recognition algorithm to establish track information, wherein the track information includes the identification information, feature information and real position information of the target, and the real position information of the target is obtained according to the pixel position information of the target in the image; the main control machine 22 is also specifically used to detect and recognize distant targets on the image of the vehicle waiting area of ​​the opposite road and the road area image corresponding to the electric police pole or the traffic light pole collected by the long-range camera, and match the recognition result with the target recognition result corresponding to the close-range camera through a preset image stitching algorithm to update the track information.

[0041] Specifically, for example, first identify the target information of the picture captured by the global exposure camera. Such as license plate information, body color and model, etc.; then detect the target in the picture captured by the close-range camera and match it with the target detected by the global exposure camera. Track the target, assign the target ID and establish track information. Use image stitching and re-identification algorithms to match the vehicle ID detected by the close-range camera with the target detected by the global exposure camera, and assign relevant information such as the license plate, color and model of the corresponding ID vehicle. Finally, use the long-range camera to detect distant targets and match them with the targets detected by the close-range camera. Generally, the picture captured by the long-range camera is a local enlargement of the far area of ​​the picture captured by the close-range camera. Therefore, the target frame captured in the long-range camera can be transformed into the close-range camera using the image stitching method.

[0042] Furthermore, the main control machine 22 is specifically used to obtain the distortion coefficient of each camera according to the compensation parameters of each camera in the camera group. The perspective transformation coefficient of each camera is obtained by selecting four non-collinear pixel points in the picture collected by each camera and the corresponding physical coordinates in the world coordinate system. The real position information of the target is obtained according to the distortion coefficient, the perspective transformation coefficient and the pixel position information of the target in the picture.

[0043] Specifically, for example, the camera distortion correction parameters can be solved by the Zhang Zhengyou calibration method or the compensation parameters given by the manufacturer. The perspective transformation coefficient is solved by selecting four non-collinear pixel points in the captured image and the corresponding physical coordinates in the world coordinate system. The real position of the target is obtained using the distortion coefficient, perspective transformation coefficient and detection pixels. In general, the detection algorithm based on deep learning is performed in the original image captured by the camera without distortion correction, and the pixel position of the target in the image is obtained. Therefore, after the distortion position compensation is completed using the correction coefficient, the perspective transformation coefficient is used to map the target pixel to the bird's-eye view.

[0044] Furthermore, the radar-vision integrated machine also includes: a configuration unit; the configuration unit is used to time-calibrate the image data collected by each camera and synchronize the timestamps corresponding to the image data collected by different cameras; the installation position of the radar sensor is spatially calibrated, and aligned with the lane information of the monitoring area according to the correction position error coefficient. It should be noted that due to the encoding and decoding of the video and the transmission delay, the timestamps of each camera in the camera group are generally not synchronized. First, the camera is timed, and after the collected picture is transmitted to the edge calculator, the timestamps of the collected data are time-aligned to ensure that the data to be processed by each camera at each moment is the same frame of data.

[0045] Furthermore, the main control computer 22 is also used to perform point cloud segmentation and clustering on the data of each moving target from the stop line at the intersection to the end of the road collected by the radar sensor to obtain speed information and position information of each target.

[0046] The present invention provides a radar-vision integrated machine for holographic intersection traffic management. Through a combination configuration of multiple cameras including a global exposure camera, a near-view camera, and a far-view camera, a configuration method for capturing near- and far-view images is adopted. The targets in the captured camera images change from large to small, and target detection and recognition are performed on the near-view and far-view images respectively. The targets are then subjected to multi-camera fusion processing, which can improve the accuracy of target detection and facilitate target tracking. At the same time, a radar sensor is used to obtain the speed information and position information of each moving target from the stop line of the intersection to the end of the road, and the information is fused with the target data detected and recognized by the camera, which can further improve the accuracy of target recognition and tracking.

[0047] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0048] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0049] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0050] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0051] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0052] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0053] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0054] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A laser-visual integrated machine for holographic intersection traffic management, characterized in that: The integrated radar and visual device is configured on an electric alarm pole or a traffic light pole at an intersection, and includes: a camera group, a main control machine, and a radar sensor; The camera group includes: a global exposure camera, a close-up camera, and a long-range camera; The global exposure camera is used to collect images of the entire intersection area corresponding to the electric police pole or the traffic light pole; The close-range camera is used to collect images of the entire intersection area corresponding to the electric police pole or the traffic light pole and the vehicle waiting area on the opposite road; The long-range camera is used to collect images of the vehicle waiting area on the opposite road corresponding to the electric alarm pole or the traffic light pole and images of the road area extending to the far end; The radar sensor is used to collect information about various moving targets from the stop line at the intersection to the end of the far road; The main control machine is used to perform target recognition on the images of different areas collected by the global exposure camera, the near-view camera, and the far-view camera to obtain identification information, feature information, and pixel position information of each target in the image; The main control machine is further used to match the target recognition results corresponding to the global exposure camera, the near-view camera, and the far-view camera to establish the target's track information, wherein the track information includes the target's identification information, feature information, and the target's real position information, wherein the target's real position information is obtained based on the pixel position information of the target in the picture; The main control machine is further used to fuse the track information of each target obtained by the camera group and the speed information and position information of each target obtained by the radar sensor, and perform traffic management based on the fused data; The main control machine is specifically used to perform target recognition on the image of the entire intersection area corresponding to the electric police pole or the traffic light pole collected by the global exposure camera, and obtain the vehicle identification information, feature information and pixel position information of the target in the image; The main control machine is further used to perform target recognition on the images of the entire intersection area and the vehicle waiting area of ​​the opposite road corresponding to the electric police pole or the traffic light pole collected by the close-range camera, and match the target detected by the global exposure camera through a preset image stitching and re-recognition algorithm according to the recognition result to establish track information, wherein the track information includes the identification information, feature information and real position information of the target, and the real position information of the target is obtained according to the pixel position information of the target in the image; The main control machine is also specifically used to detect and identify distant targets based on the vehicle waiting area images of the opposite road corresponding to the electric warning pole or traffic light pole and the road area images extending to the far end collected by the long-range camera, and match the recognition results with the target recognition results corresponding to the close-range camera through a preset image stitching algorithm to update the track information.

2. The integrated laser and visual device for holographic intersection traffic management according to claim 1, characterized in that: The main control machine is specifically used to obtain the distortion coefficient of each camera according to the compensation parameters of each camera in the camera group.

3. The integrated laser and visual device for holographic intersection traffic management according to claim 2, characterized in that: The main control machine is also specifically used to obtain the perspective transformation coefficient of each camera by selecting four non-collinear pixel points in the picture collected by each camera and the corresponding physical coordinates in the world coordinate system.

4. The integrated laser and visual device for holographic intersection traffic management according to claim 3, characterized in that: The main control machine is further used to obtain the real position information of the target according to the distortion coefficient, the perspective transformation coefficient and the pixel position information of the target in the picture.

5. The integrated laser and visual device for holographic intersection traffic management according to claim 1, characterized in that: The integrated radar and video machine also includes: a configuration unit; The configuration unit is used to time-calibrate the image data collected by each camera and synchronize the timestamps corresponding to the image data collected by different cameras; spatially calibrate the installation position of the radar sensor and align it with the lane information of the monitoring area according to the correction position error coefficient.

6. The integrated laser and visual device for holographic intersection traffic management according to claim 1, characterized in that: The main control machine is also specifically used to perform point cloud segmentation and clustering on the data of each moving target from the stop line at the intersection to the end of the road collected by the radar sensor, so as to obtain the speed information and position information of each target.

7. A laser-visual integrated machine for holographic intersection traffic management according to any one of claims 1 to 6, characterized in that: The installation angle of the integrated radar and vision machine is 3-6 degrees downward from the horizontal direction.

8. A laser-visual integrated machine for holographic intersection traffic management according to any one of claims 1 to 6, characterized in that: The angle between the center of the field of view of the global exposure camera and the close-range camera and the horizontal line is in the range of 15-45 degrees.

9. A laser-visual integrated machine for holographic intersection traffic management according to any one of claims 1 to 6, characterized in that: The angle between the center of the vision camera's field of view and the horizontal line is in the range of 10-30 degrees.

Images