A camera-based intelligent roadside system

Abstract

The present invention relates to a camera-based intelligent roadside system, including a processor and a memory, wherein the memory stores a computer program, and the processor calls the computer program to perform operations including: calibrating the camera; obtaining the video data of the camera, and performing target detection on the video data; matching the target detection result with the stored survival trajectory; wirelessly broadcasting the target positioning and tracking result to the outside according to the matching result, and performing log storage; constructing and visualizing local map data according to the target positioning and tracking result; and automatically uploading the target positioning and tracking result and local map data in real time. Compared with the prior art, the present invention can not only provide a law enforcement basis for the enforcement of traffic regulations and reduce the workload of law enforcement personnel, but also assist intelligent vehicles in perception, reduce the occurrence of traffic accidents, and improve the traffic rate at intersections.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent traffic technology, and relates to an intelligent intersection, and in particular to an intelligent roadside system based on a camera. Background Art

[0002] In intelligent driving, it is divided into intelligent vehicle-mounted and intelligent roadside. Intelligent vehicle-mounted is to install a series of high-precision sensors on the vehicle, such as laser radar, millimeter-wave radar, camera, etc., so that the vehicle has a certain perception ability and can give some danger warnings or emergency operations. However, due to the limitation of the sensor's perception range, if you want to have a more accurate perception result, there are certain requirements for the number of sensors installed, which makes the cost of intelligent driving too high. Intelligent roadside, on the other hand, is to install a series of sensors in traffic-intensive sections such as intersections to perceive the road conditions, and then use the wireless communication network to share information to assist vehicles in perception. Compared with vehicle installation, roadside installation sensors have certain advantages due to viewing angles, etc., and through information sharing, they can provide reliable perception results to make up for the vehicle's own perception defects, while reducing the number of vehicle sensors and reducing costs.

[0003] Intelligent roadside is an important part of intelligent transportation and one of the future development directions of intelligent transportation. Introducing vehicle-road cooperative perception into intelligent driving is one of the current hot research directions. However, the current roadside cameras are only used to achieve simple monitoring or take pictures of illegal behaviors of illegal vehicles, and do not provide direct and substantial effects on reducing the occurrence of accidents. Therefore, it is very necessary to design a more useful and reliable monitoring solution for intersections and other traffic-intensive sections. Summary of the invention

[0004] The purpose of the present invention is to provide a camera-based intelligent roadside system in order to overcome the defects of the above-mentioned prior art.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A camera-based intelligent roadside system includes a processor and a memory, wherein the memory stores a computer program, and the processor calls the computer program to perform operations including:

[0007] Calibrate the camera;

[0008] Acquire video data from a camera, and perform target detection on the video data;

[0009] Matching the target detection result with the stored survival trajectory;

[0010] According to the matching results, the target positioning and tracking results are wirelessly broadcasted and log stored;

[0011] Construct and visualize local map data according to the target positioning and tracking results;

[0012] The target positioning and tracking results and local map data are automatically uploaded in real time.

[0013] Furthermore, a plurality of cameras may be provided, with one camera being provided for each direction of each intersection.

[0014] Furthermore, a convolutional neural network is used to perform the target detection.

[0015] Furthermore, matching the target detection result with the stored survival trajectory specifically includes:

[0016] Extract features of detected targets;

[0017] Update the time of the survival trajectory and calculate the current position of the trajectory target. The survival trajectory includes the confirmed trajectory and the potential trajectory.

[0018] A similarity measurement is performed based on the extracted features and the historical features of the survival trajectory to obtain a matching result, where the matching result includes a successful match, an unmatched target, or an unmatched trajectory.

[0019] The features include motion features and appearance features.

[0020] Furthermore, when performing the similarity measurement, the extracted features are first measured for similarity with the historical features of the determined trajectory. If the matching result is that the target is not matched, the extracted features are then measured for similarity with the historical features of the potential trajectory.

[0021] Furthermore, a Kalman filter algorithm is used to estimate the position of the trajectory target of the survival trajectory.

[0022] Furthermore, the operations executed by the processor calling the computer program further include:

[0023] The survival track is updated.

[0024] Furthermore, updating the survival trajectory is specifically as follows:

[0025] When the matching result is a successful match, if the successfully matched survival trajectory is a confirmed trajectory, the historical feature sequence is updated. If the successfully matched survival trajectory is a potential trajectory, it is determined whether the survival reading is greater than TL. If so, the disappearance count is cleared to 0. If not, it is determined to be a false trajectory, where TL refers to the survival threshold;

[0026] If the matching result is an unmatched target, a new potential trajectory is initialized;

[0027] If the matching result is an unmatched trajectory, if the unmatched trajectory is a confirmed trajectory, the disappearance count is increased by 1. When the disappearance count is less than the set count threshold, the confirmed trajectory is converted to a potential trajectory, otherwise the confirmed trajectory is converted to a dead trajectory. If not, it is determined to be a false trajectory.

[0028] Furthermore, when the number of historical features in the historical feature sequence reaches an upper limit, the oldest feature is deleted.

[0029] Furthermore, the targets include pedestrians and vehicles.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. Compared with traditional video surveillance, the present invention adds target detection and tracking, message broadcasting, cloud services and other functions, and can provide a series of intelligent services while assisting vehicle perception. By constructing the intelligent roadside in this solution, it can effectively improve the perception ability of intelligent vehicles, reduce the occurrence of traffic accidents, and improve traffic efficiency. At the same time, it can also feedback and record road condition information, making traffic management more intelligent.

[0032] 2. The present invention can assist perception through the roadside, solving the problem that the perception cost of existing smart vehicles is too high, or the perception cannot effectively solve the problem of occlusion.

[0033] 3. The present invention classifies target trajectories and processes each type of trajectory differently, making it more convenient to process each trajectory. The effective target trajectory that needs to be maintained is determined according to the change of the trajectory, thereby reducing the amount of data.

[0034] 4. The present invention can perceive the environment of the current road section to a certain extent, and use wireless communication to feed back relevant information to passing vehicles, assisting vehicle perception, so that vehicles have an early perception of possible dangerous accidents and take danger avoidance operations in advance, which plays a certain role in reducing the occurrence of accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the present invention;

[0036] Figure 2 It is the target trajectory matching flow chart of the present invention;

[0037] Figure 3 It is a flow chart of trajectory processing of the present invention;

[0038] Figure 4 It is a hardware architecture diagram of the present invention;

[0039] Figure 5 It is a software architecture diagram of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] The present invention provides a camera-based intelligent roadside system, including a processor and a memory, wherein the memory stores a computer program, and the processor calls the computer program to perform operations including: calibrating the camera; obtaining the video data of the camera, and performing target detection on the video data; matching the target detection result with the stored survival trajectory; wirelessly broadcasting the target positioning and tracking result based on the matching result, and performing log storage; constructing and visualizing local map data based on the target positioning and tracking result; and automatically uploading the target positioning and tracking result and local map data in real time. The working principle of the intelligent roadside system is as follows: Figure 1 As shown, it realizes the interconnection between vehicle, road and cloud, assists vehicle perception and intelligent cloud services, and can realize the following functions:

[0042] 1. Provide dynamic perception services. By using cameras as perception devices, the neural network method is used to detect and track regional targets in real time, and filtering and other methods are used to make the positioning and tracking results of targets more accurate. By communicating with vehicles, it assists vehicle perception, improves vehicle perception capabilities, and improves vehicle safety performance.

[0043] 2. Provide log service. In addition to providing basic operation logs, log storage can also automatically record illegal driving or illegal behaviors based on the perception results and store them by category, which is convenient for future retrieval and improves law enforcement efficiency.

[0044] 3. Provide visualization services, build local map data based on tracking results, and conduct real-time mapping and analysis of the data to facilitate dynamic adjustment of traffic flow.

[0045] 4. Provide cloud services, which can automatically upload relevant data to the cloud in real time and share the data on the network. Vehicle users can subscribe to relevant cloud service information, understand traffic conditions in real time, and then implement travel plans to improve travel efficiency.

[0046] In some embodiments, multiple cameras may be provided, with one camera being provided for each direction of each intersection.

[0047] In some embodiments, the intelligent roadside system uses cameras as sensing devices and roadside computing units as data processing centers, and utilizes wireless communication modules and network modules to achieve data broadcasting and intelligent uploading, thereby realizing functions such as assisting vehicle perception and intelligent cloud services.

[0048] The camera in the present invention can be a common network camera or an industrial camera; the computing unit can be an industrial computer or an edge server, which can provide powerful image processing capabilities to ensure the processing delay of the entire program; the wireless communication module can be DSRC, or special vehicle communication equipment such as LTE-V; the network module can be an optical fiber or a direct network cable connection.

[0049] In some embodiments, a convolutional neural network is used to achieve real-time detection of targets such as pedestrians and vehicles.

[0050] like Figure 2 As shown, matching the target detection result with the stored survival trajectory specifically includes:

[0051] Extracting features of the detected target, wherein the features include motion features and appearance features;

[0052] The Kalman filter algorithm is used to update the time of the surviving trajectory and estimate the current position of the trajectory target. The surviving trajectory includes the confirmed trajectory and the potential trajectory.

[0053] A similarity measurement is performed based on the extracted features and the historical features of the surviving trajectory to obtain a matching result, wherein the matching result includes a successful match, an unmatched target, or an unmatched trajectory, and the similarity measurement includes motion feature similarity and appearance similarity.

[0054] When performing similarity measurement, the extracted features are first measured with the historical features of the determined trajectory. If the matching result is an unmatched target, the extracted features are then measured with the historical features of the potential trajectory.

[0055] Matching result processing includes single-camera matching result trajectory processing and multi-camera tracking result fusion. Multi-camera tracking result fusion can remove redundancy based on appearance and motion position information to ensure accurate intersection information. The final multi-target tracking results can be used to achieve a series of intelligent functions, including cloud connection, visual analysis, and wireless data broadcasting.

[0056] In another embodiment, the operation executed by the processor calling the computer program further includes: updating the surviving trajectory. The trajectories involved in the present invention include confirmed trajectories, potential trajectories and disappeared trajectories. Among them, for the processing of confirmed trajectories, it is necessary to update the motion and appearance state information of the trajectories; for the processing of potential trajectories, it is necessary to distinguish between false trajectories and newly appeared trajectories, which can be distinguished by the number of continuous detection frames; for the processing of disappeared trajectories, it is necessary to distinguish between temporary occlusion disappearance and leaving the monitoring area, and the disappearance time judgment can be set.

[0057] like Figure 3 As shown, the specific process of updating the survival trajectory is:

[0058] When the matching result is a successful match, if the successfully matched survival trajectory is a confirmed trajectory, the historical feature sequence (feature_list) is updated. When the number of historical features in the historical feature sequence reaches the upper limit, the oldest feature (i.e. the earliest feature) is deleted. If the successfully matched survival trajectory is a potential trajectory, it is determined whether the survival reading is greater than TL. If so, it means that it is already a new trajectory and no disappearance count is required. The disappearance count is cleared to 0. If not, the trajectory only appears briefly and is determined to be a false trajectory. TL (Threath Life) refers to the survival threshold, and a false trajectory refers to an abnormal position point or a situation where the target ID is not the same as the original one.

[0059] If the matching result is an unmatched target, a new potential trajectory is initialized;

[0060] If the matching result is an unmatched trajectory, if the unmatched trajectory is a confirmed trajectory, the disappearance count is increased by 1. When the disappearance count is less than the set count threshold, the confirmed trajectory is converted to a potential trajectory, otherwise the confirmed trajectory is converted to a dead trajectory. If not, it is determined to be a false trajectory.

[0061] In the above trajectory update process, if the survival time of the potential trajectory is greater than the survival threshold

[0062] Example

[0063] like Figure 4 , Figure 5 For a scene at an intersection, take a 4-way intersection as an example. A camera of the same specification is installed in each direction to monitor the corresponding direction area.

[0064] The specific working process of the camera-based intelligent roadside system of this embodiment includes:

[0065] In the first step, IP addresses are assigned to all cameras at the intersection, and all cameras are calibrated. Based on the calibration points and camera parameters, the mapping relationship between the camera's GPS and pixel coordinates is obtained;

[0066] In the second step, all current image frames are transmitted to the computing unit on the roadside, and the current target information is output using a convolutional neural network.

[0067] In the third step, based on the detection information, the feature information of all current targets is extracted, including the target's motion features and appearance features;

[0068] In the fourth step, similarity measurements are performed on the target and the trajectory, including motion feature similarity and appearance similarity;

[0069] In the fifth step, the target and trajectory are matched according to the feature similarity, as follows: Figure 2 As shown, the current matching results are output;

[0070] In the sixth step, the matching results in the fifth step are processed, as follows Figure 3 As shown;

[0071] In the seventh step, the functional modules are implemented for the current target trajectory information, including cloud connection services, visual analysis, log storage, and wireless data broadcasting;

[0072] In the eighth step, the survival trajectory information is updated, the video information of the next frame is read, and a new round of target tracking iterative update is started.

[0073] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A camera-based intelligent roadside system, It is characterized in that The system comprises a processor and a memory, wherein the memory stores a computer program, and the operations performed by the processor calling the computer program include: Calibrate the camera; Acquire video data from a camera, and perform target detection on the video data; Matching the target detection result with the stored survival trajectory; According to the matching results, the target positioning and tracking results are wirelessly broadcasted and log stored; Construct and visualize local map data according to the target positioning and tracking results; Automatically upload the target location tracking results and local map data in real time; Matching the target detection result with the stored survival trajectory specifically includes: Extract features of detected targets; Update the time of the survival trajectory and calculate the current position of the trajectory target, wherein the survival trajectory includes a confirmed trajectory and a potential trajectory; Performing similarity measurement based on the extracted features and the historical features of the survival trajectory to obtain a matching result, wherein the matching result includes a successful match, an unmatched target, or an unmatched trajectory; The operations executed by the processor calling the computer program further include: Updating the survival trajectory; The specific updating of the survival trajectory is as follows: When the matching result is a successful match, if the successfully matched survival trajectory is a confirmed trajectory, the historical feature sequence is updated. If the successfully matched survival trajectory is a potential trajectory, it is determined whether the survival reading is greater than TL. If so, the disappearance count is cleared to 0. If not, it is determined to be a false trajectory, where TL refers to the survival threshold; If the matching result is an unmatched target, a new potential trajectory is initialized; If the matching result is an unmatched trajectory, if the unmatched trajectory is a confirmed trajectory, the disappearance count is increased by 1. When the disappearance count is less than the set count threshold, the confirmed trajectory is converted to a potential trajectory, otherwise the confirmed trajectory is converted to a dead trajectory. If the unmatched trajectory is not a confirmed trajectory, it is determined to be a false trajectory.

2. The camera-based intelligent roadside system according to claim 1, It is characterized in that The target detection is performed using a convolutional neural network.

3. The camera-based intelligent roadside system according to claim 1, It is characterized in that The features include motion features and appearance features.

4. The camera-based intelligent roadside system according to claim 1, It is characterized in that When performing the similarity measurement, the extracted features are first measured for similarity with the historical features of the determined trajectory. If the matching result is that the target is not matched, the extracted features are then measured for similarity with the historical features of the potential trajectory.

5. The camera-based intelligent roadside system according to claim 1, It is characterized in that The Kalman filter algorithm is used to estimate the position of the trajectory target of the survival trajectory.

6. The camera-based intelligent roadside system according to claim 1, It is characterized in that When the number of historical features in the historical feature sequence reaches an upper limit, the oldest feature is deleted.

7. The camera-based intelligent roadside system according to any one of claims 1 to 6, It is characterized in that The targets include pedestrians and vehicles.

Images