Lane congestion detection method, device and equipment and storage medium

Through the analysis and integration of radar and video detection data, the accuracy of lane congestion detection in complex environments is solved, and the accuracy and reliability of lane congestion detection are improved.

CN119091614APending Publication Date: 2024-12-06NANJING HURYS INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411076562.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Under complex environmental conditions such as severe weather, dense buildings or electromagnetic interference, existing radar and visual tracking algorithms find it difficult to accurately detect lane congestion status, resulting in a decrease in lane congestion detection effect.

Method used

By analyzing the data of the radar detection area and the video detection area, the respective correlation coefficients are calculated and integrated to obtain the congestion coefficient of the lane, and the congestion degree of the lane is determined based on the preset congestion threshold.

Benefits of technology

It improves the accuracy of lane congestion detection, reduces the impact of environmental factors on the detection results, and can more accurately represent the level of lane congestion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091614A_ABST
    Figure CN119091614A_ABST
Patent Text Reader

Abstract

The invention discloses a lane congestion detection method, device and equipment and a storage medium, and the method comprises the steps: carrying out the analysis of radar data of a radar detection target in a radar detection region, and obtaining a plurality of pieces of first data used for representing the trajectory of the radar detection target, analyzing the video data of the video detection target in the video detection area to obtain a plurality of second data used for representing the track of the video detection target; determining a first correlation coefficient among the plurality of first data based on the plurality of first data, determining a second correlation coefficient among the plurality of second data based on the plurality of second data, the first correlation coefficient being used for describing a lane congestion condition in the radar detection area, and the second correlation coefficient being used for describing a lane congestion condition in the video detection area; integrating the first correlation coefficient and the second correlation coefficient to obtain a congestion coefficient of the lane; and based on the congestion coefficient and a preset congestion threshold value, determining a congestion detection result of the lane so as to improve the accuracy of congestion detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a lane congestion detection method, device, equipment and storage medium. Background Art

[0002] Target tracking refers to the use of sensors and related algorithms to detect and locate the position and movement of an object in space. For example, if the sensor is a radar sensor, the detection device equipped with the radar sensor can track the target and obtain its motion trajectory. The related algorithm is a visual tracking algorithm, which can detect, extract, identify and track the moving target in the image sequence to obtain the motion parameters of the moving target, and analyze and process the motion parameters to understand the behavior of the moving target.

[0003] Based on this, in practical applications, taking traffic detection scenarios as an example, taking the vehicles in the lane as the target, and tracking the target through radar detection equipment or visual tracking algorithms, it is possible to determine whether the lane is currently congested or other traffic situations. However, due to the influence of surrounding environmental factors, especially in complex environmental conditions such as bad weather, dense buildings or electromagnetic interference, the detection effect of the radar detection equipment installed on the lane on the vehicle, that is, the target tracking effect, will be greatly reduced. When using the visual tracking algorithm, the video data obtained by the traffic camera installed on the lane may also be affected by factors such as bad weather or changes in lighting, resulting in the subsequent further data analysis through the visual tracking algorithm to be less than ideal. Therefore, in traffic detection scenarios, existing target tracking methods often cannot accurately detect the congestion status of the lane. Summary of the invention

[0004] Embodiments of the present application provide a lane congestion detection method, apparatus, device, and storage medium to improve the accuracy of lane congestion detection.

[0005] In a first aspect, an embodiment of the present application provides a lane congestion detection method, comprising:

[0006] Analyzing radar data of a radar detection target within a radar detection area to obtain a plurality of first data for representing a trajectory of the radar detection target, and analyzing video data of a video detection target within a video detection area to obtain a plurality of second data for representing the trajectory of the video detection target, wherein the radar detection target is a target within the radar detection area of ​​the lane, and the video detection target is a detection target within the video detection area of ​​the lane, and the radar detection area and the video detection area are unified;

[0007] Determine a first correlation coefficient between the plurality of first data based on the plurality of first data, and determine a second correlation coefficient between the plurality of second data based on the plurality of second data, wherein the first correlation coefficient is used to describe a lane congestion condition within the radar detection area, and the second correlation coefficient is used to describe a lane congestion condition within the video detection area;

[0008] Integrating the first correlation coefficient and the second correlation coefficient to obtain a congestion coefficient of the lane;

[0009] Based on the congestion coefficient and a preset congestion threshold, a congestion detection result of the lane is determined.

[0010] Optionally, analyzing the video data corresponding to the video detection target to obtain a plurality of second data for representing the trajectory of the video detection target includes:

[0011] Determine a pixel area of ​​a first region in each frame image of the video data and a pixel area of ​​a second region in each frame image, wherein the first region is a region enclosed by an edge of the video detection target, and the second region is a region of interest in the video detection region;

[0012] Calculating the static pixel ratio in the video detection area corresponding to each frame of image based on the pixel area of ​​the first area in each frame of image and the pixel area of ​​the second area in each frame of image;

[0013] Calculate the dynamic pixel ratio in the video detection area corresponding to each frame of image based on the pixel points of each frame of image and the pixel area of ​​the second area in each frame of image;

[0014] The dynamic pixel ratio corresponding to each frame image and the static pixel ratio corresponding to each frame image are used as the plurality of second data.

[0015] Optionally, the calculating the dynamic pixel ratio in the video detection area corresponding to each frame of image based on the pixel points of each frame of image and the pixel area of ​​the second area in each frame of image includes:

[0016] Performing a differential operation on the pixel points of each frame of image and the pixel points corresponding to the positions in the previous frame of image to obtain the differential results corresponding to the pixel points in each frame of image;

[0017] Based on the differential result corresponding to the pixel points in each frame of the image and the binarization calculation threshold, the differential result corresponding to the pixel points in each frame of the image is converted into a binarization result, and the binarization result corresponding to the pixel points in each frame of the image is used to indicate whether the pixel points in each frame of the image are background pixel points or non-background pixel points;

[0018] The dynamic pixel ratio corresponding to each frame of image is calculated according to the number of non-background pixel points in the pixel points in each frame of image and the pixel area of ​​the second region in each frame of image.

[0019] Optionally, the pixel area of ​​the second region in each frame of image is determined by the following steps:

[0020] Creating an image matrix having the same size as the video data, and setting the pixels of the pixels in the image matrix to black;

[0021] Based on the target pixel point in the video detection area, modify the pixel point corresponding to the position of the target pixel point in the image matrix to white, wherein the target pixel point is a pixel point in the video detection area;

[0022] Traversing the pixel points of each frame of the image, based on the black pixel points in the image matrix, modifying the pixels of the pixel points corresponding to the positions of the black pixel points in each frame of the image to black, to obtain each new frame of the image, and performing grayscale processing on each new frame of the image to obtain each frame of the grayscale image;

[0023] Based on each frame of grayscale image, the number of non-black pixels in each frame of grayscale image is determined, and the number of non-black pixels in each frame of grayscale image is used to represent the pixel area of ​​the second region in each frame of image.

[0024] Optionally, the pixel area of ​​the first region in each frame of image is determined by the following steps:

[0025] By using a plurality of convolution kernels, the gradient of each pixel in each frame of the grayscale image in a plurality of directions is calculated, wherein the plurality of directions include a horizontal direction, a direction at 45° to the horizontal direction, a vertical direction, and a direction at 135° to the horizontal direction, and one of the plurality of convolution kernels corresponds to one of the plurality of directions;

[0026] Merging the gradients of each pixel in each frame of the grayscale image in the multiple directions to obtain the gradient amplitude of each pixel in each frame of the grayscale image;

[0027] For the i-th grayscale image frame in the video data, based on the gradient amplitude of the i-th grayscale image frame and a preset initial calculation threshold, determine the gradient calculation threshold corresponding to the i-th grayscale image frame, 3≤i≤the total number of frames of the video data, and the gradient calculation thresholds corresponding to the first grayscale image frame and the second grayscale image frame respectively are the initial calculation thresholds;

[0028] Based on the gradient calculation threshold corresponding to each frame of the grayscale image, binarize each frame of the grayscale image to obtain the first region;

[0029] The sum of the areas of the pixels in the first region in each frame of the image is calculated to obtain the pixel area of ​​the first region in each frame of the image.

[0030] Optionally, determining the gradient calculation threshold corresponding to the i-th grayscale image frame based on the gradient amplitude of the i-th grayscale image frame in the video data and a preset initial calculation threshold includes:

[0031] Based on the difference between the gradient entropy of the i-1th grayscale image frame and the gradient entropy of the i-2th grayscale image frame, adjusting the gradient calculation threshold corresponding to the i-1th grayscale image frame to obtain the gradient calculation threshold corresponding to the i-th grayscale image frame, and looping until the absolute value of the difference between the gradient calculation thresholds corresponding to the consecutive multiple grayscale images is less than or equal to the preset difference, then determining the gradient calculation thresholds corresponding to the grayscale images of the remaining frames of the video data after the consecutive multiple grayscale images based on the gradient calculation thresholds corresponding to the consecutive multiple grayscale images;

[0032] The gradient entropy of each frame of grayscale image is calculated based on the gradient amplitude of each frame of grayscale image and the gradient calculation threshold corresponding to each frame of grayscale image.

[0033] Optionally, the plurality of second data include a dynamic pixel ratio and a static pixel ratio within the video detection area;

[0034] The determining a second correlation coefficient between the plurality of second data based on the plurality of second data comprises:

[0035] Starting from the second frame image of the video data, collecting dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images respectively;

[0036] Calculating a first covariance between dynamic pixel ratios and static pixel ratios respectively corresponding to the multiple frames of images, a first standard deviation between dynamic pixel ratios respectively corresponding to the multiple frames of images, and a second standard deviation between static pixel ratios respectively corresponding to the multiple frames of images;

[0037] The second correlation coefficient is calculated based on the first covariance, the first standard deviation, and the second standard deviation.

[0038] Optionally, the plurality of first data include an area ratio of the radar detection target in a radar detection area, and an average speed of the radar detection target;

[0039] The determining a first correlation coefficient between the plurality of first data based on the plurality of first data comprises:

[0040] Starting from the second frame of the radar data, collecting area percentages and average speeds corresponding to multiple frames of radar data respectively;

[0041] Calculating a second covariance between the area proportions and the average speeds respectively corresponding to the multiple frames of radar data, a third standard deviation between the area proportions respectively corresponding to the multiple frames of radar data, and a fourth standard deviation between the average speeds respectively corresponding to the multiple frames of radar data;

[0042] The first correlation coefficient is calculated based on the second covariance, the third standard deviation, and the fourth standard deviation.

[0043] Optionally, the plurality of first data include an area ratio of the radar detection target in the radar detection area and an average speed of the radar detection target, and the plurality of second data include a dynamic pixel ratio and a static pixel ratio in the video detection area;

[0044] The integrating the first correlation coefficient and the second correlation coefficient to obtain the congestion coefficient of the lane includes:

[0045] Starting from the second frame of the radar data, collecting the area ratios and average speeds corresponding to the multiple frames of radar data; and starting from the second frame of the video data, collecting the dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images;

[0046] Calculating a first ratio between the plurality of first data based on an average value of area proportions respectively corresponding to the plurality of frames of radar data and an average value of average speeds respectively corresponding to the plurality of frames of radar data, and calculating a second ratio between the plurality of second data based on an average value of dynamic pixel ratios respectively corresponding to the plurality of frames of images and an average value of static pixel ratios respectively corresponding to the plurality of frames of images;

[0047] Calculate an intermediate coefficient based on the first correlation coefficient, the absolute value of the first correlation coefficient, the second correlation coefficient, the absolute value of the second correlation coefficient, the first ratio, and the second ratio;

[0048] The intermediate coefficient is converted through an activation function to obtain the congestion coefficient.

[0049] Optionally, the preset congestion threshold includes a first congestion threshold, a second congestion threshold, a third congestion threshold and a fourth congestion threshold, the first congestion threshold is greater than the second congestion threshold, the second congestion threshold is greater than the third congestion threshold, and the third congestion threshold is greater than the fourth congestion threshold;

[0050] The determining the congestion detection result of the lane based on the congestion coefficient and a preset congestion threshold comprises:

[0051] When the congestion coefficient is greater than or equal to the first congestion threshold, determining the congestion detection result as the lane being at a first congestion level;

[0052] When the congestion coefficient is greater than or equal to the second congestion threshold and less than the first congestion threshold, determining the congestion detection result as the lane being at a second congestion level;

[0053] When the congestion coefficient is greater than or equal to a third congestion threshold and less than the second congestion threshold, determining the congestion detection result as the lane being at a third congestion level;

[0054] When the congestion coefficient is greater than or equal to a fourth congestion threshold and less than the third congestion threshold, determining the congestion detection result as the lane being at a fourth congestion level;

[0055] The fourth congestion level is higher than the third congestion level, the third congestion level is higher than the second congestion level, and the second congestion level is higher than the first congestion level.

[0056] In a second aspect, an embodiment of the present application provides a lane congestion detection device, comprising:

[0057] a data analysis module, configured to analyze radar data of a radar detection target within a radar detection area to obtain a plurality of first data for representing a trajectory of the radar detection target, and to analyze video data of a video detection target within a video detection area to obtain a plurality of second data for representing the trajectory of the video detection target, wherein the radar detection target is a target within the radar detection area of ​​the lane, and the video detection target is a detection target within the video detection area of ​​the lane, and the radar detection area and the video detection area are unified;

[0058] a coefficient determination module, configured to determine a first correlation coefficient between the plurality of first data based on the plurality of first data, and to determine a second correlation coefficient between the plurality of second data based on the plurality of second data, wherein the first correlation coefficient is used to describe a lane congestion condition within the radar detection area, and the second correlation coefficient is used to describe a lane congestion condition within the video detection area;

[0059] A coefficient integration module, used for integrating the first correlation coefficient and the second correlation coefficient to obtain a congestion coefficient of the lane;

[0060] The congestion detection module is used to determine the congestion detection result of the lane based on the congestion coefficient and a preset congestion threshold.

[0061] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor, a memory, and a system bus;

[0062] The processor and the memory are connected via the system bus;

[0063] The memory is used to store a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes any implementation step of the above lane congestion detection method.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes any implementation step of the above-mentioned lane congestion detection method.

[0065] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0066] In an embodiment of the present application, first, the radar data of the radar detection target in the radar detection area can be analyzed to obtain a plurality of first data for representing the trajectory of the radar detection target. In addition, the video data of the video detection target in the video detection area can be analyzed to obtain a plurality of second data for representing the trajectory of the video detection target. Among them, the radar detection target is the target in the radar detection area of ​​the lane, the video detection target is the detection target in the video detection area of ​​the lane, and the radar detection area and the video detection area are unified. Then, the first correlation coefficient between the plurality of first data can be determined based on the plurality of first data, and the second correlation coefficient between the plurality of second data can be determined based on the plurality of second data. Since the first correlation coefficient is used to describe the lane congestion in the radar detection area, and the second correlation coefficient is used to describe the lane congestion in the video detection area, the congestion coefficient of the lane can be obtained by integrating the first correlation coefficient and the second correlation coefficient. Then, the congestion detection result of the lane can be determined based on the congestion coefficient and the pre-set congestion threshold.

[0067] It can be seen that by analyzing and processing the radar data and the video data, the first correlation coefficient describing the lane congestion in the radar detection area and the second correlation coefficient describing the lane congestion in the video detection area can be obtained respectively. Then, by integrating the first correlation coefficient and the second correlation coefficient, the congestion coefficient of the lane can be obtained by combining the two means of radar detection and video detection. In this way, the complementarity between the two means of radar detection and video detection can make the above congestion coefficient more accurately represent the congestion degree of the lane and minimize the impact of environmental factors. Subsequently, based on the congestion coefficient and the corresponding congestion threshold, the congestion detection result of the lane can be accurately obtained, thereby improving the accuracy of lane congestion detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A flowchart of a lane congestion detection method provided in an embodiment of the present application;

[0069] Figure 2 A flowchart of a process for obtaining second data provided in an embodiment of the present application;

[0070] Figure 3 A schematic diagram of the structure of a lane congestion detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] As mentioned above, in practical applications, taking traffic detection scenarios as an example, taking the vehicle in the lane as the target, tracking the target through radar detection equipment or visual tracking algorithms can determine whether the lane is currently congested or other traffic situations. However, due to the influence of surrounding environmental factors, especially in complex environmental conditions such as bad weather, dense buildings or electromagnetic interference, the detection effect of the radar detection equipment installed on the lane on the vehicle, that is, the target tracking effect, will be greatly reduced. When using the visual tracking algorithm, the video data obtained by the traffic camera installed on the lane may also be affected by factors such as bad weather or changes in lighting, resulting in the subsequent further data analysis through the visual tracking algorithm to achieve less than ideal results. Therefore, in traffic detection scenarios, existing target tracking methods often cannot accurately detect the congestion status of the lane.

[0072] Based on this, in order to solve the above problems, the embodiment of the present application provides a lane congestion detection method, which may include: first, the radar data of the radar detection target in the radar detection area can be analyzed to obtain multiple first data for representing the trajectory of the radar detection target. In addition, the video data of the video detection target in the video detection area can be analyzed to obtain multiple second data for representing the trajectory of the video detection target. Among them, the radar detection target is the target in the radar detection area of ​​the lane, the video detection target is the detection target in the video detection area of ​​the lane, and the radar detection area and the video detection area are unified. Then, the first correlation coefficient between the multiple first data can be determined based on the multiple first data, and the second correlation coefficient between the multiple second data can be determined based on the multiple second data. Since the first correlation coefficient is used to describe the lane congestion in the radar detection area, and the second correlation coefficient is used to describe the lane congestion in the video detection area, the congestion coefficient of the lane can be obtained by integrating the first correlation coefficient and the second correlation coefficient. Then, the congestion detection result of the lane can be determined based on the congestion coefficient and the pre-set congestion threshold.

[0073] In this way, by analyzing and processing the radar data and the video data, the first correlation coefficient describing the lane congestion in the radar detection area and the second correlation coefficient describing the lane congestion in the video detection area can be obtained respectively. Then, by integrating the first correlation coefficient and the second correlation coefficient, the congestion coefficient of the lane can be obtained by combining the two means of radar detection and video detection. In this way, the complementarity between the two means of radar detection and video detection can make the above congestion coefficient more accurately represent the congestion degree of the lane and minimize the impact of environmental factors. Subsequently, based on the congestion coefficient and the corresponding congestion threshold, the congestion detection result of the lane can be accurately obtained, thereby improving the accuracy of lane congestion detection.

[0074] It should be noted that the embodiment of the present application may not limit the execution subject of the lane congestion detection method. For example, the lane congestion detection method of the embodiment of the present application may be applied to data processing devices such as terminal devices or servers. Among them, the terminal device may be an electronic device such as a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server may be an independent server, a cluster server, or a cloud server.

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] Figure 1 A flowchart of a lane-based congestion detection method provided in an embodiment of the present application. Figure 1 As shown, the lane congestion detection method provided in the embodiment of the present application may include the following steps S101 to S104.

[0077] S101: Analyze radar data of radar detection targets in a radar detection area to obtain a plurality of first data for representing the trajectory of the radar detection targets, and analyze video data of video detection targets in a video detection area to obtain a plurality of second data for representing the trajectory of the video detection targets.

[0078] The radar detection target is a target within the radar detection area of ​​the lane, and the radar data can be obtained by the radar detection device. In the embodiment of the present application, the analysis process of the radar data is not limited, and any existing or future algorithm that analyzes radar data to achieve target detection can be used for implementation. Through the above algorithm that can achieve target detection, the trajectory of the radar detection target in each frame within the radar detection area can be obtained.

[0079] Based on this, the above-mentioned multiple first data may include the area ratio of the radar detection target in the radar detection area ratio_radar and the average speed v_mean of the radar detection target. In the radar data, for the scene where all targets are completely moving and there are no stationary targets, the area ratio of the radar detection target in the radar detection area ratio_radar should be close to the average speed v_mean of the radar detection target, and the two are correlated in data distribution. Therefore, when ratio_radar and v_mean are positively correlated, it is equivalent to that the spatial occupancy rate of the radar detection area becomes higher and the vehicle speed does not decrease; when ratio_radar and v_mean are negatively correlated, it is equivalent to that the spatial occupancy rate of the radar detection area becomes higher and the vehicle speed decreases. Therefore, the subsequent correlation between the two, that is, the first correlation coefficient, can describe the lane congestion in the radar detection area.

[0080] Specifically, for the acquisition method of ratio_radar, first, based on the trajectory of the radar detection target in each frame in the radar detection area, the number of radar detection targets in the radar detection area and the length and width of each radar detection target can be calculated to calculate the area of ​​each radar detection target. Then, the quotient of the total area of ​​all radar detection targets and the area of ​​the radar detection area can be calculated to obtain the area ratio of the radar detection target in the radar detection area ratio_radar.

[0081] As for the method of obtaining v_mean, it can be obtained by calculating the average value of the speeds of all radar detection targets in the radar detection area.

[0082] In addition, in order to facilitate the subsequent data processing process, the area ratio of the radar detection target in the radar detection area ratio_radar and the average speed v_mean of the radar detection target can be normalized and converted to [0,1].

[0083] Further, the video detection target is a detection target within the video detection area of ​​the lane, and the video data can be obtained by the video detection device. The above-mentioned multiple second data may include a dynamic pixel ratio ratio_dynamic and a static pixel ratio ratio_static within the video detection area. For the acquisition method of the above-mentioned multiple second data, please refer to the introduction made below, which will not be expanded here.

[0084] In the video data, for scenes where all targets are fully moving and there are no stationary targets, the moving pixel ratio ratio_dynamic should be close to the static pixel ratio ratio_static, and the two are correlated in data distribution. Therefore, when ratio_dynamic and ratio_static are positively correlated, it is equivalent to the spatial occupancy of the video detection area becoming higher and the vehicle speed not decreasing; when ratio_dynamic and ratio_static are negatively correlated, it is equivalent to the spatial occupancy of the video detection area becoming higher and the vehicle speed decreasing. Therefore, the subsequent correlation between the two, that is, the second correlation coefficient, can describe the lane congestion in the video detection area.

[0085] In addition, it should be noted that the radar detection area and the video detection area are unified. The unification of the two detection areas helps to accurately perform the subsequent data processing process and obtain accurate lane congestion detection results.

[0086] Based on this, before executing step S101, the coordinate system of the radar detection device and the coordinate system of the video detection device can be calibrated to obtain the calibration results between the two coordinate systems to achieve mutual mapping between the two coordinate systems, that is, to map the two coordinate systems to one coordinate system. Then, multiple vertices can be selected from the original video data, and the area enclosed by the multiple vertices can be used as the video detection area. Then, based on the calibration results between the two coordinate systems, the pixel coordinates of the multiple vertices selected above can be converted to the radar coordinate system, and the radar detection area can be generated based on the corresponding vertices in the radar coordinate system. In this way, a radar detection area that is unified with the video detection area can be generated in the radar coordinate system.

[0087] S102: Determine a first correlation coefficient between the plurality of first data based on the plurality of first data, and determine a second correlation coefficient between the plurality of second data based on the plurality of second data.

[0088] The first correlation coefficient is used to describe the lane congestion in the radar detection area, and the second correlation coefficient is used to describe the lane congestion in the video detection area.

[0089] As mentioned above, the plurality of first data includes ratio_radar, an area ratio of the radar detected target in the radar detection area, and v_mean, an average speed of the radar detected target. Based on this, the determination process of the first correlation coefficient may include the following steps 11 to 13.

[0090] Step 11: Starting from the second frame of radar data, collect the area percentage and average speed corresponding to multiple frames of radar data.

[0091] Since it is difficult to calculate the average speed corresponding to the first frame of radar data, a set of data corresponding to multiple frames of radar data can be collected starting from the second frame of radar data. Each set of data includes the area ratio ratio_radar and the average speed v_mean corresponding to each frame.

[0092] Accordingly, if the acquisition requires T seconds of data, N frames can be collected according to the frame rate of the radar data to obtain the corresponding N sets of data sequences. Among them, N frames of data can be obtained by multiplying the frame rate (Frames Per Second, FPS) of the radar data by T.

[0093] In addition, during the data collection process, the data can be collected in the form of a sliding window, that is, when the collected data reaches N groups, after the data collected in the latest frame is added to the data sequence, the first group of data in the data sequence is deleted, thereby maintaining a continuous iterative analysis of the correlation between multiple first data in the time dimension.

[0094] Step 12: Calculate the second covariance between the area proportions and the average speeds corresponding to the multiple frames of radar data, the third standard deviation between the area proportions corresponding to the multiple frames of radar data, and the fourth standard deviation between the average speeds corresponding to the multiple frames of radar data.

[0095] In addition, as mentioned above, the area ratio of the radar detection target in the radar detection area ratio_radar and the average speed v_mean of the radar detection target can be normalized and converted to [0,1]. Since ratio_radar and v_mean are always in [0,1], the natural defect of covariance being sensitive to the numerical range will not affect subsequent calculations. Therefore, the correlation between ratio_radar and v_mean can be calculated in combination with covariance and standard deviation, thereby improving the accuracy of the first correlation coefficient.

[0096] Accordingly, the second covariance between the area ratios and the average speeds corresponding to multiple frames of radar data can be expressed as Cov(ratio_radar,v_mean), the third standard deviation between the area ratios corresponding to multiple frames of radar data can be expressed as std_radar, and the fourth standard deviation between the average speeds corresponding to multiple frames of radar data can be expressed as std_v.

[0097] It should be noted that the calculation formula for the second covariance can be implemented by using any existing or future covariance calculation method, which will not be described in detail here. Similarly, the calculation process of the third standard deviation and the fourth standard deviation can be implemented by using any existing or future standard deviation calculation method, which will not be described in detail here.

[0098] Step 13: Calculate a first correlation coefficient based on the second covariance, the third standard deviation, and the fourth standard deviation.

[0099] For ease of understanding, an exemplary explanation is given below in conjunction with formula (1).

[0100]

[0101] Among them, ρ1 is the first correlation coefficient, Cov(ratio_radar,v_mean) is the second covariance, std_radar is the third standard deviation, and std_v is the fourth standard deviation.

[0102] Furthermore, as mentioned above, the plurality of second data includes a dynamic pixel ratio ratio_dynamic and a static pixel ratio ratio_static in the video detection area. Based on this, the process of determining the second correlation coefficient may include the following steps 14 to 16.

[0103] Step 14: Starting from the second frame image of the video data, the dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images are collected.

[0104] Since it is difficult to calculate the dynamic pixel ratio corresponding to the first frame of video data, it is possible to collect a set of data corresponding to multiple frames starting from the second frame of video data, each set of data including the dynamic pixel ratio ratio_dynamic and the static pixel ratio ratio_static corresponding to each frame.

[0105] Accordingly, if the acquisition requires T seconds of data, N frames can be collected according to the frame rate of the video data to obtain the corresponding N sets of data sequences. Among them, N frames of data can be obtained by multiplying T by the frame rate (Frames Per Second, FPS) of the radar data.

[0106] In addition, during the data collection process, a sliding window form can be used for collection, that is, when the collected data reaches N groups, after the latest frame of collected data is added to the data sequence, the first group of data in the data sequence is deleted, thereby maintaining continuous iterative analysis of the correlation between multiple second data in the time dimension.

[0107] Step 15: Calculate the first covariance between the dynamic pixel ratios and the static pixel ratios corresponding to the multiple frames of images, the first standard deviation between the dynamic pixel ratios corresponding to the multiple frames of images, and the second standard deviation between the static pixel ratios corresponding to the multiple frames of images.

[0108] In addition, since the dynamic pixel ratio ratio_dynamic and the static pixel ratio ratio_static are always in [0,1], the natural defect of the covariance being sensitive to the numerical range will not affect the subsequent calculations. Therefore, the correlation between ratio_dynamic and ratio_static can be calculated in combination with the covariance and the standard deviation, thereby improving the accuracy of the second correlation coefficient.

[0109] Accordingly, the first covariance between the dynamic pixel ratios and static pixel ratios corresponding to multiple frame images can be expressed as Cov(ratio_static, ratio_dynamic), the first standard deviation between the dynamic pixel ratios corresponding to multiple frame images can be expressed as std_dynamic, and the second standard deviation between the static pixel ratios corresponding to multiple frame images can be expressed as std_static.

[0110] It should be noted that the calculation formula for the first covariance can be implemented by using any existing or future covariance calculation method, which will not be described in detail here. Similarly, the calculation process of the first standard deviation and the second standard deviation can be implemented by using any existing or future standard deviation calculation method, which will not be described in detail here.

[0111] Step 16: Calculate a second correlation coefficient based on the first covariance, the first standard deviation, and the second standard deviation.

[0112] For ease of understanding, an exemplary explanation is given below in conjunction with formula (2).

[0113]

[0114] Among them, ρ2 is the second correlation coefficient, Cov(ratio_static, ratio_dynamic) is the first covariance, std_static is the second standard deviation, and std_dynamic is the third standard deviation.

[0115] S103: Integrate the first correlation coefficient and the second correlation coefficient to obtain a lane congestion coefficient.

[0116] In the embodiment of the present application, since the first correlation coefficient is based on data obtained by radar detection means, and the second correlation coefficient is based on data obtained by video detection means, integrating the first correlation coefficient and the second correlation coefficient obtained above helps to more accurately determine the congestion degree of the lane. For ease of understanding, the following is an exemplary description of the process of determining the congestion coefficient of the lane, that is, step S103, in combination with a possible implementation method.

[0117] As a possible implementation, step S103 may include the following steps 17 to 10.

[0118] Step 17: Starting from the second frame of the radar data, collect the area ratios and average speeds corresponding to the multiple frames of radar data; and, starting from the second frame of the video data, collect the dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images.

[0119] Here, the process of collecting relevant data is similar to the above steps 1 and 4, and reference may be made to the relevant description in the above embodiments, which will not be repeated here.

[0120] Step 18: Based on the average value of the area ratios corresponding to the multiple frames of radar data and the average value of the average speeds corresponding to the multiple frames of radar data, calculate the first ratio between the multiple first data, and based on the average value of the dynamic pixel ratios corresponding to the multiple frames of images and the average value of the static pixel ratios corresponding to the multiple frames of images, calculate the second ratio between the multiple second data.

[0121] The first ratio can be expressed as ratio_1, which is calculated by dividing the average value of ratio_radar corresponding to multiple frames of radar data by the average value of v_mean corresponding to multiple frames of radar data. The second ratio can be expressed as ratio_2, which is calculated by dividing the average value of dynamic pixel ratios ratio_dynamic corresponding to the multiple frames of images by the average value of static pixel ratios ratio_static corresponding to the multiple frames of images.

[0122] Step 19: Calculate an intermediate coefficient based on the first correlation coefficient, the absolute value of the first correlation coefficient, the second correlation coefficient, the absolute value of the second correlation coefficient, the first ratio, and the second ratio.

[0123] For ease of understanding, an exemplary explanation is given below in conjunction with formula (3).

[0124]

[0125] Among them, Ret is the intermediate coefficient, ratio_1 is the first ratio, ratio_2 is the second ratio, abs() represents the absolute value function, ρ1 is the first correlation coefficient, and ρ2 is the second correlation coefficient.

[0126] Step 10: Convert the intermediate coefficients through the activation function to obtain the congestion coefficient.

[0127] For ease of understanding, an exemplary explanation is given below in conjunction with formula (4).

[0128]

[0129] Among them, R is the congestion coefficient and Ret is the intermediate coefficient.

[0130] When the first correlation coefficient or the second correlation coefficient is negative, the intermediate coefficient Ret will be negative accordingly. Therefore, by converting the intermediate coefficient with the help of the activation function, the final congestion coefficient can be within [0,1], and the congestion coefficient is consistent with the data trend of the first correlation coefficient and the second correlation coefficient, that is, when the first correlation coefficient or the second correlation coefficient indicates that the corresponding data are negatively correlated, the smaller the first correlation coefficient or the second correlation coefficient is, the smaller the congestion coefficient is; when the first correlation coefficient or the second correlation coefficient indicates that the corresponding data are positively correlated, the larger the first correlation coefficient or the second correlation coefficient is, the larger the congestion coefficient is. In this way, the congestion degree of the lane can be accurately indicated with the help of the congestion coefficient.

[0131] S104: Determine a lane congestion detection result based on the congestion coefficient and a preset congestion threshold.

[0132] In the embodiment of the present application, the pre-set congestion threshold may include a first congestion threshold, a second congestion threshold, a third congestion threshold and a fourth congestion threshold, wherein the first congestion threshold is greater than the second congestion threshold, the second congestion threshold is greater than the third congestion threshold, and the third congestion threshold is greater than the fourth congestion threshold.

[0133] Based on this, the implementation process of determining the congestion detection result of the lane, that is, step S104, may include: when the congestion coefficient is greater than or equal to the first congestion threshold, determining the congestion detection result as the lane being in the first congestion level; when the congestion coefficient is greater than or equal to the second congestion threshold and less than the first congestion threshold, determining the congestion detection result as the lane being in the second congestion level; when the congestion coefficient is greater than or equal to the third congestion threshold and less than the second congestion threshold, determining the congestion detection result as the lane being in the third congestion level; when the congestion coefficient is greater than or equal to the fourth congestion threshold and less than the third congestion threshold, determining the congestion detection result as the lane being in the fourth congestion level. The fourth congestion level is higher than the third congestion level, the third congestion level is higher than the second congestion level, and the second congestion level is higher than the first congestion level.

[0134] For ease of understanding, the above congestion threshold and the corresponding congestion detection result are described below in conjunction with Table 1.

[0135] Table 1

[0136] Congestion Threshold Congestion level R≥0.6061 (first congestion threshold) Smooth (first congestion level) 0.4418 (second congestion threshold) ≤ R < 0.6061 Mild congestion (second level of congestion) 0.2694 (third congestion threshold) ≤ R < 0.4418 Moderate congestion (third level of congestion) 0 (the fourth congestion threshold) ≤ R < 0.2694 Severe congestion (the fourth level of congestion)

[0137] As shown in Table 1, the first congestion threshold can be set to 0.6061, the second congestion threshold can be set to 0.4418, the third congestion threshold can be set to 0.2694, and the fourth congestion threshold can be set to 0. Accordingly, the first congestion level can indicate that the lane is unblocked, the second congestion level can indicate that the lane is slightly congested, the third congestion level can indicate that the lane is moderately congested, and the fourth congestion level can indicate that the lane is moderately congested.

[0138] Based on the relevant contents of the above steps S101 to S104, it can be known that in an embodiment of the present application, by analyzing and processing the radar data and the video data, a first correlation coefficient describing the lane congestion situation in the radar detection area and a second correlation coefficient describing the lane congestion situation in the video detection area can be obtained respectively. Then, by integrating the first correlation coefficient and the second correlation coefficient, the congestion coefficient of the lane can be obtained by combining the two means of radar detection and video detection. In this way, the complementarity between the two means of radar detection and video detection can make the above congestion coefficient more accurately represent the congestion degree of the lane, minimize the impact of environmental factors, and subsequently, based on the congestion coefficient and the corresponding congestion threshold, the congestion detection result of the lane can be accurately obtained, thereby improving the accuracy of lane congestion detection.

[0139] Further, Figure 2 A flowchart of a second data acquisition process provided in an embodiment of the present application. Figure 2 As shown, the process of obtaining the second data provided in the embodiment of the present application may include the following steps S201 to S204.

[0140] S201: Determine the pixel area of ​​a first region in each frame image of video data and the pixel area of ​​a second region in each frame image.

[0141] The first area is the area enclosed by the edge of the video detection target. The second area is the region of interest in the video detection area, which is also the focus area of ​​image analysis in the video detection area. For ease of understanding, the following is an exemplary description of the determination method of the pixel area Ss of the first area and the pixel area Sc of the second area.

[0142] The pixel area Sc of the second region in each frame of image can be determined by the following steps 21 to 24:

[0143] Step 21: Create an image matrix with the same size as the video data, and set the pixels in the image matrix to black.

[0144] In practical applications, the pixels of all pixels of the image matrix are set to [0,0,0] to set the pixels of the pixels to black.

[0145] Step 22: Based on the target pixel in the video detection area, the pixels of the pixels corresponding to the target pixel position in the image matrix are modified to white.

[0146] The target pixel is a pixel in the video detection area. As mentioned above, multiple vertices can be selected from the original video data, and the area enclosed by the multiple vertices is used as the video detection area. Based on this, the target pixel is also a pixel in the area enclosed by the multiple vertices.

[0147] Accordingly, in a specific implementation, first, the pixel corresponding to the target pixel position can be determined in the image matrix according to the position of the target pixel. Based on step 21, it can be known that the pixels of these determined pixels have been set to black, and accordingly, the pixels of these determined pixels can be set to [255, 255, 255] to modify the pixels of these pixels to white.

[0148] Thus, in the image matrix, the pixels corresponding to the pixel positions within the video detection area are all white, so the area where the white pixels are located can be represented as the region of interest. Correspondingly, in the image matrix, the pixels corresponding to the pixel positions outside the video detection area are all black, so the area where the black pixels are located can be represented as the region of no interest.

[0149] Step 23: Traverse the pixel points of each frame of the image, based on the black pixel points in the image matrix, modify the pixels of the pixel points corresponding to the black pixel points in each frame of the image to black, obtain each new image frame, and perform grayscale processing on each new image frame to obtain each grayscale image frame.

[0150] Specifically, in the process of traversing the pixels of each frame of the image, first, the pixels of each pixel in each frame of the image can be compared with the pixels of the corresponding pixels in the image matrix. If the pixels of the pixels in the image matrix are black, the pixels of the pixels corresponding to the black pixel positions in each frame of the image can also be modified to black, that is, if the pixels of the pixels in the image matrix are [0,0,0], the pixels of the pixels corresponding to the black pixel positions in each frame of the image can also be modified to [0,0,0]. In this way, a new image can be obtained for each frame.

[0151] Then, each new image frame can be gray-scale processed to obtain each gray-scale image frame. In addition, in order to improve the image processing effect and obtain an accurate second region, in the embodiment of the present application, each gray-scale image frame can be further subjected to histogram equalization processing to balance the effect of illumination on the imaging of each gray-scale image frame, and also to balance the effect of illumination on the foreground and background segmentation during subsequent edge detection.

[0152] As mentioned above, the area where the black pixels in the image matrix are located represents the non-interest area, and the area where the white pixels are located represents the interest area. Therefore, by traversing the pixels of each frame of the image, based on the black pixels in the image matrix, the pixels corresponding to the black pixel positions in each frame of the image are modified to black, and the interest area and the non-interest area can be distinguished in each frame of the image. In each frame of the image, the interest area is the area where the non-black pixels are located, and the non-interest area is the area where the black pixels are located.

[0153] Step 24: Based on each frame of grayscale image, determine the number of non-black pixels in each frame of grayscale image.

[0154] Here, the number of non-black pixels in each frame of the grayscale image is used to represent the pixel area Sc of the second region in each frame of the image.

[0155] As mentioned above, in each frame of the image, the region of interest is the region where non-black pixels are located, and the region of non-interest is the region where black pixels are located. Therefore, in each frame of the grayscale image, by counting the number of pixels that are not black, that is, counting the number of pixels with non-zero pixel values, the pixel area Sc of the second region can be expressed.

[0156] In this way, through steps 21 to 24, the second area can be extracted from each frame of the image, and the influence of the non-interested area on the image processing can be removed, which will help to obtain accurate second data later, and then combine the first data and the second data to more accurately detect lane congestion, thereby improving the accuracy of lane congestion detection.

[0157] Furthermore, for the pixel area Ss of the first region in each frame image, in the embodiment of the present application, the first region can be extracted by an edge detection algorithm, and then the pixel area Ss of the first region can be further determined. Based on this, for the pixel area Ss of the first region in each frame image, it can be determined by the following steps 25 to 29:

[0158] Step 25: Using multiple convolution kernels, calculate the gradient of each pixel in each frame of the grayscale image in multiple directions.

[0159] In an embodiment of the present application, an edge detection algorithm can be implemented based on a Sobel operator. Specifically, an edge detection algorithm based on a Sobel operator can use multiple convolution kernels to calculate gradients in multiple directions. Among them, multiple convolution kernels correspond to multiple directions one by one, that is, one convolution kernel corresponds to one direction. Accordingly, the multiple directions include a horizontal direction, a direction at 45° to the horizontal direction, a vertical direction, and a direction at 135° to the horizontal direction.

[0160] Based on this, the convolution kernel corresponding to the horizontal direction can be expressed as Gx:

[0161]

[0162] Correspondingly, taking a pixel point (x, y) in a certain frame of grayscale image as an example, the grayscale value of the pixel point (x, y) is P(x, y), and the horizontal gradient M1 of the pixel point can be calculated through the convolution kernel Gx. It can be specifically shown in the following formula (5):

[0163] M1=P(x+1,y)-P(x,y)(5)

[0164] The convolution kernel corresponding to the direction of 45° to the horizontal direction can be expressed as Gx:

[0165]

[0166] Correspondingly, still taking a pixel point (x, y) in a certain frame of grayscale image as an example, the grayscale value of the pixel point (x, y) is P(x, y), then the gradient M2 of the pixel point in the direction 45° to the horizontal direction can be calculated through the convolution kernel Gx. It can be specifically shown in the following formula (6):

[0167] M2=P(x+1,y+1)-P(x,y)(6)

[0168] The convolution kernel corresponding to the vertical direction can be expressed as Gy:

[0169]

[0170] Correspondingly, still taking a pixel point (x, y) in a certain frame of grayscale image as an example, the grayscale value of the pixel point (x, y) is P(x, y), and the gradient M3 of the pixel point in the vertical direction can be calculated through the convolution kernel Gy. It can be specifically shown in the following formula (7):

[0171] M3=P(x,y+1)-P(x,y)(7)

[0172] The convolution kernel corresponding to the direction of 135° to the horizontal direction can be expressed as G2:

[0173]

[0174] Correspondingly, still taking a pixel point (x, y) in a certain frame of grayscale image as an example, the grayscale value of the pixel point (x, y) is P(x, y), then the gradient M4 of the pixel point in the direction 135° to the horizontal direction can be calculated through the convolution kernel G2. It can be specifically shown in the following formula (8):

[0175] M4=P(x-1,y+1)-P(x,y) (8)

[0176] In this way, based on the above four directions, four convolution kernels are used for convolution calculation, which can not only improve the accuracy of edge detection, but also will not generate a large computing performance burden.

[0177] Step 26: Merge the gradients of each pixel in each frame of the grayscale image in multiple directions to obtain the gradient amplitude of each pixel in each frame of the grayscale image.

[0178] In the embodiment of the present application, the process of merging the gradients of each pixel in each frame of grayscale image in the above multiple directions can be implemented by the following formula (9):

[0179]

[0180] Among them, for a pixel point in a frame of grayscale image, G is the gradient amplitude of the pixel point, M1 is the gradient of the pixel point in the horizontal direction, M2 is the gradient of the pixel point in the direction of 45° to the horizontal direction, M3 is the gradient of the pixel point in the vertical direction, and M4 is the gradient of the pixel point in the direction of 135° to the horizontal direction.

[0181] Furthermore, the gradient amplitude of each pixel in each frame of the grayscale image can be mapped to an integer in [0,255] to standardize the gradient amplitude, so as to accurately distinguish edge points and non-edge points and realize edge detection.

[0182] Step 27: for the i-th grayscale image frame in the video data, based on the gradient amplitude of the i-th grayscale image frame and a preset initial calculation threshold, determine the gradient calculation threshold corresponding to the i-th grayscale image frame.

[0183] Wherein, 3≤i≤total number of frames of video data. The gradient calculation thresholds corresponding to the first frame grayscale image and the second frame grayscale image are respectively the initial calculation thresholds T.

[0184] Here, considering that due to factors such as lighting or imaging methods, the grayscale distribution of the foreground and background in the grayscale image sometimes overlaps to a certain extent, making it difficult to distinguish the foreground and background. Therefore, the effect of foreground and background segmentation based on the same gradient calculation threshold is often not good enough and cannot adapt to subtle changes in the background.

[0185] Based on this, in an embodiment of the present application, the gradient amplitude of each frame of grayscale image can be determined to determine the gradient calculation threshold corresponding to each frame of grayscale image, so that each frame of grayscale image can be segmented into foreground and background according to the actual pixel distribution law.

[0186] In this way, in a specific implementation, the above step 27 can be reflected as follows: based on the difference between the gradient entropy of the i-1th frame grayscale image and the gradient entropy of the i-2th frame grayscale image, the gradient calculation threshold corresponding to the i-1th frame grayscale image is adjusted to obtain the gradient calculation threshold corresponding to the i-th frame grayscale image, and the cycle is executed until the absolute value of the difference between the gradient calculation thresholds corresponding to the consecutive multiple frames of grayscale images is less than or equal to the preset difference, and based on the gradient calculation thresholds corresponding to the consecutive multiple frames of grayscale images, the gradient calculation thresholds corresponding to the grayscale images of the remaining frames of the video data after the consecutive multiple frames of grayscale images are determined.

[0187] The gradient entropy of each frame of grayscale image is calculated based on the gradient amplitude of each frame of grayscale image and the gradient calculation threshold corresponding to each frame of grayscale image.

[0188] Specifically, in the embodiment of the present application, when the gradient entropy of a certain frame of grayscale image is the largest, the gradient calculation threshold corresponding to the frame of grayscale image can achieve the best foreground and background segmentation effect of the frame of grayscale image. As a result, the gradient calculation threshold used for the next frame calculation can be gradually selected in an iterative manner by changing the gradient entropy corresponding to the continuous frame of grayscale image within [0,255].

[0189] Based on this, the process of obtaining the gradient calculation threshold corresponding to the i-th frame grayscale image can be reflected as follows:

[0190] When the difference between the gradient entropy of the i-1th grayscale image frame and the gradient entropy of the i-2th grayscale image frame is greater than 0, the gradient calculation threshold corresponding to the i-1th grayscale image frame is increased, and the increased gradient calculation threshold is used as the gradient calculation threshold corresponding to the i-th grayscale image frame.

[0191] When the difference between the gradient entropy of the i-1th grayscale image frame and the gradient entropy of the i-2th grayscale image frame is less than 0, the gradient calculation threshold corresponding to the i-1th grayscale image frame is reduced, and the reduced grayscale calculation threshold is used as the gradient calculation threshold corresponding to the i-th grayscale image frame.

[0192] Taking the determination process of the gradient calculation thresholds corresponding to the third and fourth frames of grayscale images as an example, when the difference between the gradient entropy of the second frame of grayscale image and the gradient entropy of the first frame of grayscale image is greater than 0, the gradient calculation threshold corresponding to the second frame of grayscale image can be increased, that is, the initial calculation threshold T is increased to obtain the gradient calculation threshold corresponding to the third frame of grayscale image. When the difference is less than 0, the initial calculation threshold T can be reduced to obtain the gradient calculation threshold corresponding to the third frame of grayscale image.

[0193] Then, the gradient entropy of the third grayscale image can be calculated based on the gradient amplitude of the third grayscale image and the gradient calculation threshold corresponding to the third grayscale image. Then, the gradient calculation threshold corresponding to the third grayscale image can be adjusted based on the difference between the gradient entropy of the third grayscale image and the gradient entropy of the second grayscale image, to obtain the gradient calculation threshold corresponding to the fourth grayscale image.

[0194] In this way, the gradient calculation threshold corresponding to each frame of grayscale image can be gradually calculated in a loop iteration manner, so that each frame of grayscale image can be segmented into foreground and background according to the actual pixel distribution law.

[0195] Finally, in order to simplify the calculation process and avoid performance loss, in the actual iteration process, when the preset condition is reached, that is, the absolute value of the difference between the gradient calculation thresholds corresponding to the continuous multiple frames of grayscale images mentioned above is less than or equal to the preset difference, the adjustment of the gradient calculation threshold corresponding to the grayscale image of the subsequent frame can be stopped. Accordingly, based on the gradient calculation thresholds corresponding to the continuous multiple frames of grayscale images, the gradient calculation threshold corresponding to the grayscale images of the remaining frames of the video data after the continuous multiple frames of grayscale images can be determined.

[0196] For example, in the preset condition, the continuous multiple frames are set to 2 consecutive frames, and the preset difference is set to 2. Therefore, when the absolute value of the difference between the gradient calculation thresholds corresponding to the two consecutive grayscale image frames is less than or equal to 2, the calculation of the gradient calculation thresholds corresponding to the grayscale images of the remaining frames after the two grayscale images can be stopped.

[0197] Furthermore, in the process of determining the gradient calculation threshold corresponding to the grayscale image of the remaining frames, if the above absolute value is 0, that is, the gradient calculation thresholds corresponding to the continuous grayscale images are the same, then the same gradient calculation threshold can be used as the gradient calculation threshold corresponding to the grayscale images of the remaining frames. If the above absolute value is not 0, that is, the gradient calculation thresholds corresponding to the continuous grayscale images are different, then the average value of the gradient calculation thresholds corresponding to the continuous grayscale images can be calculated and used as the gradient calculation threshold corresponding to the grayscale images of the remaining frames.

[0198] In addition, as mentioned above, the gradient entropy of each frame of grayscale image is calculated based on the gradient amplitude of each frame of grayscale image and the gradient calculation threshold corresponding to each frame of grayscale image. Based on this, the calculation process of the gradient entropy of each frame of grayscale image is exemplified below by taking one frame as an example.

[0199] First, in a frame of grayscale image, pixel points whose gradient amplitude is less than the gradient calculation threshold corresponding to the frame of grayscale image can be determined as non-edge points, and pixel points whose gradient amplitude is greater than the gradient calculation threshold corresponding to the frame of grayscale image can be determined as non-edge points.

[0200] Next, the number S of non-edge points in the grayscale image frame and the probability P of occurrence of the gradient amplitude of each non-edge point can be obtained. i , and the probability of occurrence of the gradient amplitude of each non-edge point P i Sum and get the probability and Z of the gradient assignment of each non-edge point. i The value can be obtained by calculating the number of times the gradient amplitude of each non-edge point appears in the gradient amplitudes corresponding to all the pixels of the grayscale image of the frame and dividing it by the number N of all the pixels.

[0201] In addition, the number of edge points in the grayscale image frame and the probability of occurrence of the gradient amplitude of each edge point P can be obtained. j Among them, P j The value can be obtained by calculating the number of times the gradient amplitude of each edge point appears in the gradient amplitudes corresponding to all pixels of the grayscale image of the frame and dividing it by the number N of all pixels.

[0202] Based on this, we can further calculate the probability P of the gradient amplitude of each non-edge point based on the number S of non-edge points. i And the probability and Z of the gradient assignment of each non-edge point are used to calculate the gradient amplitude entropy of the non-edge point. And based on the number of non-edge points S and the number of all pixel points N, the probability of occurrence of the gradient amplitude of each edge point P j As well as the probability and Z of the gradient assignment of each non-edge point, the gradient amplitude entropy of the edge point is calculated. For ease of understanding, the following formula is used to explain.

[0203] The gradient magnitude entropy of non-edge points can be calculated by the following formula (10):

[0204]

[0205] Where H(B) is the gradient amplitude entropy of non-edge points, S is the number of non-edge points, and P i is the probability of occurrence of the gradient amplitude of non-edge point i, and Z is the probability and value of the gradient of each non-edge point.

[0206] The gradient magnitude entropy of the edge point can be calculated by the following formula (11):

[0207]

[0208] Among them, H(F) is the gradient amplitude entropy of the edge point, S is the number of non-edge points, N is the number of all pixels, P j is the probability of occurrence of the gradient amplitude of edge point j, and Z is the probability and value of the gradient of each non-edge point.

[0209] Finally, the gradient entropy of the non-edge point and the gradient entropy of the edge point can be integrated by the following formula (12) to obtain the gradient entropy of the grayscale image of the frame:

[0210] H=H(B)+H(F) (12)

[0211] Among them, H is the gradient entropy of the image, H(B) is the gradient amplitude entropy of non-edge points, and H(F) is the gradient amplitude entropy of edge points.

[0212] Step 28: Calculate a threshold value based on the gradient corresponding to each frame of the grayscale image, and perform binarization processing on each frame of the grayscale image to obtain a first region.

[0213] In the implementation of the present application, the acquisition process for the first area, that is, step 28, may specifically include: in each frame of grayscale image, the grayscale of the pixel point whose pixel is greater than the gradient calculation threshold corresponding to the frame of grayscale image is set to 255, and the pixel point can be represented as the foreground, and the grayscale of the pixel point whose pixel is greater than the gradient calculation threshold corresponding to the frame of grayscale image is set to 0, and the pixel point can be represented as the background.

[0214] In this way, by segmenting the foreground and the background, the area where all the pixels represented as the foreground are located can be used as the first area.

[0215] Step 29: Calculate the sum of the areas of the pixels in the first region in each frame of image to obtain the pixel area of ​​the first region in each frame of image.

[0216] Correspondingly, by summing the areas of the foreground pixels, the pixel area Ss of the first region can be obtained.

[0217] S202: Calculate the static pixel ratio in the video detection area corresponding to each frame of image based on the pixel area of ​​the first area in each frame of image and the pixel area of ​​the second area in each frame of image.

[0218] In practical applications, by calculating the pixel area Ss of the first area divided by the pixel area Sc of the second area in each frame of image, the static pixel ratio ratio_static in the video detection area corresponding to each frame of image can be obtained.

[0219] S203: Calculate the dynamic pixel ratio in the video detection area corresponding to each frame of image based on the pixel points of each frame of image and the pixel area of ​​the second area in each frame of image.

[0220] In practical applications, when the video detection target is completely still, running a differential operation between the pixels of two consecutive frames will result in a completely black result, that is, the two frames are completely consistent, and the differential results are all [0,0,0]. When the video detection target is moving, the differential result between the pixels of two consecutive frames can represent the different pixels between the two frames, that is, the edge of the moving video detection target.

[0221] Based on this, for step S203, first, a difference operation may be performed between the pixel points of each frame image and the pixel points corresponding to the position in the previous frame image to obtain the difference result corresponding to the pixel points in each frame image.

[0222] Next, the difference result corresponding to the pixel points in each frame of the image may be converted into a binarization result based on the difference result corresponding to the pixel points in each frame of the image and a binarization calculation threshold.

[0223] The binarization calculation threshold corresponding to the pixel point in each frame image is the same as the gradient calculation threshold corresponding to the grayscale image of the frame image. In addition, the binarization result corresponding to the pixel point in each frame image is used to indicate whether the pixel point in each frame image is a background pixel point or a non-background pixel point.

[0224] Based on this, in the specific implementation, the absolute value of the differential result corresponding to the pixel points in each frame of the image can be taken first to obtain the differential absolute value corresponding to the pixel points in each frame of the image. Then, among the differential absolute values ​​corresponding to the pixel points in each frame of the image, the differential absolute value greater than the binarization calculation threshold corresponding to the pixel points in each frame of the image is set to 1, and the pixel point can be represented as the foreground, and the differential absolute value less than the binarization calculation threshold corresponding to the pixel points in each frame of the image is set to 0, and the pixel point can be represented as the background. In this way, the conversion from the differential result to the binarization result is completed.

[0225] Furthermore, the binarization result may be subjected to morphological filtering, first dilation processing and then erosion processing, so as to make the contour in the image more closed and reduce the holes in the contour, and obtain the final processing result.

[0226] Finally, the dynamic pixel ratio corresponding to each frame of image may be calculated according to the number of non-background pixel points in the pixel points in each frame of image and the pixel area of ​​the second region in each frame of image.

[0227] Correspondingly, for the above-mentioned final processing results, the calculation process of the dynamic pixel ratio corresponding to each frame image can be expressed as: using the final processing results for statistics, the number of pixel points with non-zero values ​​in the final processing results is obtained, and the dynamic pixel ratio ratio_dynamic corresponding to each frame image is obtained by calculating the number of pixel points with non-zero values ​​and dividing it by the pixel area Sc of the second region.

[0228] In addition, it should be noted that in the differential operation process, the differential operation is specifically performed on the pixel points of each frame image and the pixel points corresponding to the position in the previous frame image. Therefore, in practical applications, the cache variable of the previous frame image can be saved. As the video data is constantly updated, each time a new frame image is received and processed, the current frame image is updated to the cache variable of the previous frame image. Therefore, when the data processing process is started, if the previous frame image is empty, no subsequent processing is performed, and only the cache variable of the previous frame image is updated.

[0229] S204: Taking the dynamic pixel ratio corresponding to each frame image and the static pixel ratio corresponding to each frame image as a plurality of second data.

[0230] Based on the relevant contents of the above steps S201-S204, it can be known that in an embodiment of the present application, by carefully analyzing the video data, the dynamic pixel ratio corresponding to each frame image and the static pixel ratio corresponding to each frame image can be obtained as multiple second data. In this way, it is helpful to accurately obtain the second correlation coefficient used to describe the lane congestion situation in the video detection area. Subsequently, based on the integration of the first correlation coefficient and the second correlation coefficient, the congestion coefficient of the lane can be comprehensively obtained by combining the two means of radar detection and video detection. In this way, the complementarity between the two means of radar detection and video detection can make the above congestion coefficient more accurately represent the congestion degree of the lane, and minimize the impact of environmental factors as much as possible. Subsequently, based on the congestion coefficient and the corresponding congestion threshold, the congestion detection result of the lane can be accurately obtained, thereby improving the accuracy of lane congestion detection.

[0231] Further, based on the lane congestion detection method provided in the above embodiment, the embodiment of the present application can also provide a lane congestion detection device. The lane congestion detection device is described below in conjunction with the embodiments and drawings.

[0232] Figure 3 A schematic diagram of the structure of a lane congestion detection device provided in an embodiment of the present application. Figure 3As shown, the lane congestion detection device 300 provided in the embodiment of the present application may include:

[0233] The data analysis module 301 is used to analyze radar data of a radar detection target in a radar detection area to obtain a plurality of first data for representing a trajectory of the radar detection target, and to analyze video data of a video detection target in a video detection area to obtain a plurality of second data for representing the trajectory of the video detection target, wherein the radar detection target is a target in the radar detection area of ​​the lane, and the video detection target is a detection target in the video detection area of ​​the lane, and the radar detection area and the video detection area are unified;

[0234] a coefficient determination module 302, configured to determine a first correlation coefficient between the plurality of first data based on the plurality of first data, and to determine a second correlation coefficient between the plurality of second data based on the plurality of second data, wherein the first correlation coefficient is used to describe a lane congestion condition in the radar detection area, and the second correlation coefficient is used to describe a lane congestion condition in the video detection area;

[0235] A coefficient integration module 303, configured to integrate the first correlation coefficient and the second correlation coefficient to obtain a congestion coefficient of the lane;

[0236] The congestion detection module 304 is used to determine the congestion detection result of the lane based on the congestion coefficient and a preset congestion threshold.

[0237] Optionally, the data analysis module 301 includes:

[0238] An area determination module, used to determine the pixel area of ​​a first area in each frame image of the video data and the pixel area of ​​a second area in each frame image, wherein the first area is an area enclosed by an edge of the video detection target, and the second area is an area of ​​interest in the video detection area;

[0239] A first ratio determination module, configured to calculate a static pixel ratio in a video detection area corresponding to each frame of image based on a pixel area of ​​the first area in each frame of image and a pixel area of ​​the second area in each frame of image;

[0240] A second ratio determination module, configured to calculate the dynamic pixel ratio in the video detection area corresponding to each frame of image based on the pixel points of each frame of image and the pixel area of ​​the second area in each frame of image;

[0241] The second data acquisition module is used to use the dynamic pixel ratio corresponding to each frame image and the static pixel ratio corresponding to each frame image as the plurality of second data.

[0242] Optionally, the second ratio determination module includes:

[0243] A differential operation module, used for performing differential operation on the pixel points of each frame image and the pixel points corresponding to the position in the previous frame image, to obtain the differential results corresponding to the pixel points in each frame image;

[0244] A first binarization processing module, configured to convert the differential result corresponding to the pixel points in each frame of the image into a binarization result based on the differential result corresponding to the pixel points in each frame of the image and a binarization calculation threshold, wherein the binarization result corresponding to the pixel points in each frame of the image is used to indicate whether the pixel points in each frame of the image are background pixel points or non-background pixel points;

[0245] The second ratio determination submodule is used to calculate the dynamic pixel ratio corresponding to each frame of image according to the number of non-background pixel points in the pixel points in each frame of image and the pixel area of ​​the second region in each frame of image.

[0246] Optionally, the area determination module includes:

[0247] A matrix creation module, used for creating an image matrix with the same size as the video data, and setting the pixels of the pixels in the image matrix to black;

[0248] A first pixel modification module, for modifying pixels of pixels corresponding to the position of the target pixel point in the image matrix to white based on the target pixel point in the video detection area, wherein the target pixel point is a pixel point in the video detection area;

[0249] A second pixel modification module is used to traverse the pixel points of each frame of the image, and based on the black pixel points in the image matrix, modify the pixels of the pixel points corresponding to the positions of the black pixel points in each frame of the image to black, so as to obtain each new frame of the image, and perform grayscale processing on each new frame of the image to obtain each frame of the grayscale image;

[0250] The first area determination submodule is used to determine the number of non-black pixels in each frame of grayscale image based on each frame of grayscale image, and the number of non-black pixels in each frame of grayscale image is used to represent the pixel area of ​​the second area in each frame of image.

[0251] Optionally, the area determination module further includes:

[0252] a gradient calculation module, configured to calculate the gradient of each pixel in each frame of the grayscale image in multiple directions by using multiple convolution kernels, wherein the multiple directions include a horizontal direction, a direction at 45° to the horizontal direction, a vertical direction, and a direction at 135° to the horizontal direction, and one of the multiple convolution kernels corresponds to one of the multiple directions;

[0253] A gradient merging module, used for merging the gradients of each pixel point in each frame of the grayscale image in the multiple directions to obtain the gradient amplitude of each pixel point in each frame of the grayscale image;

[0254] a threshold determination module, for determining, for an i-th grayscale image frame in the video data, a gradient calculation threshold corresponding to the i-th grayscale image frame based on the gradient amplitude of the i-th grayscale image frame and a preset initial calculation threshold, 3≤i≤the total number of frames of the video data, and the gradient calculation thresholds corresponding to the first grayscale image frame and the second grayscale image frame respectively are the initial calculation thresholds;

[0255] A second binarization processing module, configured to perform binarization processing on each frame of the grayscale image based on a gradient calculation threshold corresponding to each frame of the grayscale image, to obtain the first region;

[0256] The second area determination submodule is used to calculate the sum of the areas of the pixels in the first area in each frame of the image to obtain the pixel area of ​​the first area in each frame of the image.

[0257] Optionally, the threshold determination module is specifically used to:

[0258] Based on the difference between the gradient entropy of the i-1th grayscale image frame and the gradient entropy of the i-2th grayscale image frame, adjusting the gradient calculation threshold corresponding to the i-1th grayscale image frame to obtain the gradient calculation threshold corresponding to the i-th grayscale image frame, and looping until the absolute value of the difference between the gradient calculation thresholds corresponding to the consecutive multiple grayscale images is less than or equal to the preset difference, then determining the gradient calculation thresholds corresponding to the grayscale images of the remaining frames of the video data after the consecutive multiple grayscale images based on the gradient calculation thresholds corresponding to the consecutive multiple grayscale images;

[0259] The gradient entropy of each frame of grayscale image is calculated based on the gradient amplitude of each frame of grayscale image and the gradient calculation threshold corresponding to each frame of grayscale image.

[0260] Optionally, the plurality of second data include a dynamic pixel ratio and a static pixel ratio within the video detection area;

[0261] The coefficient determination module 302 is specifically used for:

[0262] Starting from the second frame image of the video data, collecting dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images respectively;

[0263] Calculating a first covariance between dynamic pixel ratios and static pixel ratios respectively corresponding to the multiple frames of images, a first standard deviation between dynamic pixel ratios respectively corresponding to the multiple frames of images, and a second standard deviation between static pixel ratios respectively corresponding to the multiple frames of images;

[0264] The second correlation coefficient is calculated based on the first covariance, the first standard deviation, and the second standard deviation.

[0265] Optionally, the plurality of first data include an area ratio of the radar detection target in a radar detection area, and an average speed of the radar detection target;

[0266] The coefficient determination module 302 is specifically used for:

[0267] Starting from the second frame of the radar data, collecting area percentages and average speeds corresponding to multiple frames of radar data respectively;

[0268] Calculating a second covariance between the area proportions and the average speeds respectively corresponding to the multiple frames of radar data, a third standard deviation between the area proportions respectively corresponding to the multiple frames of radar data, and a fourth standard deviation between the average speeds respectively corresponding to the multiple frames of radar data;

[0269] The first correlation coefficient is calculated based on the second covariance, the third standard deviation, and the fourth standard deviation.

[0270] Optionally, the plurality of first data include an area ratio of the radar detection target in the radar detection area and an average speed of the radar detection target, and the plurality of second data include a dynamic pixel ratio and a static pixel ratio in the video detection area;

[0271] The coefficient integration module 303 is specifically used for:

[0272] Starting from the second frame of the radar data, collecting the area ratios and average speeds corresponding to the multiple frames of radar data; and starting from the second frame of the video data, collecting the dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images;

[0273] Calculating a first ratio between the plurality of first data based on an average value of area proportions respectively corresponding to the plurality of frames of radar data and an average value of average speeds respectively corresponding to the plurality of frames of radar data, and calculating a second ratio between the plurality of second data based on an average value of dynamic pixel ratios respectively corresponding to the plurality of frames of images and an average value of static pixel ratios respectively corresponding to the plurality of frames of images;

[0274] Calculate an intermediate coefficient based on the first correlation coefficient, the absolute value of the first correlation coefficient, the second correlation coefficient, the absolute value of the second correlation coefficient, the first ratio, and the second ratio;

[0275] The intermediate coefficient is converted through an activation function to obtain the congestion coefficient.

[0276] Optionally, the preset congestion threshold includes a first congestion threshold, a second congestion threshold, a third congestion threshold and a fourth congestion threshold, the first congestion threshold is greater than the second congestion threshold, the second congestion threshold is greater than the third congestion threshold, and the third congestion threshold is greater than the fourth congestion threshold;

[0277] The congestion detection module 304 is specifically used for:

[0278] When the congestion coefficient is greater than or equal to the first congestion threshold, determining the congestion detection result as the lane being at a first congestion level;

[0279] When the congestion coefficient is greater than or equal to the second congestion threshold and less than the first congestion threshold, determining the congestion detection result as the lane being at a second congestion level;

[0280] When the congestion coefficient is greater than or equal to a third congestion threshold and less than the second congestion threshold, determining the congestion detection result as the lane being at a third congestion level;

[0281] When the congestion coefficient is greater than or equal to a fourth congestion threshold and less than the third congestion threshold, determining the congestion detection result as the lane being at a fourth congestion level;

[0282] The fourth congestion level is higher than the third congestion level, the third congestion level is higher than the second congestion level, and the second congestion level is higher than the first congestion level.

[0283] Furthermore, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a system bus;

[0284] The processor and the memory are connected via the system bus;

[0285] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any implementation step of the above-mentioned lane congestion detection method.

[0286] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on an electronic device, any step of implementing the above-mentioned lane congestion detection method is implemented.

[0287] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments. It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0288] As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0289] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0290] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lane congestion detection method, characterized in that: include: Analyzing radar data of a radar detection target within a radar detection area to obtain a plurality of first data for representing a trajectory of the radar detection target, and analyzing video data of a video detection target within a video detection area to obtain a plurality of second data for representing the trajectory of the video detection target, wherein the radar detection target is a target within the radar detection area of ​​the lane, and the video detection target is a detection target within the video detection area of ​​the lane, and the radar detection area and the video detection area are unified; Determine a first correlation coefficient between the plurality of first data based on the plurality of first data, and determine a second correlation coefficient between the plurality of second data based on the plurality of second data, wherein the first correlation coefficient is used to describe a lane congestion condition within the radar detection area, and the second correlation coefficient is used to describe a lane congestion condition within the video detection area; Integrating the first correlation coefficient and the second correlation coefficient to obtain a congestion coefficient of the lane; Based on the congestion coefficient and a preset congestion threshold, a congestion detection result of the lane is determined.

2. The lane congestion detection method according to claim 1, characterized in that: The analyzing the video data corresponding to the video detection target to obtain a plurality of second data for representing the trajectory of the video detection target includes: Determine a pixel area of ​​a first region in each frame image of the video data and a pixel area of ​​a second region in each frame image, wherein the first region is a region enclosed by an edge of the video detection target, and the second region is a region of interest in the video detection region; Calculating the static pixel ratio in the video detection area corresponding to each frame of image based on the pixel area of ​​the first area in each frame of image and the pixel area of ​​the second area in each frame of image; Calculate the dynamic pixel ratio in the video detection area corresponding to each frame of image based on the pixel points of each frame of image and the pixel area of ​​the second area in each frame of image; The dynamic pixel ratio corresponding to each frame image and the static pixel ratio corresponding to each frame image are used as the plurality of second data.

3. The lane congestion detection method according to claim 2, characterized in that: The calculating, based on the pixel points of each frame of image and the pixel area of ​​the second area in each frame of image, the dynamic pixel ratio in the video detection area corresponding to each frame of image includes: Performing a differential operation on the pixel points of each frame of image and the pixel points corresponding to the positions in the previous frame of image to obtain the differential results corresponding to the pixel points in each frame of image; Based on the differential result corresponding to the pixel points in each frame of the image and the binarization calculation threshold, the differential result corresponding to the pixel points in each frame of the image is converted into a binarization result, and the binarization result corresponding to the pixel points in each frame of the image is used to indicate whether the pixel points in each frame of the image are background pixel points or non-background pixel points; The dynamic pixel ratio corresponding to each frame of image is calculated according to the number of non-background pixel points in the pixel points in each frame of image and the pixel area of ​​the second region in each frame of image.

4. The lane congestion detection method according to claim 2, characterized in that: The pixel area of ​​the second region in each frame of image is determined by the following steps: Creating an image matrix having the same size as the video data, and setting the pixels of the pixels in the image matrix to black; Based on the target pixel point in the video detection area, modify the pixel point corresponding to the position of the target pixel point in the image matrix to white, wherein the target pixel point is a pixel point in the video detection area; Traversing the pixel points of each frame of the image, based on the black pixel points in the image matrix, modifying the pixels of the pixel points corresponding to the positions of the black pixel points in each frame of the image to black, to obtain each new frame of the image, and performing grayscale processing on each new frame of the image to obtain each frame of the grayscale image; Based on each frame of grayscale image, the number of non-black pixels in each frame of grayscale image is determined, and the number of non-black pixels in each frame of grayscale image is used to represent the pixel area of ​​the second region in each frame of image.

5. The lane congestion detection method according to claim 4, characterized in that: The pixel area of ​​the first region in each frame of image is determined by the following steps: By using a plurality of convolution kernels, the gradient of each pixel in each frame of the grayscale image in a plurality of directions is calculated, wherein the plurality of directions include a horizontal direction, a direction at 45° to the horizontal direction, a vertical direction, and a direction at 135° to the horizontal direction, and one of the plurality of convolution kernels corresponds to one of the plurality of directions; Merging the gradients of each pixel in each frame of the grayscale image in the multiple directions to obtain the gradient amplitude of each pixel in each frame of the grayscale image; For the i-th grayscale image frame in the video data, based on the gradient amplitude of the i-th grayscale image frame and a preset initial calculation threshold, determine the gradient calculation threshold corresponding to the i-th grayscale image frame, 3≤i≤the total number of frames of the video data, and the gradient calculation thresholds corresponding to the first grayscale image frame and the second grayscale image frame respectively are the initial calculation thresholds; Based on the gradient calculation threshold corresponding to each frame of the grayscale image, binarize each frame of the grayscale image to obtain the first region; The sum of the areas of the pixels in the first region in each frame of the image is calculated to obtain the pixel area of ​​the first region in each frame of the image.

6. The lane congestion detection method according to claim 5, characterized in that: The step of determining the gradient calculation threshold corresponding to the i-th grayscale image frame based on the gradient amplitude of the i-th grayscale image frame in the video data and a preset initial calculation threshold comprises: Based on the difference between the gradient entropy of the i-1th grayscale image frame and the gradient entropy of the i-2th grayscale image frame, adjusting the gradient calculation threshold corresponding to the i-1th grayscale image frame to obtain the gradient calculation threshold corresponding to the i-th grayscale image frame, and looping until the absolute value of the difference between the gradient calculation thresholds corresponding to the consecutive multiple grayscale images is less than or equal to the preset difference, then determining the gradient calculation thresholds corresponding to the grayscale images of the remaining frames of the video data after the consecutive multiple grayscale images based on the gradient calculation thresholds corresponding to the consecutive multiple grayscale images; The gradient entropy of each frame of grayscale image is calculated based on the gradient amplitude of each frame of grayscale image and the gradient calculation threshold corresponding to each frame of grayscale image.

7. The lane congestion detection method according to claim 1, characterized in that: The plurality of second data include a dynamic pixel ratio and a static pixel ratio within the video detection area; The determining a second correlation coefficient between the plurality of second data based on the plurality of second data comprises: Starting from the second frame image of the video data, collecting dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images respectively; Calculating a first covariance between dynamic pixel ratios and static pixel ratios respectively corresponding to the multiple frames of images, a first standard deviation between dynamic pixel ratios respectively corresponding to the multiple frames of images, and a second standard deviation between static pixel ratios respectively corresponding to the multiple frames of images; The second correlation coefficient is calculated based on the first covariance, the first standard deviation, and the second standard deviation.

8. The lane congestion detection method according to claim 1, characterized in that: The plurality of first data include an area ratio of the radar detection target in the radar detection area and an average speed of the radar detection target; The determining a first correlation coefficient between the plurality of first data based on the plurality of first data comprises: Starting from the second frame of the radar data, collecting area percentages and average speeds corresponding to multiple frames of radar data respectively; Calculating a second covariance between the area proportions and the average speeds respectively corresponding to the multiple frames of radar data, a third standard deviation between the area proportions respectively corresponding to the multiple frames of radar data, and a fourth standard deviation between the average speeds respectively corresponding to the multiple frames of radar data; The first correlation coefficient is calculated based on the second covariance, the third standard deviation, and the fourth standard deviation.

9. The lane congestion detection method according to claim 1, characterized in that: The plurality of first data include an area ratio of the radar detection target in the radar detection area and an average speed of the radar detection target, and the plurality of second data include a dynamic pixel ratio and a static pixel ratio in the video detection area; The integrating the first correlation coefficient and the second correlation coefficient to obtain the congestion coefficient of the lane includes: Starting from the second frame of the radar data, collecting the area ratios and average speeds corresponding to the multiple frames of radar data; and starting from the second frame of the video data, collecting the dynamic pixel ratios and static pixel ratios corresponding to the multiple frames of images; Calculating a first ratio between the plurality of first data based on an average value of area proportions respectively corresponding to the plurality of frames of radar data and an average value of average speeds respectively corresponding to the plurality of frames of radar data, and calculating a second ratio between the plurality of second data based on an average value of dynamic pixel ratios respectively corresponding to the plurality of frames of images and an average value of static pixel ratios respectively corresponding to the plurality of frames of images; Calculate an intermediate coefficient based on the first correlation coefficient, the absolute value of the first correlation coefficient, the second correlation coefficient, the absolute value of the second correlation coefficient, the first ratio, and the second ratio; The intermediate coefficient is converted through an activation function to obtain the congestion coefficient.

10. The lane congestion detection method according to claim 1, characterized in that: The preset congestion thresholds include a first congestion threshold, a second congestion threshold, a third congestion threshold and a fourth congestion threshold, the first congestion threshold is greater than the second congestion threshold, the second congestion threshold is greater than the third congestion threshold, and the third congestion threshold is greater than the fourth congestion threshold; The determining the congestion detection result of the lane based on the congestion coefficient and a preset congestion threshold comprises: When the congestion coefficient is greater than or equal to the first congestion threshold, determining the congestion detection result as the lane being at a first congestion level; When the congestion coefficient is greater than or equal to the second congestion threshold and less than the first congestion threshold, determining the congestion detection result as the lane being at a second congestion level; When the congestion coefficient is greater than or equal to a third congestion threshold and less than the second congestion threshold, determining the congestion detection result as the lane being at a third congestion level; When the congestion coefficient is greater than or equal to a fourth congestion threshold and less than the third congestion threshold, determining the congestion detection result as the lane being at a fourth congestion level; The fourth congestion level is higher than the third congestion level, the third congestion level is higher than the second congestion level, and the second congestion level is higher than the first congestion level.

11. A lane congestion detection device, characterized in that: include: a data analysis module, configured to analyze radar data of a radar detection target within a radar detection area to obtain a plurality of first data for representing a trajectory of the radar detection target, and to analyze video data of a video detection target within a video detection area to obtain a plurality of second data for representing the trajectory of the video detection target, wherein the radar detection target is a target within the radar detection area of ​​the lane, and the video detection target is a detection target within the video detection area of ​​the lane, and the radar detection area and the video detection area are unified; a coefficient determination module, configured to determine a first correlation coefficient between the plurality of first data based on the plurality of first data, and to determine a second correlation coefficient between the plurality of second data based on the plurality of second data, wherein the first correlation coefficient is used to describe a lane congestion condition within the radar detection area, and the second correlation coefficient is used to describe a lane congestion condition within the video detection area; A coefficient integration module, used for integrating the first correlation coefficient and the second correlation coefficient to obtain a congestion coefficient of the lane; The congestion detection module is used to determine the congestion detection result of the lane based on the congestion coefficient and a preset congestion threshold.

12. An electronic device, characterized in that: The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the steps of the lane congestion detection method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a terminal device, the steps of the lane congestion detection method according to any one of claims 1 to 10 are implemented.