Traffic abnormal event detection method and device, storage medium, program product and computer equipment

Abstract

The invention discloses a traffic abnormal event detection method and device, a storage medium, a program product and computer equipment, and the method comprises the steps: obtaining multi-modal data corresponding to a target region, wherein the multi-modal data comprises point cloud data and video data; on the basis of the alignment parameters, aligning the multi-modal data; performing target detection based on the aligned multi-modal data to obtain a point cloud detection result and a video detection result; fusing the aligned multi-modal data based on a point cloud detection result and a video detection result to obtain fused data; and carrying out traffic abnormal event detection based on the fused data to obtain a traffic abnormal event detection result, so that the accuracy of traffic abnormal event detection can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, storage medium, program product, and computer equipment for detecting traffic anomalies. Background Technology

[0002] In related technologies, it is common practice to analyze video footage captured by cameras to determine whether a traffic incident has occurred.

[0003] However, the images captured by the camera under certain special conditions (such as bad weather, night, and insufficient light) are not clear enough to be used for accurate event analysis, resulting in low accuracy in traffic anomaly detection. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application propose a method, apparatus, storage medium, program product, and computer equipment for detecting traffic anomalies, which can improve the accuracy of traffic anomaly detection.

[0005] In a first aspect, embodiments of this application provide a method for detecting traffic anomalies, including: Acquire multimodal data corresponding to the target area, wherein the multimodal data includes point cloud data and video data; The multimodal data is aligned based on the alignment parameters; Target detection is performed based on the aligned multimodal data to obtain point cloud detection results and video detection results; Based on the point cloud detection results and the video detection results, the aligned multimodal data are fused to obtain fused data; Traffic anomaly event detection is performed based on the fused data to obtain traffic anomaly event detection results.

[0006] Optionally, the alignment parameters include time parameters, calibration parameters, and / or reference points corresponding to the target region; The calibration parameters are obtained through the following methods: The video data in the multimodal data corresponding to the target area is mapped to the coordinate system of the point cloud data in the multimodal data corresponding to the target area to obtain video mapping data; The calibration parameters are determined using the least squares method based on the video mapping data and the point cloud data in the multimodal data corresponding to the target region.

[0007] Optionally, the step of fusing the aligned multimodal data based on the point cloud detection results and the video detection results to obtain fused data includes: The point cloud detection results are mapped to the image coordinate system corresponding to the video detection results to obtain the point cloud detection mapping results; Based on the point cloud detection mapping results and the video detection results, the aligned multimodal data are fused to obtain fused data.

[0008] Optionally, the step of fusing the aligned multimodal data based on the point cloud detection mapping result and the video detection result to obtain fused data includes: Based on the point cloud detection mapping results and the video detection results, determine the Intersection over Union (IOU) matrix set; Based on the IOU matrix set, the Hungarian algorithm is used to determine the globally optimal matching result; Based on the global optimal matching result and the preset IOU threshold, the aligned multimodal data is fused to obtain fused data.

[0009] Optionally, the step of detecting traffic anomalies based on the fused data to obtain traffic anomaly detection results includes: Based on the fused data, corresponding feature variables are constructed; Based on the aforementioned feature variables, the abnormal event detector is invoked to detect abnormal traffic events, and the results of the abnormal traffic event detection are obtained.

[0010] Optionally, the method further includes: Obtain evidence information for abnormal events corresponding to the traffic abnormality detection results.

[0011] Secondly, embodiments of this application provide a traffic anomaly detection device, comprising: A multimodal data acquisition module is used to acquire multimodal data corresponding to a target area, wherein the multimodal data includes point cloud data and video data; An alignment module is used to align the multimodal data based on alignment parameters; The target detection module is used to perform target detection based on the aligned multimodal data, and obtain point cloud detection results and video detection results; The fusion module is used to fuse the aligned multimodal data based on the point cloud detection results and the video detection results to obtain fused data; The event detection module is used to detect abnormal traffic events based on the fused data and obtain the abnormal traffic event detection results.

[0012] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0013] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0014] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0015] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, multimodal data corresponding to a target area is acquired, wherein the multimodal data includes point cloud data and video data; the multimodal data is aligned based on alignment parameters; target detection is performed based on the aligned multimodal data to obtain point cloud detection results and video detection results; the aligned multimodal data is fused based on the point cloud detection results and the video detection results to obtain fused data; and traffic anomaly detection is performed based on the fused data to obtain traffic anomaly detection results. In this way, the probability of low accuracy in traffic anomaly detection due to unclear camera footage can be reduced by combining multimodal data, and the amount of information used for event detection can be enriched even when the camera footage is sufficiently clear, thereby improving the accuracy of traffic anomaly detection. Furthermore, precise data fusion can be achieved through alignment and target detection, improving the reliability of fusion between different modal data, thereby further improving the accuracy of traffic anomaly detection. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of the traffic anomaly detection method provided in the embodiments of this application; Figure 2 This is a schematic diagram of data fusion provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating matching based on IOU provided in an embodiment of this application; Figure 4 This is a schematic diagram of the first part of the Hungarian algorithm provided in the embodiments of this application; Figure 5 This is a schematic diagram of the second part of the Hungarian algorithm provided in the embodiments of this application; Figure 6 This is a schematic diagram of the third part of the Hungarian algorithm provided in the embodiments of this application; Figure 7 This is a schematic diagram of the fourth part of the Hungarian algorithm provided in the embodiments of this application; Figure 8This is a schematic diagram of the fifth part of the Hungarian algorithm provided in the embodiments of this application; Figure 9 This is a schematic diagram of traffic anomaly detection provided in an embodiment of this application; Figure 10 This is a schematic diagram of the traffic anomaly detection device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments / examples are only a part of the embodiments / examples of this application, and not all of the embodiments / examples. Based on the embodiments / examples in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment / example" means "at least one embodiment / example"; the term "another embodiment / example" means "at least one additional embodiment / example"; the term "some embodiments / examples" means "at least some embodiments / examples."

[0019] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0020] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0021] Firstly, see [the following] Figure 1 The diagram shows a flowchart of a traffic anomaly detection method provided in an embodiment of this application. This traffic anomaly detection method can be applied to a computer device with data processing capabilities. The method includes steps S101-105, as detailed below.

[0022] S101, acquire multimodal data corresponding to the target area, wherein the multimodal data includes point cloud data and video data.

[0023] In some examples, the point cloud data may be acquired by radar equipment, and the video data may be acquired by a camera. Hardware clock synchronization devices, such as GPS time stamps or dedicated time synchronization hardware, can be used as a time source to ensure that the camera and radar equipment are in sync. For example, an accurate timestamp can be provided to the camera and radar equipment via a hardware clock synchronization device. Furthermore, the camera and radar systems are calibrated and synchronized during initial setup and installation to ensure temporal and spatial consistency.

[0024] In some examples, timestamps can be added to each camera video frame and radar point cloud data during multimodal data acquisition. The added timestamps are based on the same time source and have the same time format.

[0025] S102, Align the multimodal data based on alignment parameters.

[0026] In some examples, spatiotemporal alignment parameters can be used to align multimodal data spatiotemporally.

[0027] In some examples, reference points (such as known landmarks or markers) can be used to pinpoint the locations of cameras and radar. These reference points can serve as reference points for a spatiotemporal base to help align the data. After the radar is installed and deployed on-site, the point cloud data can be analyzed by acquiring the radar's perception data for a specific location range. This allows for the adjustment of the radar's position parameters to obtain the radar's latitude, longitude, and heading angle at the optimal point cloud matching state, thereby determining the radar's precise location.

[0028] S103, target detection is performed based on the aligned multimodal data to obtain point cloud detection results and video detection results.

[0029] In some examples, target tracking or object detection algorithms can be used to perform target detection on the point cloud data and video data in the above-aligned multimodal data, respectively, to obtain point cloud detection results and video detection results.

[0030] S104, Based on the point cloud detection results and the video detection results, the aligned multimodal data is fused to obtain fused data.

[0031] In some examples, each detected target in the point cloud detection results can be associated and matched with each detected target in the video detection results (e.g., analyzing the position of the same target in the point cloud detection results and the video detection results). Then, the aligned multimodal data can be fused according to the association and matching results to obtain fused data.

[0032] In some examples, video data and point cloud data can be fused to improve trajectory accuracy and accurately obtain target location, identity, and motion information. Fusion of radar point cloud data and camera video data typically involves multi-sensor fusion, which includes transformations between sensor coordinate systems and data correlation. This process will be explained in a simplified manner below, using mathematical principles.

[0033] Since radar and cameras are typically located in different coordinate systems, their data can be transformed into a common coordinate system for fusion. For example, suppose there is a radar coordinate system corresponding to the point cloud detection results. and camera coordinate system and a common coordinate system .

[0034] For point cloud data, the points (In the radar coordinate system) it can be transformed to a common coordinate system: ,in, It is the transformation matrix from the radar coordinate system to the common coordinate system.

[0035] For camera data, the points on its image (In the camera coordinate system) it can also be transformed to a common coordinate system: ,in, It is the transformation matrix from the camera coordinate system to the common coordinate system.

[0036] Thus, after the aligned multimodal data is transformed into the aforementioned common coordinate system, the multimodal data transformed into the aforementioned common coordinate system can be fused based on the point cloud detection results and the video detection results to obtain fused data.

[0037] Understandably, radar point cloud data can be used to provide distance and velocity information, while camera video data can be used to provide color and texture information.

[0038] In some examples, multi-sensor fusion techniques, such as Kalman filtering and / or particle filtering, can be used during fusion to obtain fused data.

[0039] S105, based on the fused data, traffic anomaly event detection is performed to obtain traffic anomaly event detection results.

[0040] In some examples, large language models can be used to detect traffic anomalies on fused data to obtain traffic anomaly detection results.

[0041] In one optional implementation, the alignment parameters include time parameters, calibration parameters, and / or a reference point corresponding to the target region; The calibration parameters are obtained through the following methods: The video data in the multimodal data corresponding to the target area is mapped to the coordinate system of the point cloud data in the multimodal data corresponding to the target area to obtain video mapping data; The calibration parameters are determined using the least squares method based on the video mapping data and the point cloud data in the multimodal data corresponding to the target region.

[0042] In some examples, calibration algorithms can be used to estimate the relative positions and attitudes between different sensors (radar, camera, etc.) corresponding to multimodal data. By mounting sensors in the measurement system and ensuring the accuracy of their position and orientation, the mounting positions and angles of the camera and radar can be adjusted based on the measurement results to align the sensor measurement data with the coordinates of the calibration target. In this embodiment, the two-dimensional data of the image in the video data can be mapped to the three-dimensional world of the point cloud data. Through coordinate transformation, the coordinate system of the camera's video data can be made consistent with that of the point cloud data of the calibration target. After data alignment, the coordinates of the sensor measurement data and the calibration target can be processed based on the least squares method to calculate calibration parameters. By analyzing the relationship between the sensor measurement data and the coordinates of the calibration target, calibration parameters, such as the camera's internal parameters (focal length, distortion coefficient, etc.) and external parameters (camera attitude, position, etc.), can be calculated.

[0043] Taking camera focal length calibration as an example, the camera's focal length may have some errors. The actual focal length of the camera can be estimated using the least squares method, based on the measured pixel position of the calibration target in the image and the actual distance of the calibration target in the real world (which can be estimated from point cloud data). The corresponding focal length can be estimated using the following formula: ,in, It is the pixel distance of the object in the image. It refers to the camera's focal length, and the parameters to be estimated. It is the actual distance of an object in the real world.

[0044] Suppose there is a set of N measurement points, namely The goal is to find the focal length that minimizes the residual. The residual is the difference between the predicted and observed values ​​at each measurement point. The sum of squared residuals in the least squares method can be defined as: .

[0045] The goal is to find ,make Minimize, this can be achieved by... Find the answer by taking the partial derivatives and setting them to zero: .

[0046] Solving the above equations yields the least-squares estimate of the focal length. .

[0047] Thus, this formula describes the process of estimating the camera's focal length using the least squares method. By measuring the pixel distance and actual distance of an object, this method can be used to calibrate the camera to obtain a more accurate focal length estimate.

[0048] In addition, sensors can be periodically inspected and recalibrated to ensure consistency in their spatiotemporal base. Factors such as temperature, humidity, and mechanical vibration can cause changes in sensor performance, necessitating recalibration.

[0049] In one optional implementation, the step of fusing the aligned multimodal data based on the point cloud detection results and the video detection results to obtain fused data includes: The point cloud detection results are mapped to the image coordinate system corresponding to the video detection results to obtain the point cloud detection mapping results; Based on the point cloud detection mapping results and the video detection results, the aligned multimodal data are fused to obtain fused data.

[0050] In some examples, see Figure 2This method maps point cloud detection results to the camera coordinate system, ensuring that the point cloud detection results and camera data reside on the same three-dimensional spatial reference. Subsequently, based on the camera's intrinsic and extrinsic parameters, the three-dimensional position in this three-dimensional space can be further converted into a two-dimensional pixel position in the image, thereby mapping the point cloud detection results to the image coordinate system to obtain the point cloud detection mapping result. Furthermore, before fusing the aligned multimodal data, the point cloud detection mapping result (e.g., point cloud mapped to a two-dimensional target bounding box) can be offset to correct potential deviations in the target bounding box (i.e., the rectangular area selecting the target in the image) generated after mapping the point cloud to the image coordinate system (e.g., due to sensor calibration errors, mapping algorithm accuracy issues, etc., causing the target bounding box not to accurately fit the actual target in the image).

[0051] In one optional implementation, the step of fusing the aligned multimodal data based on the point cloud detection mapping result and the video detection result to obtain fused data includes: Based on the point cloud detection mapping results and the video detection results, determine the Intersection over Union (IOU) matrix set; Based on the IOU matrix set, the Hungarian algorithm is used to determine the globally optimal matching result; Based on the global optimal matching result and the preset IOU threshold, the aligned multimodal data is fused to obtain fused data.

[0052] In some examples, IOU can be used as a metric (ranging from 0 to 1, with higher values ​​indicating greater overlap) to measure the degree of overlap between two bounding boxes (a bounding box of an object in a point cloud detection mapping and a bounding box of an object in a video detection mapping). The IOUs of any two bounding boxes can form an IOU matrix, and this set of IOU matrices can include all IOU matrices.

[0053] In some examples, see Figure 2 and Figure 3 The IOU matrix set can be input into the Hungarian algorithm to determine the global optimal matching result, so that the sum of the IOU of all matching pairs (i.e. matching pairs consisting of two target boxes) is maximized, thereby avoiding erroneous matching caused by local optima.

[0054] In some examples, each matching pair with an IOU not less than the IOU threshold can be selected from all matching pairs indicated by the global optimal matching result, and each selected matching pair can be divided into three results: successful matching, video detected but point cloud not detected, and point cloud detected but video not detected.

[0055] In some examples, the fused data may include the target ID of the successfully fused target, the video detection results of the unfused target (e.g., carrying the target ID, target bounding box position, camera ID, confidence score, and target category), and the point cloud detection results of the successfully fused target (e.g., carrying the heading angle, center point coordinates, target ID, dimension, longitude, target size, velocity, 3D information corresponding to the target's 8 corner points, frame number, camera ID, confidence score, lane number, and category).

[0056] In the embodiments of this application, the weakness of a single sensor can be effectively reduced by using multimodal data. For example, if the camera's field of view is blocked by a large vehicle or the radar is suspected of being blocked, small target vehicles may be lost for a short time, and the tracking chain may be broken.

[0057] It's important to note that IOU matching is a matching method in object detection used to match predicted bounding boxes with ground truth bounding boxes. IOU matching calculates an overlap metric based on the intersection and union of two boxes, measuring their similarity. Specifically, given a predicted bounding box and a ground truth bounding box, their intersection and union areas are first calculated. Then, the intersection area is divided by the union area to obtain the IOU value. If the IOU value is greater than a set threshold, the predicted bounding box is matched with the ground truth bounding box. IOU matching is a simple and effective matching method that takes into account the positional and size relationships between the predicted and ground truth bounding boxes, thus improving the accuracy of object detection. Furthermore, due to its simplicity and efficiency, IOU matching is widely used in object tracking, object segmentation, and other fields. For example, the IOU matching process is as follows.

[0058] Step 1: For each video detection result and point cloud detection result, calculate their 2D bounding boxes on the image; Step 2: For each video detection result and point cloud detection result, calculate their 3D bounding boxes; Step 3: For each video detection result and point cloud detection result, project their 3D bounding boxes onto the image to obtain their 2D bounding boxes on the image. Step 4: For each video detection result and point cloud detection result, calculate their IOU value, which is the area of ​​the overlap of their 2D bounding boxes on the image divided by the area of ​​the union of their 3D bounding boxes, thus obtaining the IOU matching result.

[0059] In some examples, the Hungarian algorithm, also known as the Kuhn-Munkres algorithm, is an algorithm used to solve the maximum weight perfect matching problem in a bipartite graph. In this embodiment, given a bipartite graph where each vertex has a weight, the goal is to find a matching such that the sum of the weights of all matching vertices is maximized, and each vertex can be matched at most once.

[0060] The Hungarian algorithm continuously increases the number of matchings by finding augmenting paths until no more augmenting paths can be found. The algorithm initializes each vertex in the bipartite graph as unmatched. In each iteration, starting from an unmatched vertex, it finds another vertex that can match it. If that vertex is not matched, the two vertices are matched directly; otherwise, it searches for the next unmatched vertex after the matched vertex, continuing until an unmatched vertex is found. If an augmenting path can be found—that is, starting from an unmatched vertex, traversing several matched edges to reach an unmatched vertex—then all matched edges on this augmenting path are flipped, thus adding a matching. This process is repeated until no more augmenting paths can be found, resulting in the maximum matching of the bipartite graph.

[0061] The core of the algorithm is augmenting path search, specifically a maximum matching algorithm for a bipartite graph that utilizes augmenting paths. Assume G = (V, E) is an undirected graph. An undirected graph contains a set of vertices and edges, where V represents the set of vertices and E represents each edge connecting a vertex. The vertex set V can be divided into two disjoint subsets X and Y. (See [reference needed]). Figure 4 The two vertices associated with each edge in the graph belong to two disjoint subsets, and the vertices in these two subsets are not adjacent.

[0062] The necessary and sufficient condition for determining whether a graph is bipartite is that all cycles in G have even lengths and at least two vertices. (See also: [link to relevant documentation]). Figure 5 The process involves coloring vertices to determine their suitability. Uncolored vertices are colored, and their neighboring vertices are compared. If a vertex is uncolored, it is colored with a different color than its neighbors. If a vertex is colored and its color matches that of its neighbors, it is not a bipartite graph. If the colors are different, BFS and DFS can be used for further analysis.

[0063] See also Figure 6 In a subgraph M of a bipartite graph G, M is called a matching if no two edges in the edge set E of M are connected to the same vertex. Vertices associated with the edge set of the matching M are saturated vertices; otherwise, they are unsaturated vertices.

[0064] See also Figure 7 An alternating path represents a path in graph G, where edges in the path alternate between belonging to M and not belonging to M.

[0065] See also Figure 8 An augmenting path is an alternating path with an odd number of edges, where neither the first nor the last edge belongs to M. Reversing the matching patterns of the edges in the path results in a larger matching M'.

[0066] In this way, the criterion of the Hungarian algorithm, "minimum loss," can be achieved. The loss can be represented in the form of a loss matrix to describe the cost of matching two elements in two sets.

[0067] In one optional implementation, the step of detecting traffic anomalies based on the fused data to obtain traffic anomaly detection results includes: Based on the fused data, corresponding feature variables are constructed; Based on the aforementioned feature variables, the abnormal event detector is invoked to detect abnormal traffic events, and the results of the abnormal traffic event detection are obtained.

[0068] In some examples, the common input features for each type of traffic anomaly event can be shown in Table 1.

[0069] Table 1

[0070] In addition, considering the definitions and characteristics of different event types, feature engineering can be established to construct feature variables specific to each event type, as shown in Table 2.

[0071] Table 2

[0072] In some examples, an anomaly detector can be constructed to perform anomaly detection.

[0073] In some examples, lane changing can be used as an example to illustrate abnormal event detection. When a vehicle moves from one lane to another in an area where lane changing is not permitted, this is considered an illegal lane change event. The corresponding algorithm design can be as follows.

[0074] 1) For motor vehicle targets, the lane information and trajectory information contain the current target; 2) Statistical analysis of "lane number - target number" data within 20 consecutive historical frames; 3) Target tracking time exceeds 1 second; 4) Calculate the median lateral displacement of the target using historical target trajectory data within 2 seconds; 5) The difference between the current target's lateral distance and the median target lateral distance is greater than 1.3 (event definition); 6) The number of lanes in which the current target has been located is greater than 1 (event definition); 7) The event confidence level is greater than 0.5; 8) Within 5 consecutive seconds, the target's position confidence is greater than 0.1, velocity confidence is greater than 0.1, velocity standard deviation is less than 3, length standard deviation is less than 3, and average displacement is less than 2, and the proportion of predicted frames is less than 0.3 within 2 consecutive seconds; 9) If the target satisfies the conditions for 5 or more consecutive frames, report the lane change event.

[0075] In an optional implementation, the method further includes: Obtain evidence information for abnormal events corresponding to the traffic abnormality detection results.

[0076] In some examples, see Figure 9 By statistically analyzing the characteristic variables of the traffic anomaly detection results before, during, and after the event, the corresponding timestamps can be obtained and sent to the main control or central control platform. This allows the platform to obtain the camera's capture time and record photos before, during, and after the event, thus completing the closed loop of evidence collection.

[0077] Taking lane change as an example, in addition to the characteristic variables common to all event types (such as speed standard deviation, length standard deviation, forward displacement, prediction ratio, etc.), lane change-specific variables can also be constructed, such as target lateral displacement and / or lane number set.

[0078] Lateral displacement refers to the lateral distance between the target and the centerline of its lane. Calculating this lateral displacement determines whether the target has left the lane centerline. If the lateral distance exceeds a certain threshold, the target is considered to have changed lanes. The lane number set is constructed by creating a dictionary of target lane numbers, using the target ID as the key and the lane number containing that ID as the value. By recording 20 consecutive frames of key-value pairs, it's determined whether the target's lane has changed. If the number of lane numbers containing the target ID is only 1, the target is considered not to have changed lanes; conversely, if the number of lane numbers containing the target ID is greater than 1, the target is considered to have changed lanes.

[0079] In some examples, the determination of whether a target has changed lanes can be made using two variables: lateral displacement and lane number set. For example, a target ID is considered to have reliably changed lanes only if it simultaneously meets the policy conditions constructed by the two variables.

[0080] Secondly, correspondingly, the embodiments of this application also provide a traffic anomaly event detection device, which can implement all the processes of the traffic anomaly event detection method provided in the above embodiments.

[0081] See Figure 10 The diagram shows a schematic of the traffic anomaly detection device 1000 provided in an embodiment of this application. The traffic anomaly detection device 1000 includes: The multimodal data acquisition module 1001 is used to acquire multimodal data corresponding to the target area, wherein the multimodal data includes point cloud data and video data; Alignment module 1002 is used to align the multimodal data based on alignment parameters; The target detection module 1003 is used to perform target detection based on the aligned multimodal data to obtain point cloud detection results and video detection results; The fusion module 1004 is used to fuse the aligned multimodal data based on the point cloud detection results and the video detection results to obtain fused data; The event detection module 1005 is used to detect traffic anomalies based on the fused data and obtain the traffic anomaly detection results.

[0082] In one optional implementation, the alignment parameters include time parameters, calibration parameters, and / or a reference point corresponding to the target region; The calibration parameters are obtained through the following methods: The video data in the multimodal data corresponding to the target area is mapped to the coordinate system of the point cloud data in the multimodal data corresponding to the target area to obtain video mapping data; The calibration parameters are determined using the least squares method based on the video mapping data and the point cloud data in the multimodal data corresponding to the target region.

[0083] In one optional implementation, the step of fusing the aligned multimodal data based on the point cloud detection results and the video detection results to obtain fused data includes: The point cloud detection results are mapped to the image coordinate system corresponding to the video detection results to obtain the point cloud detection mapping results; Based on the point cloud detection mapping results and the video detection results, the aligned multimodal data are fused to obtain fused data.

[0084] In one optional implementation, the step of fusing the aligned multimodal data based on the point cloud detection mapping result and the video detection result to obtain fused data includes: Based on the point cloud detection mapping results and the video detection results, determine the Intersection over Union (IOU) matrix set; Based on the IOU matrix set, the Hungarian algorithm is used to determine the globally optimal matching result; Based on the global optimal matching result and the preset IOU threshold, the aligned multimodal data is fused to obtain fused data.

[0085] In one optional implementation, the step of detecting traffic anomalies based on the fused data to obtain traffic anomaly detection results includes: Based on the fused data, corresponding feature variables are constructed; Based on the aforementioned feature variables, the abnormal event detector is invoked to detect abnormal traffic events, and the results of the abnormal traffic event detection are obtained.

[0086] In one alternative embodiment, the device further includes: The evidence collection module is used to obtain evidence information of abnormal events corresponding to the traffic abnormal event detection results.

[0087] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0088] Fourthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above-described embodiments.

[0089] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0090] See Figure 11 The computer device in this embodiment includes a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101, such as a traffic anomaly detection program. When the processor 1101 executes the computer program, it implements the steps in the various traffic anomaly detection method embodiments described above, for example... Figure 1 The steps S101-S105 are shown.

[0091] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 1102 and executed by the processor 1101 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0092] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 1101 and a memory 1102. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0093] The processor 1101 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 1101 can be any conventional processor. The processor 1101 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0094] The memory 1102 can be used to store the computer programs and / or modules. The processor 1101 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 1102 and calling the data stored in the memory 1102. The memory 1102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0095] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a non-transitory computer-readable storage medium. When the computer program is executed by the processor 1101, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0096] In summary, the embodiments of this application have at least the following beneficial effects: By employing the embodiments of this application, multimodal data corresponding to a target area is acquired, wherein the multimodal data includes point cloud data and video data; the multimodal data is aligned based on alignment parameters; target detection is performed based on the aligned multimodal data to obtain point cloud detection results and video detection results; the aligned multimodal data is fused based on the point cloud detection results and the video detection results to obtain fused data; and traffic anomaly detection is performed based on the fused data to obtain traffic anomaly detection results. In this way, the probability of low accuracy in traffic anomaly detection due to unclear camera footage can be reduced by combining multimodal data, and the amount of information used for event detection can be enriched even when the camera footage is sufficiently clear, thereby improving the accuracy of traffic anomaly detection. Furthermore, precise data fusion can be achieved through alignment and target detection, improving the reliability of fusion between different modal data, thereby further improving the accuracy of traffic anomaly detection.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0098] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for detecting abnormal traffic events, characterized in that, include: Acquire multimodal data corresponding to the target area, wherein the multimodal data includes point cloud data and video data; The multimodal data is aligned based on the alignment parameters; Target detection is performed based on the aligned multimodal data to obtain point cloud detection results and video detection results; Based on the point cloud detection results and the video detection results, the aligned multimodal data are fused to obtain fused data; Traffic anomaly event detection is performed based on the fused data to obtain traffic anomaly event detection results.

2. The method according to claim 1, characterized in that, The alignment parameters include time parameters, calibration parameters, and / or reference points corresponding to the target region; The calibration parameters are obtained through the following methods: The video data in the multimodal data corresponding to the target area is mapped to the coordinate system of the point cloud data in the multimodal data corresponding to the target area to obtain video mapping data; The calibration parameters are determined using the least squares method based on the video mapping data and the point cloud data in the multimodal data corresponding to the target region.

3. The method according to claim 1, characterized in that, The step involves fusing the aligned multimodal data based on the point cloud detection results and the video detection results to obtain fused data, including: The point cloud detection results are mapped to the image coordinate system corresponding to the video detection results to obtain the point cloud detection mapping results; Based on the point cloud detection mapping results and the video detection results, the aligned multimodal data are fused to obtain fused data.

4. The method according to claim 3, characterized in that, The step of fusing the aligned multimodal data based on the point cloud detection mapping result and the video detection result to obtain fused data includes: Based on the point cloud detection mapping results and the video detection results, determine the Intersection over Union (IOU) matrix set; Based on the IOU matrix set, the Hungarian algorithm is used to determine the globally optimal matching result; Based on the global optimal matching result and the preset IOU threshold, the aligned multimodal data is fused to obtain fused data.

5. The method according to claim 1, characterized in that, The traffic anomaly event detection based on the fused data, to obtain traffic anomaly event detection results, includes: Based on the fused data, corresponding feature variables are constructed; Based on the aforementioned feature variables, the abnormal event detector is invoked to detect abnormal traffic events, and the results of the abnormal traffic event detection are obtained.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Obtain evidence information for abnormal events corresponding to the traffic abnormality detection results.

7. A traffic anomaly detection device, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal data corresponding to a target area, wherein the multimodal data includes point cloud data and video data; An alignment module is used to align the multimodal data based on alignment parameters; The target detection module is used to perform target detection based on the aligned multimodal data, and obtain point cloud detection results and video detection results; The fusion module is used to fuse the aligned multimodal data based on the point cloud detection results and the video detection results to obtain fused data; The event detection module is used to detect abnormal traffic events based on the fused data and obtain the abnormal traffic event detection results.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-6.

10. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-6.

Images