Intelligent network connected vehicle perception anomaly event oriented evidence generation method and system

Abstract

The application provides a kind of evidence generation method and system for intelligent network connection car perception abnormal event. The specific method comprises: obtaining driving video, extracting video image information frame by frame;Obtain the time sequence perception data list, calculate the sampling frequency of time sequence perception data;Anchoring node data, aligning video frame sequence and time sequence perception data;Calibration camera-vehicle rigid body transformation matrix, establish the virtual imaging model of driving camera;Visualize the labeling of perception object structure data and generate driving video labeling file that meets the evidence attribute of perception abnormal event. The evidence generation method and system for intelligent network connection car perception abnormal event of the application effectively solve the problems of low efficiency, poor accuracy and insufficient evidence effectiveness in existing intelligent network connection car event investigation, trial and other judicial activities due to lack of technical methods, while improving efficiency and ensuring accuracy, realizing easy identification and traceability of perception abnormal event generated evidence.

Description

Technical Field

[0001] This application mainly relates to the fields of road traffic accident investigation and forensic identification, and in particular to an evidence generation method and system for intelligent connected vehicles to perceive abnormal events. Background Technology

[0002] In recent years, the penetration rate of intelligent connected vehicles, represented by assisted driving vehicles, has been continuously increasing. Vehicles with L2+ assisted driving functions already possess autonomous driving capabilities in specific scenarios. On the other hand, unmanned delivery vehicles and unmanned taxis with L4 autonomous driving functions have also undergone a certain scale of real-vehicle road testing. The coupling between intelligent connected vehicles and the traffic environment is becoming increasingly close, but this has also led to frequent accidents involving intelligent connected vehicles, especially traffic accidents involving the activation of automated driving functions in intelligent connected vehicles, which have become a focus of social attention. Accurately conducting investigations and assessments of intelligent connected vehicle accidents is a prerequisite for accident liability determination, hazard tracing, and functional improvement.

[0003] Unlike traditional vehicles, intelligent connected vehicles possess features such as environmental perception, autonomous decision-making, and automatic control. As the input to the control and execution modules of intelligent connected vehicles, the operational status of the environmental perception module is particularly crucial. According to domestic and international statistics on intelligent connected vehicle accidents, abnormal perception is one of the significant causes of such accidents. How to conduct investigations, analyses, and evidence generation for abnormal perception events in intelligent connected vehicles is a common challenge faced by public safety, judicial administration, and market regulation sectors.

[0004] The technical approaches used for environmental perception in intelligent connected vehicles can be mainly divided into two schemes: pure vision and multi-sensor fusion. These different approaches differ significantly in hardware deployment and software algorithms. Furthermore, the sheer volume and high sampling frequency of accident data in intelligent connected vehicles make it difficult to develop a universal, accurate, efficient, and intuitive data processing solution. This is particularly challenging in the investigation, analysis, and evidence generation for abnormal perception events in intelligent connected vehicles. Specifically, in the industry, handling abnormal perception events in intelligent connected vehicles often involves specialized technicians meticulously analyzing each item of the massive amounts of perception data recorded by the vehicle. This process is not only time-consuming and labor-intensive but also prone to overlooking crucial perception data. Additionally, the resulting investigation reports or expert opinions are often highly specialized, significantly reducing their evidentiary value in court proceedings and resulting in a substantial waste of public resources such as judicial and administrative resources.

[0005] Therefore, the industry still lacks an evidence generation scheme for perceived abnormal events in intelligent connected vehicles. Summary of the Invention

[0006] The technical problem to be solved by this application is to provide an evidence generation method for abnormal events perceived by intelligent connected vehicles, which can realize the process flow, automation and visualization of data analysis based on different environmental perception technology routes.

[0007] To address the aforementioned technical problems, this application provides a method for generating evidence of anomalous events perceived by intelligent connected vehicles, comprising the following steps: acquiring driving videos recorded by a driving camera of the intelligent connected vehicle, extracting video image information frame by frame, wherein the video image information includes the file frame rate of the driving video, video playback time information, and video frame sequence; acquiring a time-series perception data list of the intelligent connected vehicle, and calculating the sampling frequency of the time-series perception data, wherein the time-series perception data list contains multiple rows of time-series perception data, wherein the time-series perception data includes the vehicle's self-positioning data and perceived object structure data; determining an aligned time scale based on the file frame rate of the driving video and the sampling frequency of the time-series perception data, and determining an aligned time node based on the key events corresponding to the driving video and the time-series perception data, wherein... The time scale includes the time interval between adjacent video frames in the video frame sequence, or the time interval between two adjacent rows of time-series sensing data in the time-series sensing data list; object control points are deployed within the field of view of the driving video, the pose transformation matrix between the object coordinate system and the driving camera coordinate system is solved, and a camera-vehicle rigid body transformation matrix between the driving camera coordinate system and the vehicle body coordinate system of the intelligent connected vehicle is established through the object coordinate system; and a virtual imaging model of the driving camera is established by combining the camera-vehicle rigid body transformation matrix and the time-series sensing data, and the time-series sensing data aligned with selected frames in the video frame sequence is labeled in the video frame sequence according to the virtual imaging model, thereby generating a driving video annotation file, which is used to realize the generation of evidence for the intelligent connected vehicle's perception of abnormal events.

[0008] Optionally, the key events include state change data and / or continuous operation data. The state change data includes the traffic light indication status at the intersection and the turn and brake light indication status of surrounding vehicles. The continuous operation data includes the vehicle speed change characteristics of the intelligent connected vehicle.

[0009] Optionally, the time node determined based on the state change data is the moment when the state change data occurs; the step of determining the aligned time node based on the continuous operation data includes selecting from the driving video according to the video playback time information. The vehicle speed of the intelligent connected vehicle is calculated by comparing the driving video with a sliding window of varying duration. and the vehicle speed in the time-series sensing data list The difference of squares between Time-aware data is considered to be in good condition when the following conditions are met. The time frame is aligned with the starting frame containing video frames within the sliding window period: .

[0010] Optionally, the number of object control points is not less than 4, and any 3 of the object control points are not collinear. The method further includes analyzing the pose transformation matrix of the vehicle camera coordinate system and the object coordinate system, and measuring the axial distance between the object coordinate system and the vehicle coordinate system, thereby establishing the camera-vehicle rigid body transformation matrix.

[0011] Optionally, the step of establishing the virtual imaging model of the vehicle camera further includes solving the intrinsic parameter matrix and distortion coefficient of the vehicle camera using the Zhang Zhengyou calibration method, and establishing the virtual imaging model of the vehicle camera by combining the intrinsic parameter matrix, the distortion coefficient and the camera-vehicle rigid body transformation matrix.

[0012] Optionally, when annotating the temporal sensing data in the video frame sequence with the selected frames in the video frame sequence, the method further includes performing a distortion adaptive warp annotation operation, wherein the distortion adaptive warp annotation includes fitting the geometric boundary of the sensed object with a spline curve for the nonlinear offset of the geometric boundary caused by the lens distortion of the driving camera.

[0013] Optionally, the method further includes calculating the geometric boundary of the perceived object based on the time-series perception data, and generating the driving video annotation file after reprojecting the geometric boundary using the virtual imaging model of the driving camera. The geometric boundary is a cuboid that encloses the perceived object, and the cuboid contains 8 vertices and 12 edge center points. The step of calculating the geometric boundary of the perceived object based on the time-series perception data further includes solving for the geometric center, geometric dimensions, and driving direction of the intelligent connected vehicle based on the cuboid. The geometric dimensions of the cuboid include the length, width, and height of the cuboid.

[0014] Optionally, the method further includes fitting the 8 vertices and 12 edge center points of the cuboid with spline curves to achieve adaptive distortion annotation of the 12 edges of the cuboid in the driving video, wherein the vertices of the cuboid... and the center point of the edge The calculation method is as follows: , , in, The geometric center of the sensed object in the vehicle coordinate system. Let be the geometric dimensions of the sensed object, where The length of the sensed object, The width of the perceived object, and The height of the sensed object. The angle between the direction of travel of the sensed object and the direction of travel of the intelligent connected vehicle. The vertex of the perceived object The adjacent vertices.

[0015] Optionally, if the time-series sensing data uses a global coordinate system, then the geometric center of the sensed object in the vehicle coordinate system is... The calculation method is as follows: , in: The geometric center of the perceived object in the global coordinate system. Let be the spatial position of the intelligent connected vehicle in the global coordinate system. The heading angle of the intelligent connected vehicle in the global coordinate system represents the driving direction.

[0016] Another aspect of this application proposes an evidence generation system for perceived abnormal events in intelligent connected vehicles, comprising: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the data processing method proposed in any embodiment of this application.

[0017] Another aspect of this application proposes a computer-readable medium storing computer program code that, when executed by a processor, implements the data processing method proposed in any embodiment of this application.

[0018] Compared with the prior art, this application has the following advantages: (1) It realizes the automated and process-oriented processing of massive sensing data of intelligent connected vehicles, greatly reduces the workload of accident analysts, significantly improves the efficiency of vehicle sensing data processing, and can especially realize the rapid discovery and accurate positioning of sensing abnormal events.

[0019] (2) Without changing the vehicle perception data, the original data fields can be visualized and labeled on the driving video frame by spatial transformation and projection transformation. This combines the objectivity and readability of the original data fields as evidence. When applied to fields such as judicial administration, it can effectively solve the problem of the original data fields being highly specialized and lacking evidentiary effect in judicial administration.

[0020] (3) It realizes the time alignment of driving video data and vehicle perception data, which to a certain extent solves the problems of complex recorded data and weak correlation between evidence caused by differences in recording mechanisms in intelligent connected vehicle accidents. When applied to fields such as judicial administration, it can provide a solid foundation for subsequent work such as evidence chaining, evidence archiving, and evidence retention. Attached Figure Description

[0021] The accompanying drawings are included to provide a further understanding of this application; they are incorporated into and constitute a part of this application. The drawings illustrate embodiments of this application and, together with this specification, serve to explain the principles of this application. In the drawings: Figure 1 This is a flowchart illustrating an evidence generation method for perceived abnormal events in intelligent connected vehicles according to an embodiment of this application. Figure 2 This is a schematic diagram of a frame from a driving video in an evidence generation method for perceiving abnormal events in intelligent connected vehicles according to an embodiment of this application. Figure 3 This is a schematic diagram of a time-series sensing data list, which includes time-series sensing data, according to an embodiment of the present application of an evidence generation method for sensing anomalies in intelligent connected vehicles. Figure 4 This is a schematic diagram of camera-vehicle rigid body transformation in an evidence generation method for perceptual anomalies in intelligent connected vehicles according to an embodiment of this application. Figure 5 This is a schematic diagram of the geometric boundary of a perceived object in an evidence generation method for perceived abnormal events in intelligent connected vehicles according to an embodiment of this application. Figure 6 This is a schematic diagram of a frame from a driving video annotation file in an evidence generation method for perceptual anomalies in intelligent connected vehicles according to an embodiment of this application. Figure 7 This is a system block diagram of an evidence generation system for perceiving abnormal events in intelligent connected vehicles, according to an embodiment of this application. Detailed Implementation

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this application. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0023] An embodiment of this application is referred to. Figure 1 A method for generating evidence of anomalous events perceived by intelligent connected vehicles is proposed

[10] . This method can streamline, automate, and visualize the data analysis process, based on different perception technology approaches applicable to different environments. For example... Figure 1 As shown, the evidence generation method 10 for perceiving abnormal events in intelligent connected vehicles according to this application includes the following steps.

[0024] Step 11 involves acquiring driving video footage captured by the intelligent connected vehicle's camera, extracting video image information frame by frame. This video image information includes the file frame rate of the driving video, video playback time information, and the video frame sequence. Specifically, step 11 includes acquiring the file frame rate of the driving video. Playback Time Sequence (PTS) information, where PTS is the relative time of any frame in the driving video compared to the first frame of the driving video. For example, for intelligent connected vehicles, the driving video involved in step 11 includes a portion of the raw data read from the vehicle / cloud. On the other hand, the file frame rate of the driving video... The PTS information of the frame sequence was detected using the ffprobe command-line tool, and the command-line instructions are as follows: ffprobe -i filenames; ffprobe -i filenames -show_frames, Here, `filenames` represents the filenames of the driving video files. For example, Figure 2 The diagram shows a frame from the driving video in step 11 above, as shown below. Figure 2 As shown, the frame rate of the driving video is 25fps, and the time interval between adjacent video frames in the video frame sequence is 0.04s.

[0025] Further reference Figure 1 As shown, step 12 involves obtaining a time-series perception data list for the intelligent connected vehicle and calculating the sampling frequency of the time-series perception data. The time-series perception data list contains multiple rows of time-series perception data. This data includes the vehicle's localization data and the structure data of the perceived objects. The structure data includes the object's localization data and its state data. The vehicle's localization data and the object's localization data are divided into global coordinates and local coordinates. The sampling frequency of the time-series perception data list can be obtained through the Unix timestamps in the list. Specifically, step 12 can convert the perception data based on Unix time to, for example, Beijing time, to obtain the sampling frequency of the perception data. For example, this time-series perception data list can be obtained from the vehicle or the cloud. Specifically, Unix time has a precision of milliseconds and needs to be converted to Coordinated Universal Time (UTC) first. Adding 8 hours to UTC gives Beijing time.

[0026] For example, converting Unix time to UTC time utilizes the C++ `std::chrono` time library. The `std::gmtime` function converts the Unix time to a `tm` structure, and the `std::put_time` function formats the `tm` structure into UTC time (yyyy-mm-dd hh:mm:ss.000). Figure 3 In the time-aware data list shown, the first row of collected Unix time is 1734141573.62984, which is converted to Beijing time as 2024-12-14 09:59:33.630.

[0027] Furthermore, the sampling frequency of time-aware data in this embodiment can be understood as the number of rows in the time-aware data list per unit time. Assuming... The time-aware data list within seconds contains a total of Then the sampling frequency of the time-series sensing data It can be represented as: , For example, Figure 3 In the time-series sensing data list shown, there are 10 data items within the time frame of 09:59:34 on 2024-12-14. Therefore, the sampling frequency of the sensing data is 10Hz.

[0028] Further reference Figure 1 As shown, step 13 involves determining the alignment time scale based on the file frame rate of the driving video and the sampling frequency of the time-series sensing data, and determining the alignment time node based on the key events corresponding to the driving video and the time-series sensing data. The time scale includes the time interval between adjacent video frames in the video frame sequence, or the time interval between two adjacent rows of sensing data in the time-series sensing data list. Specifically, this step achieves "fixed-point" alignment by anchoring node data and "fixed-length" alignment by comparing sampling frequencies, ultimately completing the alignment of the video frame sequence with the sensing data in absolute time. Step 13 uses the lower of the sampling frequency of the driving video and the sampling frequency of the time-series sensing data list as a benchmark to align node data from the other data source, thus establishing the basis for subsequent calculations and analyses.

[0029] Specifically, "fixed point" refers to anchoring the same moment using state change data or continuous operation data. State change data refers to moments of significant change in environmental state during driving, such as color changes of traffic lights at intersections, and the status of turn signals and brake lights of surrounding vehicles. Continuous operation data refers to the speed changes of the vehicle and surrounding vehicles. For example, the moment when the vehicle's speed begins to decrease can be used as a basis for time node alignment.

[0030] For example, when selecting a duration of The driving video clip is used to calculate the vehicle speed change within the clip based on the video images. The functional relationship, comparison Vehicle speed in time-series sensing data within a sliding window of duration The vehicle speed shown in the video image of the driving video. Time-aware data is considered valid when the following conditions are met. The time frame is aligned with the starting frame containing video frames within the sliding window period: , In the above formula, Pick The set of all timestamps of time-series-aware data within a duration window. Among them, it is considered... The moment with the smallest squared difference within the sliding window of duration is the alignment moment, which means that the alignment moment between the time-series sensing data within the sliding window and the starting frame of the driving video segment is taken as the alignment time node.

[0031] For example, take Figure 2 In the driving video shown, [9, 14] represents the duration of the sliding window (for example, if the collision time of the car is the 14th second of the acquired driving video, this time period [9, 14] corresponds to the time period 5 seconds before the collision). By comparing the speed calculated from the driving video with the vehicle speed in the time-series perception data, it is determined that the video playback time of the driving video is the video frame at 9.00 seconds, which corresponds to the data row in the time-series perception data with the timestamp 1734141570.12985.

[0032] On the other hand, "fixed length" refers to comparing the frame rate of the driving video. sampling frequency of time-series-aware data The time scale for alignment is determined to achieve matching and alignment of video frames from the driving video with the time-series perceived data list on the absolute time axis. The rules for determining the time scale are as follows: , For example, as described above, Figure 2 The driving video shown has a frame rate of 25fps, meaning the time interval between adjacent video frames is 0.04s. Figure 3 The sampling frequency of the time-aware data table shown is 10Hz. Therefore, according to the above rules for determining the time scale, the time scale for this data processing procedure is selected. (i.e., corresponding) (The situation).

[0033] The "fixed-length" processing method described above determines the aligned time scale based on the file frame rate of the driving video and the sampling frequency of the time-series sensing data. Since the data processing method 10 of this application relies on two data sources—bicycle video and time-series sensing data—during computation, these two data sources are often obtained through different sensors. Generally, because the parameter information error of the time-series sensing data list is very small and its accuracy is higher than that of the driving video, it can be considered to contain absolute time. In reality, some driving videos do not contain watermarks with time information, and even if they do, there may be an error of several seconds compared to the actual situation. Therefore, this application adopts the above-mentioned "fixed-length" processing method to efficiently unify the two data sources. It should be noted that this application does not limit the processing order of "fixed-point" and "fixed-length". A preferred approach is to align the time scale first, especially when the file frame rate of the driving video and the sampling frequency of the time-series sensing data are different, this method saves more computational power.

[0034] Further reference Figure 1 As shown, step 14 involves setting up object control points within the field of view of the driving video, solving for the pose transformation matrix between the object coordinate system and the driving camera coordinate system, and establishing the camera-vehicle rigid body transformation matrix between the driving camera coordinate system and the vehicle body coordinate system of the intelligent connected vehicle through the object coordinate system. Specifically, step 14 involves setting up object control points within the field of view of the driving video, preferably using the perspective N-point problem algorithm to estimate the pose of the driving camera in the object coordinate system, and then measuring the axial distance between the vehicle body coordinate system and the object coordinate system. Using the object coordinate system as an intermediary, the camera-vehicle rigid body transformation matrix is ​​then solved.

[0035] Specifically, the data input to the perspective N-point problem algorithm includes 3D-2D point pairs of object-side control points and the intrinsic parameter matrix and distortion coefficients of the vehicle camera. In this embodiment, the Zhang Zhengyou calibration method is preferably used to calibrate the camera's intrinsic parameter matrix and distortion coefficients. On the other hand, the pose information of the camera coordinate system can be calibrated using the SQPnP pose estimation algorithm, for example. The data output of the perspective N-point problem algorithm is the translation vector and rotation vector of the vehicle camera in the object-side coordinate system. The rotation vector, after being transformed by Rodrigues, can be used to resolve the rotation matrix between the monitoring camera coordinate system and the object-side coordinate system.

[0036] For example, in some embodiments of this application, the perspective N-point problem algorithm can use the SolvePnP function in OpenCV, and select the SOLVEPNP_SQPNP solution mode of the SolvePnP function. The Rodrigues transform uses the Rodrigues function in OpenCV. More specifically, the 3D-2D points of the object control points are the spatial positions of the object control points in the object coordinate system and the pixel positions in the image coordinate system, respectively. In this embodiment, it is preferred to set the number of object control points to at least 4, and any 3 object control points are not collinear, that is, the object control points are not coplanar.

[0037] Preferred, refer to Figure 4 The object coordinate system used is a left-handed coordinate system, with the center of the left front wheel's contact point as the origin, and the line connecting the centers of the left and right front wheels' contact points as the coordinate system. The positive direction of the axis is parallel to the longitudinal axis of the vehicle and the forward direction. The positive direction of the axis, with the direction perpendicular to the ground upward as... Positive direction of the axis.

[0038] Furthermore, the intrinsic parameter matrix and distortion coefficients of the vehicle camera were determined using the Matlab camera calibration toolbox. The intrinsic parameter matrix included focal length and image center, and the distortion coefficients included first-order radial distortion coefficients. and second-order radial distortion coefficient The intrinsic parameter matrix K can be expressed as: , In the above formula: In order to be in Equivalent focal length in direction, In order to be in Equivalent focal length in direction, In order to be in Imaging center of direction, In order to be in The imaging center in the direction.

[0039] For example, for taking pictures Figure 2 The intrinsic parameter matrix and distortion coefficients of the driving camera for the mid-video image frame are as follows: ,

[0040] Translation vector and rotation vector All are 1×3 in size. The rotation vector is described by the rotation axis and the rotation angle, where the direction of the rotation vector is the same as the rotation axis, and the length of the rotation vector is equal to the magnitude of the rotation angle. Rotation matrix The size is 3×3.

[0041] Furthermore, according to Figure 4 Within the field of view of the driving video, at least four object control points shall be deployed, among which... Figure 4 Five examples, cp1 to cp5, are shown. Based on these, the pose transformation matrix between the object coordinate system and the vehicle camera coordinate system is solved. Using the object coordinate system as a link, the camera-vehicle rigid body transformation matrix between the vehicle camera coordinate system and the vehicle body coordinate system of the intelligent connected vehicle is established. Specifically, the vehicle body coordinate system generally uses the center of the front or rear axle as the origin. The axial distance between the vehicle body coordinate system and the object coordinate system is... Lateral distance along the axis , longitudinal distance along the axis and Axial elevation distance The camera-vehicle rigid body transformation matrix includes the translation matrix. and rotation matrix , can be represented as: , , In the formula: This represents the coordinate system from the camera coordinate system to the vehicle coordinate system. and These represent the translation and rotation matrices in the camera-vehicle rigid body transformation, respectively.

[0042] For example, the translation vector calculated after pose estimation and Rodriguez transformation. and rotation matrix (Camera-object coordinate rigid body transformation) are as follows: , , For example, taking the center of the vehicle's front axle as the origin of the vehicle's coordinate system, the lateral distance is... Longitudinal distance and elevation distance They can be represented as: , In the above formula: The front wheel track. This refers to the diameter of the front tires. According to the specifications in the announcement for intelligent connected vehicles, , .

[0043] Further reference Figure 1As shown, step 15 involves establishing a virtual imaging model of the driving camera by combining the camera-vehicle rigid body transformation matrix and temporal sensing data. Based on the virtual imaging model, temporal sensing data aligned with selected frames in the video frame sequence is annotated, thereby generating a driving video annotation file. This driving video annotation file is used to generate evidence of abnormal events perceived by intelligent connected vehicles. Step 15 is essentially a visual annotation of the structured data of the perceived object. Specifically, step 15 converts the positional information in the structured data of the perceived object into geometric boundaries. Combining the camera-vehicle rigid body transformation matrix solved in step 14 with the intrinsic parameter matrix and distortion coefficients of the driving camera, a virtual imaging model of the driving camera is constructed, thereby achieving the visual annotation of the structured data of the perceived object in the driving video. For this application, the perceived object is an object that may cause an accident, such as road environment information, including other motor vehicles, non-motor vehicles, pedestrians, obstacles, etc. Specifically, the structured data of the perceived object includes information such as object type, geometric center, geometric dimensions, driving direction, and driving speed. For example, refer to... Figure 5 As shown, Figure 5 An example of a perceived object being a cuboid (such as a car) is shown. For a cuboid, its geometric boundary can be represented by the 8 vertices and 12 edge centers of the cuboid.

[0044] For example, refer to Figure 5 and combined Figure 4 As shown, if the time-series sensing data uses the vehicle coordinate system, it is based on the geometric center of the sensed object. Geometric dimensions and the angle between its driving direction and the driving direction of the intelligent connected vehicle ,in, l To perceive the length of an object, w To perceive the width of an object, and h To perceive the height of an object, one must perceive the vertices of the object's geometric boundaries. and the center point of the edge They can be expressed as the following formulas, where The vertex of the perceived object Adjacent vertices: , , If the time-series sensing data uses a global coordinate system, then the geometric center of the sensed object in the vehicle coordinate system is... It can be represented as: , In the above formula: To perceive the geometric center of an object in the global coordinate system, This refers to the spatial position of the intelligent connected vehicle in the global coordinate system. This represents the heading angle of the intelligent connected vehicle in the global coordinate system. It should be noted that, in order to achieve... Figure 6 The annotation effect shown illustrates that if the time-series perception data uses a global coordinate system, the coordinates of the perceived objects in the global coordinate system must be converted to coordinates in the vehicle coordinate system (i.e., the local coordinate system). Depending on the configuration of the intelligent connected vehicle, if its time-series perception data itself belongs to the vehicle coordinate system, the process of converting from the global coordinate system to the local coordinate system can be omitted.

[0045] For example, refer to Figure 2 and Figure 6 As shown, Figure 2 This is a diagram of one frame from the original driving video. Figure 6 To perceive the annotation results of object structure data in video frames, the temporal perception data in this embodiment adopts a global coordinate system, where Vehicle_position represents the coordinates of the intelligent connected vehicle, and Obj_0 represents... Figure 6 The electric bicycle in the middle left front, according to calculation If the angle is approximately -61°, then the geometric center of the perceived object in the vehicle coordinate system is: , Similarly, the angle between the direction of travel of the perceived object and the direction of travel of the intelligent connected vehicle. The angle is calculated to be approximately 78°, which is the vertex of the perceived geometric boundary of the object. and the center point of the edge They can be calculated separately.

[0046] For example, taking an electric bicycle as an example, its length, width, and height dimensions are 1.9 m × 0.5 m × 1.02 m. Then, the right front vertex of its geometric boundary can be represented as: , Specifically, the virtual imaging model of the vehicle camera can be represented as: , , In the above formula: In the vehicle coordinate system The projected coordinates of a point that satisfy the laws of perspective imaging. To account for the projected coordinates under lens distortion, This represents the pixel distance from the feature pixel to the imaging center. For example, by inputting the right front vertex of the electric bicycle's geometric boundary into the virtual imaging model of the driving camera, the pixel coordinates of the right front vertex in the video frame of the driving video can be obtained. .

[0047] Specifically, visual annotation involves reprojecting the structured data of the perceived object into the virtual imaging model of the driving camera to determine the geometric boundaries of the perceived object. In this embodiment, considering the distortion problem of driving cameras, especially wide-angle cameras, taking a cuboid as an example, spline curve fitting is preferably used to fit the edges of the rectangular frame, thereby achieving nonlinear annotation of the geometric boundaries of the perceived object. More specifically, the edges of the rectangular frame consist of two vertices and a center point. Through spline curve fitting, the nonlinear annotations of the 12 edges of the rectangular frame in the driving video can be obtained. The aforementioned distortion adaptive warp annotation of the perceived object's structural data includes the definition of the perceived object's boundaries and the annotation of the perceived object's state information. The definition of the perceived object's boundaries is based on the data dimension, using two-dimensional planar definition or three-dimensional solid definition. The perceived object's state information includes the type of the perceived object, its driving speed, and the color of traffic lights, etc.

[0048] It should be noted that for planar objects, spline curve fitting is not necessarily performed using the same methods as for cubic objects. Additionally, Figure 6 The utility poles on the left and right sides are bent. If they are directly marked according to the above data processing flow, the shape of the bent utility poles cannot be accurately represented, affecting the data annotation effect. However, using spline curve fitting can improve the visual appeal of the markings. In this application, the perceived objects are not limited to vehicles, pedestrians, and fixed objects, but also include traffic lights, road speed limit signs, etc. Step 15 only uses vehicle-type perceived objects as an example for illustration. The visualization annotation of other types of perceived objects can be performed in the same way.

[0049] Based on this, step 15 above finally generates the driving video annotation file. Specifically, this operation will... Figure 2 The system converts a list of time-series sensing data, recorded as a data stream, into a driving video annotation file recorded as a video stream. This unifies the scale of heterogeneous time-series sensing data to the video frame, enabling intuitive comparison and verification between the sensing data and the actual vehicle situation. When applied in judicial and administrative scenarios, it can significantly enhance the evidentiary value in judicial activities such as traffic accident investigations and trials. Specifically, with... Figure 2 For example, suppose in a vehicle accident, the intelligent connected car recording the driving video fails to see the electric bicycle about to collide with it because the truck in front obstructs its view. In existing methods, it would be necessary to... Figure 3 Data analysis is performed, but the data format is quite complex. In judicial and administrative practice, it is directly adopted... Figure 3 The data format makes it difficult to present evidence. And... Figure 3 Equivalent use of data information Figure 2 In the process, and after going through the "fixed-point" and "fixed-length" methods of this application, as well as a series of subsequent data processing procedures, the final result is obtained. Figure 6 The image format in which Figure 6 The red box in the image indicates the object being labeled. Compared to... Figure 3 , Figure 6 This is more conducive to the presentation of evidence. That is, it will... Figure 3 The data can be visualized for intuitive interpretation. The video frame sequence with completed distortion adaptive warp annotation is used to generate a driving video annotation file. This file contains information such as the geometric boundaries, historical trajectory, running status, and / or object type of the perceived object, thus generating evidence of perceived abnormal events. For example, the driving video annotation file may contain not only the geometric boundaries of the perceived object but also its historical trajectory (e.g., in...). Figure 6 Based on this, the subsequent video frames are annotated to indicate the historical trajectory of the marker points in consecutive frames, the operating status (such as vehicle speed for other vehicles), and / or the object type (electric vehicle, car, bus, guardrail, etc.), thereby further improving the generation of evidence for perceiving abnormal events.

[0050] Specifically, the encoding parameters and container format of the driving video annotation file and the driving video are consistent. The encoding parameters include the video frame rate, video resolution, and encoding format, while the container format is the file type of the driving video. In other words, the driving video annotation file can be generated from the video frame sequence that has been annotated (or has distortion adaptive distortion annotation) by referring to the file attribute information of the driving video. Maintaining the same attribute information between the driving video annotation file and the driving video facilitates cross-referencing and processing. For example, Figure 2 This is a single frame from the driving video (original video). Figure 6 By labeling driving video files with the same attribute information, it becomes easier to compare them and perform subsequent archiving processes.

[0051] Through the above methods, driving data from intelligent connected vehicles can be automatically processed and labeled, such as image and information extraction. When applied in the field of accident investigation, evidence can be quickly generated using these methods. The advantage of this application compared to existing technologies is that it can convert massive amounts of sensory structure data into labeled driving video data, enabling visualization of the working status of the intelligent connected vehicle's sensing modules. This effectively improves the efficiency of the investigation of such cases in traffic accidents and enhances the evidentiary value of the investigation results. In particular, this application effectively solves the problems of low efficiency, poor accuracy, and insufficient evidentiary value in existing judicial activities such as investigations and trials of intelligent connected vehicle incidents due to a lack of technical methods. While improving efficiency and ensuring accuracy, it also enables easy identification and traceability of evidence generated from abnormal sensing events.

[0052] An embodiment of this application also proposes a method such as Figure 7The illustrated evidence generation system 20 for perceived abnormal events in intelligent connected vehicles. According to... Figure 7 The data processing system 20 may include an internal communication bus 21, a processor 22, a read-only memory (ROM) 23, a random access memory (RAM) 24, and a communication port 25. When applied to a personal computer, the data processing system 20 may also include a hard disk 26.

[0053] The internal communication bus 21 enables data communication between components of the data processing system 20. The processor 22 can perform judgments and issue prompts. In some embodiments, the processor 22 may consist of one or more processors. The communication port 25 enables data communication between the data processing system 20 and external systems. In some embodiments, the data processing system 20 can send and receive information and data from a network via the communication port 25.

[0054] The data processing system 20 may also include different types of program storage units and data storage units, such as a hard disk 26, a read-only memory (ROM) 23, and a random access memory (RAM) 24, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor 22. The processor executes these instructions to implement the main part of the method. The results of the processor processing are transmitted to the user equipment through a communication port and displayed on the user interface.

[0055] In addition, this application also proposes a computer-readable medium storing computer program code, which, when executed by a processor, implements the aforementioned method for generating evidence of perceived abnormal events for intelligent connected vehicles.

[0056] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0057] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0058] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0059] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0060] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of the present application requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0061] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of scope in some embodiments of this application are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0062] Although this application has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate this application, and various equivalent changes or substitutions can be made without departing from the spirit of this application. Therefore, any changes or modifications to the above embodiments within the essential spirit of this application will fall within the scope of the claims of this application.

Claims

1. A method for generating evidence of anomalous events perceived by intelligent connected vehicles, characterized in that, Includes the following steps: The system acquires driving videos captured by a driving camera in an intelligent connected vehicle, and extracts video image information frame by frame. The video image information includes the file frame rate, video playback time information, and video frame sequence of the driving video. Obtain the time-series perception data list of the intelligent connected vehicle and calculate the sampling frequency of the time-series perception data. The time-series perception data list contains multiple rows of time-series perception data, which includes the vehicle positioning data and perceived object structure data of the intelligent connected vehicle. The alignment time scale is determined based on the file frame rate of the driving video and the sampling frequency of the time-series sensing data, and the alignment time node is determined based on the key events corresponding to the driving video and the time-series sensing data. The time scale includes the time interval between adjacent video frames in the video frame sequence or the time interval between two adjacent rows of time-series sensing data in the time-series sensing data list. Object control points are deployed within the field of view of the driving video. The pose transformation matrix between the object coordinate system and the driving camera coordinate system is solved. A camera-vehicle rigid body transformation matrix between the driving camera coordinate system and the vehicle body coordinate system of the intelligent connected vehicle is established using the object coordinate system. A virtual imaging model of the driving camera is established by combining the camera-vehicle rigid body transformation matrix and the temporal sensing data. Based on the virtual imaging model, the temporal sensing data aligned with selected frames in the video frame sequence is annotated, thereby generating a driving video annotation file. This driving video annotation file is used to generate evidence of abnormal events perceived by the intelligent connected vehicle. The method further includes calculating the geometric boundary of the perceived object based on the time-series perception data, and generating the driving video annotation file after reprojecting the geometric boundary using the virtual imaging model of the driving camera. The geometric boundary is a cuboid that encloses the perceived object, and the cuboid contains 8 vertices and 12 edge center points. The step of calculating the geometric boundary of the perceived object based on the time-series perception data further includes solving for the geometric center, geometric dimensions, and driving direction of the intelligent connected vehicle based on the cuboid. The geometric dimensions of the cuboid include the length, width, and height of the cuboid.

2. The method as described in claim 1, characterized in that, The key events include state change data and / or continuous operation data. The state change data includes the traffic light indication status at the intersection and the turn and brake light indication status of surrounding vehicles. The continuous operation data includes the vehicle speed change characteristics of the intelligent connected vehicle.

3. The method as described in claim 2, characterized in that, The time node determined based on the state change data is the moment when the state change data occurs; the step of determining the alignment time node based on the continuous operation data includes selecting from the driving video according to the video playback time information. The vehicle speed of the intelligent connected vehicle is calculated by comparing the driving video with a sliding window of varying duration. and the vehicle speed in the time-series sensing data list The difference of squares between Time-aware data is considered to be in good condition when the following conditions are met. The time frame is aligned with the starting frame containing video frames within the sliding window period: 。 4. The method as described in claim 1, characterized in that, The number of object control points is not less than 4, and any 3 of the object control points are not collinear. The method also includes analyzing the pose transformation matrix between the vehicle camera coordinate system and the object coordinate system, and measuring the axial distance between the object coordinate system and the vehicle coordinate system, thereby establishing the camera-vehicle rigid body transformation matrix.

5. The method as described in claim 1, characterized in that, The step of establishing the virtual imaging model of the vehicle camera further includes solving the intrinsic parameter matrix and distortion coefficient of the vehicle camera using the Zhang Zhengyou calibration method, and establishing the virtual imaging model of the vehicle camera by combining the intrinsic parameter matrix, the distortion coefficient and the camera-vehicle rigid body transformation matrix.

6. The method as described in claim 1, characterized in that, When annotating the temporal sensing data in the video frame sequence with the selected frames in the video frame sequence, the method further includes performing a distortion adaptive warp annotation operation, the distortion adaptive warp annotation including nonlinear offset of geometric boundaries caused by lens distortion of the driving camera, and fitting the geometric boundaries of the sensed object with spline curves.

7. The method as described in claim 1, characterized in that, It also includes fitting the eight vertices and the center points of the twelve edges of the cuboid using spline curves, thereby achieving adaptive distortion annotation of the twelve edges of the cuboid in the driving video. The vertices of the cuboid... and the center point of the edge The calculation method is as follows: ， ， in, The geometric center of the sensed object in the vehicle coordinate system. Let be the geometric dimensions of the sensed object, where l The length of the sensed object, w The width of the perceived object, and h The height of the sensed object. The angle between the direction of travel of the sensed object and the direction of travel of the intelligent connected vehicle. The vertex of the perceived object The adjacent vertices.

8. The method as described in claim 7, wherein if the time-series sensing data uses a global coordinate system, then the geometric center of the sensed object in the vehicle coordinate system is... The calculation method is as follows: ， in: The geometric center of the perceived object in the global coordinate system. Let be the spatial position of the intelligent connected vehicle in the global coordinate system. The heading angle of the intelligent connected vehicle in the global coordinate system represents the driving direction.

9. An evidence generation system for perceived abnormal events in intelligent connected vehicles, comprising: Memory is used to store instructions that can be executed by the processor; and a processor for executing the instructions to implement the method as described in any one of claims 1-8.

10. A computer-readable medium storing computer program code that, when executed by a processor, implements the method as claimed in any one of claims 1-8.

Images