Video augmentation method and system based on terrain

Abstract

The invention relates to the technical field of data recharge testing, and provides a terrain-based video augmentation method and system, and the method comprises the following steps: S1, carrying out the setting of pavement height change parameters for a map file according to the needs; s2, analyzing the modified map file, and in combination with a vehicle position file, finding out a modification point through which a vehicle can pass as a vehicle attitude change point, and finding out a video moment corresponding to the vehicle attitude change point; s3, extracting a main frame picture of the vehicle attitude change point in the video file, and transforming the main frame picture according to the modified road surface height change parameter; and S4, carrying out consistency processing on all the transformed main frame pictures, and synthesizing a new video file. According to the technical scheme, after the map file is modified, the collected video is changed synchronously, and the video in the form of the vehicle on the corresponding road is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of data re-injection testing, and in particular to a terrain-based video augmentation method and system. Background Art

[0002] Data re-injection testing is a software testing method, which is widely used in the testing of intelligent systems (such as intelligent transportation systems, intelligent security systems, etc.). It mainly re-inputs the real data collected in advance (such as video files, vehicle location files, map files) into the system, allowing the system to process these data according to the normal operation process and processing logic, so as to verify whether the system's functions and performance meet expectations.

[0003] In the development and optimization of today's intelligent systems, data injection testing has become a key link in evaluating system performance, and the test of the robustness of intelligent systems is of utmost importance. Robustness, as a core indicator to measure whether intelligent systems can operate stably and efficiently in the face of complex and unpredictable situations, requires us to conduct comprehensive and in-depth testing of the adaptability of intelligent systems under various extreme and special conditions.

[0004] The data types that the recharge test relies on are rich and varied, including video files, vehicle location files, and map files. These data are carefully collected by the vehicle during actual driving and operation with the help of various high-precision sensors and data acquisition equipment. Video files are like the "eyes" of the intelligent system, which can intuitively record the dynamic changes of the vehicle's surrounding environment; vehicle location files are like the "navigator" of the intelligent system, accurately locating the coordinates and running trajectory of the vehicle in space; map files are like the "atlas" of the intelligent system, providing comprehensive geographic information and road layout guidance for vehicle driving.

[0005] Focusing on the video perception module, we found that it showed excellent performance in conventional horizontal road driving scenarios. It can accurately and accurately perform target recognition and target tracking with extremely high accuracy and agile response speed, whether it is identifying pedestrians and vehicles on the road, or tracking the movement trajectory of specific targets. However, the actual application scenarios of intelligent systems are far from being so simple and ideal. When the vehicle encounters some special conditions, such as Figure 1 As shown in the figure, when a vehicle drives onto a curb on one side, the vehicle's body posture will tilt and become unbalanced, causing subtle but critical changes in the camera's shooting angle and field of view. Another example is when a wheel on one side falls into a shallow drainage ditch, the vehicle's driving stability is seriously challenged, the jitter and shaking of the image are aggravated, and the relative position and movement pattern of the target become more complex and unpredictable. Another example is when a vehicle drives onto a curb on one side, causing the vehicle's body posture to tilt and become unbalanced, causing subtle but critical changes in the camera's shooting angle and field of view. Figure 2As shown in the figure, when driving on an inclined road, the reflection and refraction effects of light will also be different due to the changes in the slope and curvature of the road, which poses a severe test to the accuracy of target recognition and the continuity of target tracking. Under these special conditions, whether the video perception module can still maintain excellent performance is undoubtedly a key issue that needs to be further verified.

[0006] But the reality is that the collection of video data for some special scenes faces many difficulties and challenges. Especially those extreme cases and boundary cases called corner cases, which are difficult to reproduce in themselves, often require specific environmental conditions, precise vehicle control and rare geographical scenes to cooperate with each other to be realized. And it is even more difficult to obtain videos of vehicles in tilted scenes. Because this not only requires finding a suitable tilted road site, but also requires ensuring the safety, stability and validity of the data during the collection process. These objective problems directly lead to the current robustness test being difficult to achieve all-round and no-dead-angle coverage, which makes the performance evaluation of intelligent systems in dealing with special scenarios have obvious shortcomings and deficiencies.

[0007] Existing intelligent system tests often use the technology of re-collecting data, that is, redeploying the collection equipment to obtain data such as videos, vehicle locations and map files according to the process for re-injection testing. However, its disadvantages are obvious. The collection is difficult, special scenes are not easy to reproduce, and there are many restrictions on environmental conditions. For example, special road conditions are difficult to deliberately reproduce and the data quality is affected by weather, light and season. The cost is also very high. The equipment purchase, maintenance and operation are expensive, professional operators are required, and the labor cost is high. In addition, the collection takes a long time and the time cost is high, which can easily cause the product to miss the market launch window. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a terrain-based video augmentation method and system, which can realize the modification of the map file and the synchronous modification of the collected video to generate a video of the vehicle on the corresponding road.

[0009] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0010] A terrain-based video augmentation method comprises the following steps:

[0011] S1: Setting the road height change parameters of the map file as needed;

[0012] S2: Analyze the modified map file, combine it with the vehicle position file, find out the modification points that the vehicle will pass through as vehicle posture change points, and find out the video moments corresponding to the vehicle posture change points;

[0013] S3: extracting a main frame image of the vehicle posture change point in the video file, and transforming the main frame image according to the modified road height change parameter;

[0014] S4: performing consistency processing on all the transformed main frame images and synthesizing the new video file.

[0015] Furthermore, before step S1, the method further includes reading and collecting acquisition configuration information including the diagonal field of view of the camera and the camera installation position, specifically:

[0016] Reading the diagonal field of view DFOV of a camera installed on a moving vehicle;

[0017] The installation position of the acquisition camera in the original video file is obtained, including the position information including the distance d between the center point of the camera lens and the outermost side of the right tire in the horizontal direction, the distance h between the center point of the camera lens and the ground in the vertical direction, and the vehicle wheelbase D.

[0018] Furthermore, in step S1, the road height change parameters are set for the map file as required, specifically:

[0019] The setting of the road height change parameter includes setting superelevation information and adding road obstacles. The total height change value caused by the road height change parameter is the superelevation change + the height of the road obstacle.

[0020] Obtaining the original map file including road geometry information, elevation information, and attribute information, parsing the original map file into an operational data structure, defining superelevation modification ranges and parameters, and creating a template for the road obstacle including speed bumps;

[0021] Modify the superelevation information of the map file, select the target road section and key nodes that need to modify the superelevation in the operable data structure according to preset conditions, calculate and update the superelevation values ​​of the key nodes according to the defined superelevation modification range and parameters, and then rebuild the map file;

[0022] Perform the process of adding the road obstacle, determine the location of the road obstacle according to the test scenario requirements, select a suitable template of the road obstacle, adjust the attribute parameters of the template of the road obstacle, and add the template of the road obstacle with set attributes to the operational data structure.

[0023] Further, in step S2, the modified map file is analyzed, combined with the vehicle position file, to find the modification points that the vehicle will pass through as vehicle posture change points, and to find the video moments corresponding to the vehicle posture change points, specifically:

[0024] Obtaining the modification point of the superelevation information and the location where the road obstacle is added in the operable data structure of the map file, and marking the information including coordinates, type, and attributes;

[0025] Read the vehicle position file to construct a vehicle driving trajectory, match the constructed vehicle driving trajectory with the map file, determine the road route of the vehicle, and predict the modification points passed by the vehicle during driving according to the modification points marked on the map file, where the modification points passed are the vehicle posture change points;

[0026] Determine the correspondence between the timestamp in the vehicle position file and the timestamp in the video file, establish a time synchronization model based on the correspondence, and calculate the video moment corresponding to the vehicle posture change point based on the time synchronization model.

[0027] Further, in step S3, the main frame image of the vehicle posture change point in the video file is extracted, and the main frame image is transformed according to the modified road height change parameter, specifically:

[0028] Extracting the key frame IFrame from the video file to form the main frame image, and extracting the main frame image as the vehicle posture change point according to the video moment corresponding to the calculated vehicle posture change point;

[0029] According to the set road height change parameters, the main frame image of the vehicle posture change point is transformed including the vehicle posture change in the image and the calculation of a new target area to form a new main frame image of the vehicle posture change point to replace the original main frame image before the change.

[0030] Further, according to the set road height change parameter, the main frame image of the vehicle posture change point is transformed including the vehicle posture change in the image and the calculation of the new target area, specifically:

[0031] The extracted main frame image is rotated to realize the vehicle posture change, and then the original main frame image is cut out according to the posture change perspective after rotation to obtain a new target area, so as to form a new main frame image of the vehicle posture change point.

[0032] Furthermore, the extracted main frame image is rotated to realize the change of the vehicle posture, and then the original main frame image is intercepted according to the posture change view angle after the rotation to obtain a new target area, specifically:

[0033] Vehicle posture changes in the picture:

[0034] Calculating the height difference of the left and right tires relative to the original ground according to the road height change parameters of the road where the left and right tires of the vehicle are located, and rotating the left and right tires in the order of first the right tire and then the left tire, or first the left tire and then the right tire, according to the height difference of the left and right tires, so as to achieve a change in the vehicle posture;

[0035] Compute the new target region:

[0036] The vehicle's inclination angle is calculated based on the height difference between the left and right tires, and the center point of the new target area is recalculated based on the installation position information of the camera collected in the original main frame image. A largest rectangular area with the same aspect ratio as the original video is intercepted in the original main frame image with the center point of the new target area as the center as the new target area.

[0037] Furthermore, the vehicle posture changes in the image and the new target area is calculated as follows:

[0038] Calculate the vehicle's tilt, H 左 H is the height difference between the bottom of the left tire and the original ground. 右 H is the height difference between the bottom of the right tire and the original ground. 左 and H 右 Positive means higher than the horizontal ground, negative means lower than the horizontal ground, and the value is calculated based on the road surface height change parameters of the road surface where the left and right tires are located;

[0039] A video coordinate system is constructed with the center point of the original main frame image as the origin, wherein the positive direction of the X-axis of the video coordinate system is parallel to the ground and in the direction from the right tire to the left tire, the positive direction of the Y-axis is perpendicular to the X-axis and upward, and the width of the original main frame image is A pixels and the height is B pixels;

[0040] Taking the right tire as the reference, the vehicle posture is changed according to the parameters of the set vehicle tilt degree, and the right tire is moved vertically by H 右 After that, the right tire remains unchanged and the left tire is moved vertically by H 左 ;

[0041] Calculate the vehicle's tilt angle θ as:

[0042]

[0043] The calculation needs to obtain the center point of the new target area:

[0044] a: Calculate the angle β between the center point of the camera lens in the original main frame image and the line connecting the contact point between the outer side of the original right tire and the ground:

[0045]

[0046] b: Calculate the length l of the line between the center of the camera lens and the contact point between the outer side of the original right tire and the ground in the original main frame image:

[0047]

[0048] c: Calculate the transformation coefficient γ between video and physical space:

[0049]

[0050] d: Calculate the center point of the new target area in the video coordinate system as:

[0051] x0'=(l*cos(β+θ)-d)*γ

[0052] y0'=(l*sin(β+θ)-h)*γ

[0053] If the center point of the new target area is not in the original video frame of the original main frame image, or the distance from the two sides of the original video frame is only 20% of the short side distance, the augmentation fails and the task ends;

[0054] Calculate a new target area, the new target is a rectangular area based on the center point with the same aspect ratio as the original main frame image, and take the maximum value within the range of the original main frame image, specifically:

[0055] a: Calculate the angle between the two diagonals of the original video frame and the X-axis:

[0056]

[0057] b: Calculate the angle between the two diagonals of the new target area and the X-axis:

[0058] θ1′=θ1+θ

[0059] θ2′=θ2+θ

[0060] c: Draw two diagonal lines with angles θ1' and θ2' with the X-axis through the center point of the new target area, find the intersection points a, b, c, d of the diagonal lines with the original video frame, take the shortest distance between the intersection point and the center point * 2 as the diagonal length L of the target area, and draw the new target area;

[0061] Pixels of a new target area are intercepted and collected from the original main frame image as an initial target image for forming a new main frame image.

[0062] Furthermore, in step S4, consistency processing is performed on all the transformed main frame images, and a new video file is synthesized, specifically:

[0063] For each initial target image, calculate the reduction ratio λ i :

[0064]

[0065] Calculate the minimum reduction ratio of all initial target images:

[0066] λ=minλ i (i=1...N)

[0067] For each initial target image, change it so that the size of all the main frame images is consistent:

[0068] For the changed main frame picture, cut λ / λ from the center point of the initial target picture i As the new main frame picture, for the main frame picture that has not changed, directly cut out a picture of size λ from the center point of the original main frame picture as the new main frame picture, synthesize all the main frame pictures to generate the new video file.

[0069] A terrain-based video augmentation system for executing the terrain-based video augmentation method as described above, comprising:

[0070] Modify the change setting module, used to modify the road height change parameters of the map file as needed;

[0071] A map file analysis module is used to analyze the modified map file, combine the vehicle position file, find the modification points that the vehicle will pass through as vehicle posture change points, and find the video moments corresponding to the vehicle posture change points;

[0072] A main frame image transformation module, used to extract the main frame image of the vehicle posture change point in the video file, and transform the main frame image according to the modified road height change parameter;

[0073] The video file synthesis module is used to perform consistency processing on all the transformed main frame pictures and synthesize the new video file.

[0074] A computer-readable storage medium stores computer codes. When the computer codes are executed, the above method is executed.

[0075] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0076] (1) Solving the problem of video acquisition in special scenarios: In view of the difficulties in acquiring video data for special scenarios, especially the difficulty in reproducing extreme and boundary conditions, and the difficulty in acquiring videos of scenes such as vehicle tilt, by modifying the map file, the acquired video can be synchronized and the corresponding video can be generated, which makes up for the shortcoming that robustness testing is difficult to cover in all aspects, making the performance evaluation of intelligent systems in special scenarios more comprehensive.

[0077] (2) Reduce testing costs and difficulty: Compared with existing data re-collection technologies, this method avoids the problems of high difficulty and high cost of data collection. There is no need to redeploy data collection equipment, which reduces equipment purchase, maintenance and operation, and labor costs. It also saves a lot of time and avoids missing the product market launch window due to time-consuming data collection.

[0078] (3) Assist in the performance evaluation of intelligent systems: The ability to generate videos of vehicles driving on roads of different terrains helps to verify the performance of the video perception module of the intelligent system under special conditions, provides more effective data support for the robustness testing of the intelligent system, and promotes the research and development and optimization of the intelligent system. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A schematic diagram of driving onto a curb on one side of the present invention;

[0080] Figure 2 This is a schematic diagram of the present invention when driving on an inclined road;

[0081] Figure 3 is an overall flow chart of the terrain-based video augmentation method of the present invention;

[0082] Figure 4 Schematic diagram of the diagonal field of view DFOV of the camera of the present invention;

[0083] Figure 5 A schematic diagram of the installation position of the original video acquisition camera obtained by the present invention;

[0084] Figure 6 Schematic diagram of width and height of the original video of the present invention;

[0085] Figure 7 The vehicle video logic relationship diagram after the change of the present invention;

[0086] Figure 8 It is the overall structure diagram of the terrain-based video augmentation system of the present invention. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0088] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0089] First embodiment

[0090] like Figure 3 As shown, the present invention provides a terrain-based video augmentation method, which is characterized by comprising the following steps:

[0091] S1: Set the road height change parameters of the map file as needed.

[0092] First, in this embodiment, before setting the road height change parameters for the map file, it is necessary to read the acquisition configuration information including the diagonal field of view of the camera and the camera installation position, specifically:

[0093] like Figure 4 As shown, the diagonal field of view DFOV of the camera installed on the moving vehicle is read. The diagonal field of view DFOV refers to the angle range in the diagonal direction that the camera can capture. It describes the angle that can be covered from the center of the camera lens to both ends of the diagonal. Imagine that you are standing at the top of a cone. The base of the cone is the range that the camera can see. The angle between the generatrix of the cone and the central axis (in the diagonal direction) is the diagonal field of view.

[0094] like Figure 5As shown, the installation position of the acquisition camera in the original video file is obtained, which refers to the specific installation location of the camera on the vehicle (assuming it is a vehicle-mounted video acquisition system) or other equipment. This position is a spatial coordinate, including information in multiple dimensions such as height, horizontal position and angle. For example, on a car, the camera may be installed at the front, rear, roof or rearview mirror, and has a specific orientation and height. In this embodiment, the position information includes the distance d between the center point of the camera lens and the outermost side of the right tire in the horizontal direction, the distance h between the center point of the camera lens and the ground in the vertical direction, and the vehicle wheelbase D.

[0095] In step S1 of this embodiment, the road height change parameters are set for the map file as needed, specifically:

[0096] The setting of the road height change parameter includes setting superelevation information and adding road obstacles. The total height change value caused by the road height change parameter is the superelevation change + the height of the road obstacle.

[0097] Obtaining the original map file including road geometry information, elevation information, and attribute information, parsing the original map file into an operational data structure, defining superelevation modification ranges and parameters, and creating a template for the road obstacle including speed bumps;

[0098] Modify the superelevation information of the map file, select the target road section and key nodes that need to modify the superelevation in the operable data structure according to preset conditions, calculate and update the superelevation values ​​of the key nodes according to the defined superelevation modification range and parameters, and then rebuild the map file;

[0099] Perform the process of adding the road obstacle, determine the location of the road obstacle according to the test scenario requirements, select a suitable template of the road obstacle, adjust the attribute parameters of the template of the road obstacle, and add the template of the road obstacle with set attributes to the operational data structure.

[0100] The above technical solution can adopt the following process from the specific process steps of the technical solution:

[0101] 1. Data preparation stage

[0102] 1. Map file acquisition and analysis

[0103] The original map file is obtained from the storage system. The map file should contain road geometry information (such as road centerline, boundary line, etc.), elevation information, attribute information (such as road type, speed limit, etc.), etc.

[0104] Use specialized map parsing libraries or tools to parse map files into operable data structures. For example, convert map data into a node-edge graph structure, where nodes represent key road locations and edges represent road sections. At the same time, elevation information is associated with corresponding nodes or sections, and attribute information is also stored in a structured manner for subsequent query and modification.

[0105] 2. Define modification parameters and templates

[0106] According to the test requirements, determine the modification range and rules of superelevation. For example, set the superelevation increase value or change ratio range, as well as the corresponding road type or curve curvature conditions.

[0107] For road obstacles such as speed bumps, a standard template library of obstacles is created. The template should include information such as the speed bump's geometry (such as length, width, height, etc.), location information (such as offset on the road cross section), and material properties (such as the friction coefficient that affects vehicle driving).

[0108] 2. Super elevation modification process

[0109] 1. Road screening and positioning

[0110] According to pre-set conditions (such as specific road type, curve range, etc.), the target road section that needs to modify the superelevation is screened out in the parsed map data structure.

[0111] For the selected road sections, determine their key nodes (such as the starting point, end point, midpoint, etc. of the curve), which will serve as the main operating points for superelevation modification.

[0112] 2. Superelevation calculation and update

[0113] According to the set superelevation modification rules, the new superelevation value is calculated for each key node. For example, if the modification is based on a fixed increase value, the original superelevation is directly added to the set value; if the modification is based on a proportion, the new value is calculated based on the original superelevation and the set proportion.

[0114] The calculated new superelevation value is updated to the node or section attributes in the corresponding map data structure to ensure that the superelevation information of the entire road section is accurately modified.

[0115] 3. Map file reconstruction and verification

[0116] Use the updated map data structure to rebuild the map file through map generation tools or algorithms. During the reconstruction process, ensure the geometric continuity of the road and the smooth transition of the elevation to avoid unreasonable situations such as road breaks or sudden elevation changes.

[0117] Perform preliminary verification on the reconstructed map file to check whether the superelevation modification is in line with expectations. Visual tools can be used to view the superelevation corresponding to the modified road cross section and compare it with the set modification target.

[0118] 3. Process of adding road obstacles such as speed bumps

[0119] 1. Determine the location of obstacles

[0120] According to the test scenario requirements, select the appropriate road location in the map data structure to add speed bumps. The location of speed bumps can be determined based on a random location generation algorithm (random selection within a specified road type or area) or according to specific layout rules (such as adding one at a certain interval, adding one at a certain distance before an intersection, etc.).

[0121] The selected location information is converted into a coordinate point in a map data structure, and its exact position on the road cross section is determined (such as the offset relative to the center line of the road).

[0122] 2. Setting and adding obstacle attributes

[0123] Select a suitable speed bump template from the speed bump template library and adjust the attribute parameters of the template according to actual test requirements, such as the actual length, width, and height of the speed bump.

[0124] Add the speed bump object with set properties to the map data structure and associate it with the corresponding road location information to ensure its correct representation in the map.

[0125] 3. Map file integration and testing

[0126] The map data structure with the added speed bumps is integrated into the original map file to generate a new map file containing the speed bumps.

[0127] Functional testing of the new map file can be carried out by simulating the vehicle's driving path planning algorithm running on the new map to check whether the vehicle can correctly identify the location of the speed bump and make corresponding driving strategy adjustments (such as deceleration). At the same time, it can also be checked in a visual way whether the display effect of the speed bump on the map meets expectations.

[0128] 4. Final Verification and Storage Phase

[0129] 1. Comprehensive verification

[0130] Perform comprehensive verification on the map file after the super elevation is modified or the road obstacles are added. In addition to the above verification steps for individual modifications, the integrity, consistency and compatibility of the entire map file with other related systems (such as the positioning module and navigation module in the intelligent system) must also be checked.

[0131] Conduct multi-scenario simulation tests, using different vehicle driving trajectories and operating conditions, to verify the performance of the intelligent system when running based on the modified map file in a virtual environment, ensuring that map file modifications do not cause other potential problems.

[0132] 2. Storage and version management

[0133] If the verification is successful, the modified map file will be stored in the designated storage location, and the version will be marked and recorded according to the version management specification. The record should include the time of modification, the content of the modification (superelevation modification details, added obstacle information, etc.), the person who modified it, and other information for subsequent tracing and management.

[0134] Regularly back up stored map files to prevent data loss or damage and to be able to quickly restore to a previous version when needed.

[0135] S2: Analyze the modified map file, combine it with the vehicle position file, find out the modification points that the vehicle will pass through as vehicle posture change points, and find out the video moments corresponding to the vehicle posture change points.

[0136] In this embodiment, step S2 is specifically as follows:

[0137] Obtaining the modification point of the superelevation information and the location where the road obstacle is added in the operable data structure of the map file, and marking the information including coordinates, type, and attributes;

[0138] Read the vehicle position file to construct a vehicle driving trajectory, match the constructed vehicle driving trajectory with the map file, determine the road route of the vehicle, and predict the modification points passed by the vehicle during driving according to the modification points marked on the map file, where the modification points passed are the vehicle posture change points;

[0139] Determine the correspondence between the timestamp in the vehicle position file and the timestamp in the video file, establish a time synchronization model based on the correspondence, and calculate the video moment corresponding to the vehicle posture change point based on the time synchronization model.

[0140] The above technical solution can adopt the following process from the specific process steps of the technical solution:

[0141] 1. Map file analysis stage

[0142] 1. Map data analysis and feature extraction

[0143] Use professional map parsing software or custom-developed parsing modules to load and parse map files into structured data. Extract key geometric features of the road, such as curvature change points, slope change points, intersections, special landmarks (such as bridges, tunnel entrances and exits), and other information, all of which may cause changes in vehicle posture.

[0144] Special marks and records are made for superelevation modification points (such as the change position of different superelevation settings at curves) and added road obstacles (such as the position of speed bumps) because they are clear factors affecting vehicle posture. The coordinate information, type information, and related attribute information (such as superelevation value, obstacle size, etc.) of these special points are stored in a special data structure for subsequent fast query and matching.

[0145] 2. Vehicle location file processing stage

[0146] 1. Data reading and trajectory construction

[0147] Read the vehicle location file, which usually contains the latitude and longitude coordinates, altitude, speed, direction and other information of the vehicle at different times. Based on these data points, the vehicle's driving trajectory is constructed. Interpolation algorithms (such as linear interpolation or spline interpolation) can be used to supplement the missing data points to ensure the continuity and integrity of the trajectory.

[0148] 2. Driving route matching and change point prediction

[0149] The constructed vehicle trajectory is matched with the parsed map data to determine the specific road route of the vehicle. Based on the special points marked on the map (such as the above-mentioned posture change points), it is predicted which modification points the vehicle will pass through during driving. This can be achieved by calculating the geometric distance relationship between the vehicle trajectory and the special points. When the vehicle approaches a special point within a certain threshold range, it is determined that the vehicle will pass through the modification point.

[0150] 3. Video moment association stage

[0151] 1. Video timestamp calibration

[0152] Determine the correspondence between the timestamps in the vehicle position file and the timestamps in the video file. This may require some calibration operations, such as recording the position data and the video start timestamp at the same time when the collection vehicle starts, and recording the time information of both again at specific calibration points (such as when passing a known landmark), so as to establish an accurate time synchronization model.

[0153] 2. Determine the corresponding video time

[0154] Once it is determined that the vehicle will pass a certain modification point, the video moment corresponding to the modification point is calculated according to the time synchronization model. For example, if the vehicle is determined to pass a speed bump at a certain time point in the position file, the corresponding frame sequence can be found in the video file through the time synchronization relationship. These frame sequences are the video records of the vehicle passing the speed bump.

[0155] 4. Result Integration and Output Stage

[0156] 1. Data integration and storage

[0157] The found vehicle posture change points (including their coordinates on the map, type and other information), the time information in the corresponding vehicle location file and the video moment information are integrated and stored in a special data structure or database table. This makes it convenient to query, analyze and use these data later, for example, in the intelligent system robustness test, as a key basis for evaluating the system performance under different vehicle posture change scenarios.

[0158] 2. Visualization and report generation

[0159] In order to display the results more intuitively, a visualization interface can be developed to visualize the vehicle's driving trajectory, posture change points, corresponding video clips and other information on the map background. At the same time, a detailed report document is generated, including the analysis process, result data, possible error analysis, etc., so that technicians can better understand and evaluate the accuracy and effectiveness of the entire process.

[0160] S3: extracting a main frame image of the vehicle posture change point in the video file, and transforming the main frame image according to the modified road height change parameter.

[0161] In this embodiment, step S3 is specifically as follows:

[0162] S31: extracting the key frame IFrame from the video file to form the main frame image, and extracting the main frame image as the vehicle posture change point according to the video moment corresponding to the calculated vehicle posture change point.

[0163] For a given video file, extract the key frame IFrame to form the main frame image. The key frame IFrame is a frame in video encoding that can be independently decoded to obtain complete image information. It contains the key content of the video screen. Extracting these key frames is crucial to accurately grasp the important information in the video.

[0164] After obtaining the video moments corresponding to the vehicle posture change points, we will filter the extracted main frame images based on these time points. Specifically, in the process of extracting key frames to form main frame images, we will record the time information corresponding to each frame. Then, we will compare this time information with the video moments corresponding to the calculated vehicle posture change points.

[0165] When the time information of the main frame image matches the video moment corresponding to the vehicle posture change point, the main frame image is extracted as the main frame image of the vehicle posture change point. These images can accurately reflect the scene information of the vehicle at the moment of posture change, and provide intuitive and critical image basis for the subsequent in-depth analysis of the performance of the intelligent system under different vehicle postures, such as the recognition and tracking effect of the video perception module on the target. The main frame images extracted in this way can effectively serve related research and application work such as intelligent system robustness testing, and help evaluate the performance of intelligent systems in complex scenarios.

[0166] S32: transforming the main frame image of the vehicle posture change point according to the set road height change parameters, including the vehicle posture change in the image and calculating a new target area, to form a new main frame image of the vehicle posture change point to replace the original main frame image before the change.

[0167] In simple terms, the main technical solution for transforming the main frame image at the vehicle posture change point, including the vehicle posture change in the image and calculating the new target area, is: rotating the extracted main frame image to achieve the vehicle posture change, and then cutting the original main frame image according to the posture transformation perspective after rotation to obtain the new target area, so as to form a new main frame image at the vehicle posture change point.

[0168] The extracted main frame image is rotated to realize the vehicle posture change, and then the original main frame image is intercepted according to the posture change view after rotation to obtain a new target area, specifically:

[0169] Vehicle posture changes in the picture:

[0170] Calculating the height difference of the left and right tires relative to the original ground according to the road height change parameters of the road where the left and right tires of the vehicle are located, and rotating the left and right tires in the order of first the right tire and then the left tire, or first the left tire and then the right tire, according to the height difference of the left and right tires, so as to achieve a change in the vehicle posture;

[0171] Compute the new target region:

[0172] The vehicle's inclination angle is calculated based on the height difference between the left and right tires, and the center point of the new target area is recalculated based on the installation position information of the camera collected in the original main frame image. A largest rectangular area with the same aspect ratio as the original video is intercepted in the original main frame image with the center point of the new target area as the center as the new target area.

[0173] Based on the above technical solution, the specific process from the perspective of algorithm steps can be as follows:

[0174] Calculate the vehicle's tilt, H 左 H is the height difference between the bottom of the left tire and the original ground. 右 H is the height difference between the bottom of the right tire and the original ground. 左 and H 右 Positive means higher than the horizontal ground, negative means lower than the horizontal ground, and the value is calculated based on the road surface height change parameters of the road surface where the left and right tires are located;

[0175] like Figure 6 As shown, a video coordinate system is constructed with the center point of the original main frame image as the origin, the positive direction of the X-axis of the video coordinate system is parallel to the ground and in the direction from the right tire to the left tire, the positive direction of the Y-axis is perpendicular to the X-axis and upward, and the width of the original main frame image is A pixels and the height is B pixels;

[0176] Taking the right tire as the reference, the vehicle posture is changed according to the parameters of the set vehicle tilt degree, and the right tire is moved vertically by H 右 After that, the right tire remains unchanged and the left tire is moved vertically by H 左 ,The logical relationship between the changed vehicles and videos is as follows Figure 7 shown.

[0177] Calculate the vehicle's tilt angle θ as:

[0178]

[0179] The calculation needs to obtain the center point of the new target area:

[0180] a: Calculate the angle β between the center point of the camera lens in the original main frame image and the line connecting the contact point between the outer side of the original right tire and the ground:

[0181]

[0182] b: Calculate the length l of the line between the center of the camera lens and the contact point between the outer side of the original right tire and the ground in the original main frame image:

[0183]

[0184] c: Calculate the transformation coefficient γ between video and physical space:

[0185]

[0186] d: Calculate the center point of the new target area in the video coordinate system as:

[0187] x0'=(l*cos(β+θ)-d)*γ

[0188] y0'=(l*sin(β+θ)-h)*γ

[0189] If the center point of the new target area is not in the original video frame of the original main frame image, or the distance from the two sides of the original video frame is only 20% of the short side distance, the augmentation fails and the task ends;

[0190] Calculate a new target area, the new target is a rectangular area based on the center point with the same aspect ratio as the original main frame image, and take the maximum value within the range of the original main frame image, specifically:

[0191] a: Calculate the angle between the two diagonals of the original video frame and the X-axis:

[0192]

[0193] b: Calculate the angle between the two diagonals of the new target area and the X-axis:

[0194] θ1′=θ1+θ

[0195] θ2′=θ2+θ

[0196] c: Draw two diagonal lines with angles θ1' and θ2' with the X-axis through the center point of the new target area, find the intersection points a, b, c, d of the diagonal lines with the original video frame, take the shortest distance between the intersection point and the center point * 2 as the diagonal length L of the target area, and draw the new target area;

[0197] Pixels of a new target area are intercepted and collected from the original main frame image as an initial target image for forming a new main frame image.

[0198] S4: performing consistency processing on all the transformed main frame images and synthesizing the new video file.

[0199] In this embodiment, step S4 is specifically as follows:

[0200] For each initial target image, calculate the reduction ratio λ i :

[0201]

[0202] Calculate the minimum reduction ratio of all initial target images:

[0203] λ=minλ i (i=1...N)

[0204] For each initial target image, change it so that the size of all the main frame images is consistent:

[0205] For the changed main frame picture, cut λ / λ from the center point of the initial target picture i As the new main frame picture, for the main frame picture that has not changed, directly cut out a picture of size λ from the center point of the original main frame picture as the new main frame picture, synthesize all the main frame pictures to generate the new video file.

[0206] Second embodiment

[0207] like Figure 8 As shown, this embodiment provides a terrain-based video augmentation system for executing the terrain-based video augmentation method in the first embodiment, characterized by comprising:

[0208] Modification and change setting module 1, used to modify the road height change parameters of the map file as needed;

[0209] A map file analysis module 2 is used to analyze the modified map file, combine the vehicle position file, find the modification points that the vehicle will pass through as vehicle posture change points, and find the video moments corresponding to the vehicle posture change points;

[0210] A main frame image transformation module 3, used for extracting the main frame image of the vehicle posture change point in the video file, and transforming the main frame image according to the modified road height change parameter;

[0211] The video file synthesis module 4 is used to perform consistency processing on all the transformed main frame pictures and synthesize the new video file.

[0212] A computer-readable storage medium stores a computer code. When the computer code is executed, the above method is executed. A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium can include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0213] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0214] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0215] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A terrain-based video augmentation method, characterized in that: The following steps are involved: S1: Setting the road height change parameters of the map file as needed; S2: Analyze the modified map file, combine it with the vehicle position file, find out the modification points that the vehicle will pass through as vehicle posture change points, and find out the video moments corresponding to the vehicle posture change points; S3: extracting a main frame image of the vehicle posture change point in the video file, and transforming the main frame image according to the modified road height change parameter; S4: performing consistency processing on all the transformed main frame images and synthesizing the new video file.

2. The terrain-based video augmentation method according to claim 1, characterized in that: Before step S1, the process also includes reading and collecting acquisition configuration information including the diagonal field of view of the camera and the camera installation position, specifically: Reading the diagonal field of view DFOV of a camera installed on a moving vehicle; The installation position of the acquisition camera in the original video file is obtained, including the position information including the distance d between the center point of the camera lens and the outermost side of the right tire in the horizontal direction, the distance h between the center point of the camera lens and the ground in the vertical direction, and the vehicle wheelbase D.

3. The terrain-based video augmentation method according to claim 1, characterized in that: In step S1, the road height change parameters are set for the map file as required, specifically: The setting of the road height change parameter includes setting superelevation information and adding road obstacles. The total height change value caused by the road height change parameter is the superelevation change + the height of the road obstacle. Obtaining the original map file including road geometry information, elevation information, and attribute information, parsing the original map file into an operational data structure, defining superelevation modification ranges and parameters, and creating a template for the road obstacle including speed bumps; Modify the superelevation information of the map file, select the target road section and key nodes that need to modify the superelevation in the operable data structure according to preset conditions, calculate and update the superelevation values ​​of the key nodes according to the defined superelevation modification range and parameters, and then rebuild the map file; Perform the process of adding the road obstacle, determine the location of the road obstacle according to the test scenario requirements, select a suitable template of the road obstacle, adjust the attribute parameters of the template of the road obstacle, and add the template of the road obstacle with set attributes to the operational data structure.

4. The terrain-based video augmentation method according to claim 3, characterized in that: In step S2, the modified map file is analyzed, combined with the vehicle position file, to find the modification points that the vehicle will pass through as vehicle posture change points, and to find the video moments corresponding to the vehicle posture change points, specifically: Obtaining the modification point of the superelevation information and the location where the road obstacle is added in the operable data structure of the map file, and marking the information including coordinates, type, and attributes; Read the vehicle position file to construct a vehicle driving trajectory, match the constructed vehicle driving trajectory with the map file, determine the road route of the vehicle, and predict the modification points passed by the vehicle during driving according to the modification points marked on the map file, where the modification points passed are the vehicle posture change points; Determine the correspondence between the timestamp in the vehicle position file and the timestamp in the video file, establish a time synchronization model based on the correspondence, and calculate the video moment corresponding to the vehicle posture change point based on the time synchronization model.

5. The terrain-based video augmentation method according to claim 1, characterized in that: In step S3, the main frame image of the vehicle posture change point in the video file is extracted, and the main frame image is transformed according to the modified road height change parameter, specifically: Extracting the key frame IFrame from the video file to form the main frame image, and extracting the main frame image as the vehicle posture change point according to the video moment corresponding to the calculated vehicle posture change point; According to the set road height change parameters, the main frame image of the vehicle posture change point is transformed including the vehicle posture change in the image and the calculation of a new target area to form a new main frame image of the vehicle posture change point to replace the original main frame image before the change.

6. The terrain-based video augmentation method according to claim 5, characterized in that: According to the set road height change parameter, the main frame image of the vehicle posture change point is transformed including the vehicle posture change in the image and the calculation of the new target area, specifically: The extracted main frame image is rotated to realize the vehicle posture change, and then the original main frame image is cut out according to the posture change perspective after rotation to obtain a new target area, so as to form a new main frame image of the vehicle posture change point.

7. The terrain-based video augmentation method according to claim 6, characterized in that: The extracted main frame image is rotated to realize the vehicle posture change, and then the original main frame image is intercepted according to the posture change view after rotation to obtain a new target area, specifically: Vehicle posture changes in the picture: Calculating the height difference of the left and right tires relative to the original ground according to the road height change parameters of the road where the left and right tires of the vehicle are located, and rotating the left and right tires in the order of first the right tire and then the left tire, or first the left tire and then the right tire, according to the height difference of the left and right tires, so as to achieve a change in the vehicle posture; Compute the new target region: The vehicle's inclination angle is calculated based on the height difference between the left and right tires, and the center point of the new target area is recalculated based on the installation position information of the camera collected in the original main frame image. A largest rectangular area with the same aspect ratio as the original video is intercepted in the original main frame image with the center point of the new target area as the center as the new target area.

8. The terrain-based video augmentation method according to claim 5, characterized in that: The vehicle posture changes in the image and the calculation of the new target area are as follows: Calculate the vehicle's tilt, H 左 H is the height difference between the bottom of the left tire and the original ground. 右 H is the height difference between the bottom of the right tire and the original ground. 左 and H 右 Positive means higher than the horizontal ground, negative means lower than the horizontal ground, and the value is calculated based on the road surface height change parameters of the road surface where the left and right tires are located; A video coordinate system is constructed with the center point of the original main frame image as the origin, wherein the positive direction of the X-axis of the video coordinate system is parallel to the ground and in the direction from the right tire to the left tire, the positive direction of the Y-axis is perpendicular to the X-axis and upward, and the width of the original main frame image is A pixels and the height is B pixels; Taking the right tire as the reference, the vehicle posture is changed according to the parameters of the set vehicle tilt degree, and the right tire is moved vertically by H 右 After that, the right tire remains unchanged and the left tire is moved vertically by H 左 ; Calculate the vehicle's tilt angle θ as: The calculation needs to obtain the center point of the new target area: a: Calculate the angle β between the center point of the camera lens in the original main frame image and the line connecting the contact point between the outer side of the original right tire and the ground: b: Calculate the length l of the line between the center of the camera lens and the contact point between the outer side of the original right tire and the ground in the original main frame image: c: Calculate the transformation coefficient γ between video and physical space: d: Calculate the center point of the new target area in the video coordinate system as: x0'=(l*cos(β+θ)-d)*γ y0'=(l*sin(β+θ)-h))*γ If the center point of the new target area is not in the original video frame of the original main frame image, or the distance from the two sides of the original video frame is only 20% of the short side distance, the augmentation fails and the task ends; Calculate a new target area, the new target is a rectangular area based on the center point with the same aspect ratio as the original main frame image, and take the maximum value within the range of the original main frame image, specifically: a: Calculate the angle between the two diagonals of the original video frame and the X-axis: b: Calculate the angle between the two diagonals of the new target area and the X-axis: θ1′=θ1+θ θ2′=θ2+θ c: Draw two diagonal lines with angles θ1' and θ2' with the X-axis through the center point of the new target area, find the intersection points a, b, c, d of the diagonal lines with the original video frame, take the shortest distance between the intersection point and the center point * 2 as the diagonal length L of the target area, and draw the new target area; Pixels of a new target area are intercepted and collected from the original main frame image as an initial target image for forming a new main frame image.

9. The terrain-based video augmentation method according to claim 8, characterized in that: In step S4, consistency processing is performed on all the transformed main frame images, and a new video file is synthesized, specifically: For each initial target image, calculate the reduction ratio λ i : Calculate the minimum reduction ratio of all initial target images: λ=minλ i (i=1...N) For each initial target image, change it so that the size of all the main frame images is consistent: For the changed main frame picture, cut λ / λ from the center point of the initial target picture i As the new main frame picture, for the main frame picture that has not changed, directly cut out a picture of size λ from the center point of the original main frame picture as the new main frame picture, synthesize all the main frame pictures to generate the new video file.

10. A terrain-based video augmentation system for executing the terrain-based video augmentation method according to any one of claims 1 to 9, characterized in that: include: Modify the change setting module, used to modify the road height change parameters of the map file as needed; A map file analysis module is used to analyze the modified map file, combine the vehicle position file, find the modification points that the vehicle will pass through as vehicle posture change points, and find the video moments corresponding to the vehicle posture change points; A main frame image transformation module, used to extract the main frame image of the vehicle posture change point in the video file, and transform the main frame image according to the modified road height change parameter; The video file synthesis module is used to perform consistency processing on all the transformed main frame pictures and synthesize the new video file.

11. A computer-readable storage medium storing a computer code, wherein when the computer code is executed, the method according to any one of claims 1 to 9 is executed.

Images