Video target fusion method for road test sensing system

Abstract

The invention discloses a video target fusion method for a road test sensing system. Comprising the following steps: acquiring an intersection video through a camera; 10 feature points in the video are selected and marked; measuring GPS information of the feature points; converting the GPS coordinates into Gaussian projection coordinates; calculating an internal parameter and an external parameter of the camera through the pixel point coordinate information and the Gaussian projection coordinates, and generating a conversion formula; according to a conversion formula, scene pixel coordinate points shot in real time are converted into Gaussian projection coordinates; converting the Gaussian projection coordinates into GPS information again; and multi-scene video fusion is realized through region division. According to the method, the Yolov8 model is adopted for reasoning, at least 200 pictures need to be collected for annotation training for each scene, multiple types of target results detected by the model are subjected to multi-scene fusion, blind areas can be reduced, the detection accuracy can be improved, the fused video data contain more types of information, and the video quality is improved. And the traffic flow, the vehicle behavior and the pedestrian dynamic state can be analyzed more accurately.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a video target fusion method for a road test perception system. Background Art

[0002] With the development of modern intelligent transportation systems and autonomous driving technology, road test perception has become more and more widespread, and its application scenarios are as follows: (1) Traffic monitoring: Real-time monitoring of traffic flow, vehicle speed and driving trajectory on the road, helping traffic management departments to promptly detect and deal with traffic congestion, accidents and other problems. Safety warning: By detecting pedestrians, vehicles and other obstacles on the road, safety warnings are provided to reduce the occurrence of traffic accidents. (2) Data collection: Collect a large amount of traffic data, including vehicle type, license plate number, driving speed, etc., to provide data support for traffic planning and management. Intelligent traffic control: Combined with real-time traffic data, optimize the control of traffic lights, improve traffic efficiency, and reduce vehicle waiting time. (3) Law enforcement support: By monitoring and recording traffic violations such as speeding and running red lights, traffic law enforcement is assisted to improve road traffic order. However, the accuracy of these scenarios still needs to be continuously optimized.

[0003] Since road test cameras are usually installed in dynamic environments and are affected by factors such as lighting changes, weather conditions and wind, traditional calibration methods are difficult to guarantee their accuracy and stability. In addition, the calibration process is cumbersome and time-consuming, and often requires an oversized chessboard or circular calibration plate for calibration. The special location of the camera increases the difficulty of calibration. In addition, the camera detection method has a limited field of view and requires splicing the results of several cameras. Video data processing and fusion require a lot of computing resources, which increases the complexity and cost of the system. These problems will affect the accuracy of detection. Summary of the invention

[0004] The purpose of the present invention is to provide a video target fusion method for a road test perception system to solve the problems raised in the above background technology.

[0005] The technical solution adopted by the present invention to solve the technical problem comprises the following steps: Step 1: Install a camera at a designated intersection, adjust camera parameters, and specify the camera frame rate to collect video in real time; Step 2: For each scene corresponding to each video, select 10 feature points that meet the requirements for each scene for annotation, and obtain the pixel coordinates corresponding to each feature point; Step 3: Measure the GPS coordinates of the feature points in step 2 in the actual scene, which include longitude and latitude; Step 4: Use the conversion formula in geodesy to convert the acquired longitude and latitude information into Gaussian projection coordinates. The conversion process involves converting spherical coordinates (longitude and latitude) into plane coordinates (Gaussian projection coordinates); Step 5: The pixel coordinates in step 2 and the Gaussian projection coordinates in step 4 are calculated by formula to obtain the intrinsic and extrinsic parameters of the camera and generate a transformation matrix; Step 6: Each camera needs to annotate at least 200 pictures, including 9 categories of targets, and use the Yolov8 model to train the target detection model; Step 7: Use the Yolov8 model to infer the video frames collected by the camera in real time, and use the conversion matrix to convert the pixel coordinate information of the target into GPS information in real time; Step 8: Divide the shooting area into regions based on the actual situation. In this way, the results of all cameras are integrated and GPS information is output in real time.

[0006] Furthermore, in step 1, the basic parameters of the camera, including resolution, exposure time, and white balance, are set through the camera's configuration interface or management software, and the camera's frame rate is adjusted according to actual needs.

[0007] Furthermore, in step three, in the actual scene, a high-precision GPS device is used to measure the actual coordinates of each feature point; the GPS device will provide the longitude and latitude information of the feature point; the longitude and latitude represent the specific location of the feature point on the earth's surface.

[0008] Furthermore, the Gaussian projection coordinates in step 4 are plane coordinates. The longitude and latitude measured by the GPS device are converted according to the WGS84 ellipsoid parameters. The basic parameters include the major semi-axis of the ellipsoid, the flattening of the ellipsoid, the minor semi-axis of the ellipsoid, the first eccentricity of the ellipsoid, and the second eccentricity of the ellipsoid. The Gaussian projection is used to convert the coordinates into projection coordinates. The conversion formula is as follows: ; in, represents the transformed projection coordinates, , and All represent constants. Indicates latitude, Indicates longitude, ,in is the central meridian longitude, is the meridian arc length, is the meridian curvature radius, expressed as: ; in, and Both represent constants, meridian arc length It is expressed as: ; in , , , , is a constant, expressed as: ;

[0009] in , , , , It is expressed as: ; ; ; ; ; Input the longitude and latitude information in step 3, calculate through the above formula, output Gaussian projection coordinates, and complete the conversion from GPS coordinates to projection coordinates.

[0010] Furthermore, step five is implemented as follows: Based on the pixel coordinates and Gaussian projection coordinates of the 10 feature points obtained in steps 2 and 4, the Gaussian projection coordinates Transform to the camera coordinate system through rotation and translation operations , the camera coordinate system is transformed into the image coordinate system, that is, the pixel coordinate system, through the local intrinsic parameters of the camera. The formula for converting Gaussian projection to camera coordinates is expressed as: ;

[0011] in is the rotation matrix, is the translation matrix, which projects the points in the camera coordinate system to the image plane as follows: ;

[0012] in, represents the coordinates of the image plane point, is the camera intrinsic parameter matrix, which is expressed as: ;

[0013] The conversion formula from pixel coordinates to Gaussian projection coordinates is expressed as: ; in, The vector coordinates representing the Gaussian projection coordinates, Indicates the corresponding pixel coordinates, Represent the matrix as the intrinsic parameters of the camera: The main optical axis point, usually the center of the image; and is the focal length, and the rotation and translation matrices of the camera intrinsic parameters and Gaussian projection coordinates to the camera coordinates are obtained based on these points; and All represent constants.

[0014] Furthermore, step six is ​​implemented as follows: At least 200 images need to be collected for annotation for each scene. The annotations are divided into 9 categories, namely Cyclist, Car, Trafficcon, Pedestrian, Motorcyclist, Bus, Van, Truck, and Barrowlist. After the annotations are completed, Yolov8 model training is used; Backbone of Yolov8 is mainly responsible for feature extraction. It adopts CSPDarkNet structure and introduces C2f module. Neck of Yolov8 uses PANet structure for feature fusion. The Head part of Yolov8 adopts the Decoupled-Head structure to separate the classification and detection heads; the Loss calculation of Yolov8 adopts the positive and negative sample allocation strategy and Loss calculation; the task alignment allocator is used to dynamically adjust the allocation ratio of positive and negative samples; specifically, the positive sample is selected according to the combination of classification score and regression score; the Loss calculation includes two branches: classification and regression; VFL Loss proposes an asymmetric weighted operation, which is expressed as: ;

[0015] Among them, p and q represent dynamic variables; by using The factor scaling loss of Yolov8 is divided into two parts: CIou_Loss and Distribution Focal Loss; Distribution Focal Loss selects the optimization label Left and right positions , The probability of making the network distribution focus on the label value, DistributionFocal Loss is expressed as: ;

[0016] Furthermore, step seven is implemented as follows: The pixel coordinates inferred by Yolov8 in real time are converted by the formula in step 5 to obtain the corresponding Gaussian projection coordinates. The Gaussian projection coordinates need to be converted into GPS coordinates. The latitude coordinates are expressed as: ; ; ;

[0017] in, is the central meridian longitude, , , , , It represents the variable obtained by calculating the Gaussian projection coordinates. Substituting multiple Gaussian projection coordinates into the formula respectively, the GPS coordinates can be derived.

[0018] Furthermore, step eight is specifically implemented as follows: regional division is performed based on the actual number of scenes. Since the camera field of view is fixed and the distance of the scene that can be photographed is limited, it is necessary to divide each scene area, that is, the scene detected by each camera is different, thereby removing duplicate targets, splicing the GPS information output by step seven at the same time, and finally outputting the fused GPS information.

[0019] The above technical solutions in the embodiments of the present application have at least the following technical effects: The method provided by the implementation of this application first collects video images and GPS information of corresponding points, and converts GPS information into Gaussian projection coordinates; secondly, the conversion formula is used to iterate pixel coordinates and corresponding Gaussian projection coordinates to obtain the camera's internal and external parameters, complete the camera calibration, and obtain the conversion matrix; then, through the Yolov8 training model, real-time inference of video frame images, obtain the pixel coordinates of multiple targets, substitute the target pixel coordinates and conversion matrix and other parameters into the formula, and convert them into longitude and latitude information; finally, the regional pixel coordinates of multiple scenes are divided to ensure that the area detected by each camera is not repeated, and multiple scenes can be spliced, and the longitude and latitude information at the same time after splicing conversion is completed to complete the video target fusion. This process uses a single-stage model reasoning, and the use of matrix conversion operations can shorten the operation time, improve efficiency, ensure the real-time effectiveness of information, and save equipment costs. In addition, the method of this paper adopts a calibration method without a calibration plate, which only needs to select 10 points, is simple to operate, and the calibration error range is approximately within 0.6m when the camera is not shaken, which has practical application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flow chart of a video target fusion method for a road test perception system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] Before introducing the embodiments of the present disclosure, it should be noted that: Some embodiments of the present disclosure are described as processing flows. Although the various operation steps of the flow may be given sequential step numbers, the operation steps therein may be implemented in parallel, concurrently or simultaneously.

[0022] The term “and / or” may be used in embodiments of the present disclosure. “And / or” includes any and all combinations of one or more of the associated features listed.

[0023] In order to make the technical solutions and advantages of the embodiments of the present disclosure more clearly understood, the exemplary embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0024] Embodiment 1: Step 1: Fix the camera at the designated location, ensure the camera parameters, and obtain the video in real time by adjusting the camera parameters; Step 2: Get the picture of the scene, filter out 10 feature points, and record the pixel coordinates; Step 3: Measure the GPS coordinates of the feature points in step 2 in the actual scene, which includes information such as latitude, longitude, and altitude; Step 4: Use the formula to convert the collected longitude and latitude information into Gaussian projection coordinates, so that the longitude and latitude information can be mapped to a two-dimensional plane, which is convenient for subsequent calculation operations; Step 5: The pixel coordinates in step 2 and the Gaussian projection coordinates in step 4 are calculated using the formula to obtain the camera's intrinsic and extrinsic parameters. The formula includes parameters such as the rotation vector, translation vector, and camera intrinsic matrix. Step 6: Label the objects in the scene and use the Yolov8 model to train the data. By adjusting the input size and data enhancement, the detection accuracy is not significantly changed, the running time is shortened, and the real-time information is guaranteed. Step 7: Use the Yolov8 model to infer the video frames collected by the camera in real time, convert the pixel coordinate information of the target into GPS coordinates in real time and output them; In the specific implementation process, Example 1 is mainly applied to a single intersection scene, which requires real-time pulling of the camera's video frames. A single scene does not need to consider transmission time and fusion issues. In step 4, the longitude and latitude information can be converted into a unified Gaussian projection coordinate by combining parameters such as pi, ellipsoid major semi-axis, ellipsoid flattening, ellipsoid minor semi-axis, ellipsoid first eccentricity, and ellipsoid second eccentricity. Substituting into the conversion formula from pixel coordinates to camera coordinates to Gaussian projection coordinates, the formula can be expressed as: ;

[0025] The camera’s intrinsic and extrinsic parameters are calculated, and the pixel coordinates inferred by Yolov8 are converted into longitude and latitude coordinates using the above formula. The results can be directly output for a single scene to expand the detection area as much as possible.

[0026] Embodiment 2: Step 1: Fix multiple cameras at the specified location, ensure the camera parameters, and adjust the camera parameters to obtain scene video images in real time. Generally, the camera frame rate is 25fps, and the image is transmitted once every 40ms. Step 2: Get the video frames of each scene and select 10 feature points for each scene; Step 3: Measure the GPS coordinates of multiple sets of feature points in step 2 in the actual scene, which include information such as latitude, longitude, and altitude; Step 4: Use the formula to convert the collected longitude and latitude information into Gaussian projection coordinates, so that the longitude and latitude information can be mapped to a two-dimensional plane, which is convenient for subsequent calculation operations; Step 5: The pixel coordinates in step 2 and the Gaussian projection coordinates in step 4 are calculated by formula to obtain the camera's intrinsic and extrinsic parameters. The formula includes parameters such as rotation vector, translation vector, and camera intrinsic matrix. A corresponding conversion file is generated for each scene. Step 6: Label the objects in multiple scenes and use the Yolov8 model to train these data. By adjusting the input size and data enhancement, the detection accuracy is not significantly changed, the running time is shortened, and the real-time information is guaranteed. Step 7: Use the Yolov8 model to infer the video frames collected by the camera in real time, convert the pixel coordinates of the target into GPS coordinates in real time through the conversion file and output them; Step 8: Divide the shooting area into regions based on actual conditions to ensure that the areas detected by each camera are different and can be spliced ​​together. Use regional division to splice GPS information of different scenes at the same time.

[0027] In the specific implementation process, Example 2 is mainly applied to multiple intersection scenes. Multiple scenes need to consider the time interval of camera transmission. This method controls the time alignment of multi-camera transmission to ensure the accuracy of subsequent data fusion. Through step five, the conversion files of different cameras are obtained, corresponding to different scenes. Video frames from different cameras are received at the same time. The Yolov8 model performs reasoning in a serial manner, and the inference results are converted into GPS information. Different cameras are divided into different areas to ensure that these areas can be spliced, and the GPS information output at the same time is spliced ​​to complete the fusion of video targets.

[0028] Working principle of the invention: The invention proposes a video target fusion method for a road test perception system, which performs calibration without a calibration board, converts the pixel coordinates inferred by the model into GPS coordinates, and finally fuses multiple videos according to regional division. This method improves accuracy while reducing resource utilization, and has practical application significance.

[0029] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in the present examples. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0030] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A video target fusion method for a drive test perception system, characterized in that: The following steps are involved: Step 1: Install a camera at a designated intersection, adjust camera parameters, and specify the camera frame rate to collect video in real time; Step 2: For each scene corresponding to each video, select 10 feature points that meet the requirements for each scene for annotation, and obtain the pixel coordinates corresponding to each feature point; Step 3: Measure the GPS coordinates of the feature points in step 2 in the actual scene, which include longitude and latitude; Step 4: Use the conversion formula in geodesy to convert the acquired longitude and latitude information into Gaussian projection coordinates. The conversion process involves converting spherical coordinates into plane coordinates. Step 5: The pixel coordinates in step 2 and the Gaussian projection coordinates in step 4 are calculated by formula to obtain the intrinsic and extrinsic parameters of the camera and generate a transformation matrix; Step 6: Each camera needs to annotate at least 200 pictures, including 9 categories of targets, and use the Yolov8 model to train the target detection model; Step 7: Use the Yolov8 model to infer the video frames collected by the camera in real time, and use the conversion matrix to convert the pixel coordinate information of the target into GPS information in real time; Step 8: Divide the shooting area into regions based on the actual situation. In this way, the results of all cameras are integrated and GPS information is output in real time.

2. The video target fusion method for a drive test perception system according to claim 1, characterized in that: In the step 1, the basic parameters of the camera, including resolution, exposure time, and white balance, are set through the camera configuration interface or management software, and the frame rate of the camera is adjusted according to actual needs.

3. The video target fusion method for a drive test perception system according to claim 2, characterized in that: In the step 3, in the actual scenario, a high-precision GPS device is used to measure the actual coordinates of each feature point; the GPS device will provide the longitude and latitude information of the feature point; the longitude and latitude represent the specific position of the feature point on the earth's surface.

4. The video target fusion method for a drive test perception system according to claim 3, characterized in that: The Gaussian projection coordinates in step 4 are plane coordinates. The longitude and latitude measured by the GPS device are converted according to the WGS84 ellipsoid parameters. The basic parameters include the major semi-axis of the ellipsoid, the flattening of the ellipsoid, the minor semi-axis of the ellipsoid, the first eccentricity of the ellipsoid, and the second eccentricity of the ellipsoid. The Gaussian projection is used to convert the coordinates into projection coordinates. The conversion formula is as follows: ； in, represents the transformed projection coordinates, , and All represent constants. Indicates latitude, Indicates longitude, ,in is the central meridian longitude, is the meridian arc length, is the meridian curvature radius, expressed as: ； in, and Both represent constants, meridian arc length It is expressed as: ； in , , , , is a constant, expressed as: ； in , , , , It is expressed as: ； ； ； ； ； Input the longitude and latitude information in step 3, calculate through the above formula, output Gaussian projection coordinates, and complete the conversion from GPS coordinates to projection coordinates.

5. The video target fusion method for a drive test perception system according to claim 4, characterized in that: The step five is specifically implemented as follows: Based on the pixel coordinates and Gaussian projection coordinates of the 10 feature points obtained in steps 2 and 4, the Gaussian projection coordinates Transform to the camera coordinate system through rotation and translation operations , the camera coordinate system is transformed into the image coordinate system, that is, the pixel coordinate system, through the local intrinsic parameters of the camera. The formula for converting Gaussian projection to camera coordinates is expressed as: ； in is the rotation matrix, is the translation matrix, which projects the points in the camera coordinate system to the image plane as follows: ； in, represents the coordinates of the image plane point, is the camera intrinsic parameter matrix, which is expressed as: ； The conversion formula from pixel coordinates to Gaussian projection coordinates is expressed as: ； in, The vector coordinates representing the Gaussian projection coordinates, Indicates the corresponding pixel coordinates, Represent the matrix as the intrinsic parameters of the camera: The main optical axis point, usually the center of the image; and is the focal length, and the rotation and translation matrices of the camera intrinsic parameters and Gaussian projection coordinates to the camera coordinates are obtained based on these points; and All represent constants.

6. The video target fusion method for a drive test perception system according to claim 5, characterized in that: The step six is ​​specifically implemented as follows: At least 200 images need to be collected for annotation for each scene. The annotations are divided into 9 categories, namely Cyclist, Car, Trafficcon, Pedestrian, Motorcyclist, Bus, Van, Truck, and Barrowlist. After the annotations are completed, Yolov8 model training is used; Backbone of Yolov8 is mainly responsible for feature extraction. It adopts CSPDarkNet structure and introduces C2f module. Neck of Yolov8 uses PANet structure for feature fusion. The head part of Yolov8 adopts the Decoupled-Head structure to separate the classification and detection heads; the loss calculation of Yolov8 adopts the positive and negative sample allocation strategy and loss calculation; The task alignment allocator is used to dynamically adjust the allocation ratio of positive and negative samples; Specifically, positive samples are selected based on the combination of classification scores and regression scores; Loss calculation includes two branches: classification and regression; VFL Loss proposes an asymmetric weighted operation, which is expressed as: ； Among them, p and q represent dynamic variables; by using The factor scaling loss of Yolov8 is divided into two parts: CIou_Loss and Distribution Focal Loss; Distribution Focal Loss selects the optimization label Left and right positions , The probability of making the network distribution focus on the label value, Distribution FocalLoss is expressed as: 。 7. The video target fusion method for a drive test perception system according to claim 6, characterized in that: The step seven is specifically implemented as follows: The pixel coordinates inferred by Yolov8 in real time are converted by the formula in step 5 to obtain the corresponding Gaussian projection coordinates. The Gaussian projection coordinates need to be converted into GPS coordinates. The latitude coordinates are expressed as: ； ； ； in, is the central meridian longitude, , , , , It represents the variable obtained by calculating the Gaussian projection coordinates. Substituting multiple Gaussian projection coordinates into the formula respectively, the GPS coordinates can be derived.

8. The video target fusion method for a drive test perception system according to claim 7, characterized in that: The specific implementation of step eight is as follows: regional division is performed based on the actual number of scenes. Since the camera field of view is fixed and the distance of the scene that can be photographed is limited, it is necessary to divide each scene area, that is, the scene detected by each camera is different, so as to remove duplicate targets, splice the GPS information output by step seven at the same time, and finally output the fused GPS information.

Images