Depth optimization-based ectopic dual-camera intelligent vehicle speed measurement method and system

Abstract

The invention relates to the technical field of video monitoring, and provides an ectopic dual-camera intelligent vehicle speed measurement method and system based on deep optimization. The method comprises the following steps: acquiring images acquired by two cameras which are arranged at different positions on the same side of a road; extracting features of each vehicle based on each vehicle detected by the first image and the second image, and matching the features of the vehicles in the first image and the second image; when a target vehicle reaches a speed measurement point area, determining a three-dimensional geometric projection constraint residual error according to depth estimation values of the two cameras and three-dimensional geometric projection constraints of the target vehicle and the two cameras, constructing an error-minimized target function in combination with a depth consistency error, and optimizing the depth estimation values of the two cameras to obtain a target vehicle speed measurement result; obtaining depth estimation optimization values of the first camera and the second camera; and converting the two-dimensional track pixel coordinates of the target vehicle into three-dimensional world coordinates based on the depth estimation optimization values of the first camera and the second camera, and calculating the final speed of the target vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of video surveillance, and particularly to an intelligent vehicle speed measurement method and system based on depth optimization with two cameras at different positions. Background Art

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] Existing vehicle speed measurement technologies mainly rely on radar, monocular cameras or binocular stereo vision systems. Among them, the monocular camera has insufficient speed measurement accuracy due to the lack of depth information; traditional binocular stereo vision requires binocular cameras with a fixed baseline distance, which are complex to install and costly. If a new speed measurement point needs to be set, additional radar or binocular cameras need to be installed, and the existing surveillance cameras widely present on traffic roads cannot be utilized.

[0004] In the current speed measurement methods relying on radar, monocular cameras or binocular stereo vision, obtaining or estimating the depth information of the scene is an important step in the speed measurement process. Among them, installing devices in addition to the conventional road surveillance cameras is required to obtain depth information using radar or based on binocular cameras. The estimated depth information obtained only using the existing road surveillance monocular cameras often has large errors.

[0005] In addition, when calibrating the camera, a calibration board is generally used for manual calibration, which is not applicable to some traffic scenes without manual setting of the calibration board and requires labor costs. Summary of the Invention

[0006] In order to solve the technical problems existing in the above background art, the present invention provides an intelligent vehicle speed measurement method and system based on depth optimization with two cameras at different positions. The present invention can make full use of the existing road surveillance cameras without additional installation of equipment; at the same time, it solves the problem of low speed measurement accuracy caused by inaccurate depth information of the monocular camera; the automatic calibration technology is adopted to improve the intelligence of the test scheme and greatly improve the scene adaptability of the present invention, facilitating the application and popularization of the scheme.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The first aspect of the present invention provides an intelligent vehicle speed measurement method based on depth optimization with two cameras at different positions.

[0009] An intelligent vehicle speed measurement method based on depth optimization with two cameras at different positions, comprising:

[0010] Obtaining the images collected by two cameras arranged at different positions on the same side of the road, and through processing, obtaining a first image and a second image;

[0011] For each vehicle detected based on the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image;

[0012] When the target vehicle reaches the speed measurement point area, based on the depth estimation values of the two cameras and the three-dimensional geometric projection constraints of the target vehicle with the two cameras, determine the three-dimensional geometric projection constraint residuals, and combine with the depth consistency error to construct an objective function for minimizing the error, and optimize the depth estimation values of the two cameras to obtain the optimized depth estimation value of the first camera and the optimized depth estimation value of the second camera;

[0013] Based on the optimized depth estimation value of the first camera and the optimized depth estimation value of the second camera, convert the two-dimensional trajectory pixel coordinates of the target vehicle into three-dimensional world coordinates, calculate the first speed and the second speed of the target vehicle, and obtain the final speed of the target vehicle.

[0014] Further, after processing, obtain the first image and the second image; the method includes:

[0015] Detect the vehicle corner points in each image as key feature points, extract multi-scale scene features, optimize the matching of the key feature points in the two images, and screen to obtain high-confidence matching point pairs;

[0016] Based on the matching point pairs, obtain the essential matrix; based on the essential matrix, obtain the external parameters of the camera; based on the external parameters and internal parameters of the camera, construct dynamic weight constraints;

[0017] Based on the dynamic weight constraints and semantic constraints, construct an objective function for minimizing the external parameters of the camera to optimize the two images and obtain the first image and the second image.

[0018] Further, for each vehicle detected based on the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image; the method includes:

[0019] Based on each vehicle detected in the first image and the second image, extract the timestamp, bounding box coordinates, unique identifier ID, and vehicle appearance features;

[0020] Based on the timestamps of the first image and the second image, select the first candidate matching pairs that meet the preset time threshold;

[0021] Based on the bounding box coordinates of the first image and the second image, select the second candidate matching pairs that meet the preset distance threshold from the first candidate matching pairs;

[0022] Based on the vehicle appearance features of the first image and the second image, in the second candidate matching pairs, select the final matching pair with the highest similarity to determine the same target vehicle located in the first image and the second image.

[0023] Further, the three-dimensional geometric projection constraint is expressed by the following formula:

[0024]

[0025] Wherein, K1 and K2 represent the depth estimation values of two cameras, and L represents the camera baseline vector between the two cameras;

[0026] The objective function for minimizing the error is expressed by the following formula:

[0027]

[0028] Wherein, represents the three-dimensional geometric projection constraint residual, λ×((A1 - B1) 2 +(A2 - B2) 2 ) represents the depth consistency error, λ is a weight coefficient, A1 and A2 represent the Euclidean distances after coordinate transformation, and B1 and B2 represent the deviations of the depth estimation values of the two cameras.

[0029] Further, optimize the depth estimation values of the two cameras to obtain the optimized first camera depth estimation value and the optimized second camera depth estimation value; The method includes:

[0030] If then enter the non-linear optimization; otherwise, output the optimized camera depth estimation value;

[0031] The non-linear optimization includes: initializing the deviation of the depth estimation values of the two cameras, optimizing the objective function for minimization, and iteratively updating the deviation of the depth estimation values of the two cameras; Design a residual vector, if the residual vector is less than the second set value, then stop the iteration and output the optimized first camera depth estimation value and the optimized second camera depth estimation value; otherwise, loop and optimize until convergence; Wherein, ∈ represents the first set value.

[0032] Further, calculate the first speed and the second speed of the target vehicle to obtain the final speed of the target vehicle; The method includes: based on the three-dimensional world coordinates of the target vehicle, obtain the initial position and the end position of the target vehicle in the speed measurement point area, combine the passing time, calculate the first speed and the second speed of the target vehicle corresponding to the videos collected by the two cameras, and calculate the average value of the first speed and the second speed to obtain the final speed.

[0033] The second aspect of the present invention provides an intelligent vehicle speed measurement system based on depth optimization with displaced dual cameras.

[0034] An intelligent vehicle speed measurement system with two off - site cameras based on deep optimization, comprising:

[0035] A data acquisition and processing module, which is configured to: acquire images collected by two off - site cameras located on the same side of the road, and after processing, obtain a first image and a second image;

[0036] A feature matching module, which is configured to: for each vehicle detected based on the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image;

[0037] A deep optimization module, which is configured to: when the target vehicle reaches the speed measurement point area, determine the three - dimensional geometric projection residual based on the depth estimation values of the two cameras and the three - dimensional geometric projection constraints of the target vehicle with the two cameras, combine the depth consistency error, construct an objective function for minimizing the error, optimize the depth estimation values of the two cameras, and obtain the optimized depth estimation value of the first camera and the optimized depth estimation value of the second camera;

[0038] A vehicle speed measurement module, which is configured to: based on the optimized depth estimation value of the first camera and the optimized depth estimation value of the second camera, convert the two - dimensional trajectory pixel coordinates of the target vehicle into three - dimensional world coordinates, calculate the first speed and the second speed of the target vehicle, and obtain the final speed of the target vehicle.

[0039] The third aspect of the present invention provides a computer device, which includes:

[0040] A processor, suitable for executing a computer program;

[0041] A computer - readable storage medium, in which a computer program is stored. When the computer program is executed by the processor, the steps in the method for intelligent vehicle speed measurement with two off - site cameras based on deep optimization described in the first aspect above are implemented.

[0042] The fourth aspect of the present invention provides a computer - readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor to perform the steps in the method for intelligent vehicle speed measurement with two off - site cameras based on deep optimization described in the first aspect above.

[0043] The fifth aspect of the present invention provides a computer program product or a computer program.

[0044] The present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the method for intelligent vehicle speed measurement based on depth-optimized non-coplanar dual cameras as described in the first aspect above.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] Based on the automatic estimation of depth information by a monocular camera, the present invention proposes a method for optimizing depth information based on the positional relationship between non-coplanar dual cameras, which constraints and optimizes the depth information estimated by the monocular camera, improves the accuracy of depth information, and thus improves the accuracy of speed measurement.

[0047] The present invention proposes a non-coplanar dual monocular camera based on a known positional relationship, and uses its positional relationship to constraint and optimize the three-dimensional geometric projection constraint and depth consistency verification optimization method for the depth information of the two cameras. Based on the double-loop iteration of geometric constraint verification and multivariable nonlinear optimization, the collaborative correction of depth estimation is realized, the accuracy of depth information is improved, and thus the accuracy of speed measurement is improved.

[0048] The present invention proposes an automatic calibration and camera displacement recalibration method based on the joint optimization of semantic segmentation filtering and geometric constraints. On the basis of combining vehicle corner detection and multi-scale feature extraction, a noise point (such as leaves and shadows) filtering part based on DeepLabV3+ semantic segmentation is added to improve the calibration accuracy. In addition, compared with the application of the traditional Bundle Adjustment optimization algorithm (minimizing the reprojection error through nonlinear optimization), the present invention designs and adds a semantic enhancement constraint term in the Bundle Adjustment framework, and fuses semantic constraints (lane line parallelism, vehicle symmetry) to jointly optimize the internal and external parameters of the camera to improve the robustness to complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0050] Figure 1 is a flowchart of the method for intelligent vehicle speed measurement based on depth-optimized non-coplanar dual cameras shown in an embodiment of the present invention;

[0051] Figure 2 is a specific flowchart of the method for intelligent vehicle speed measurement based on depth-optimized non-coplanar dual cameras shown in an embodiment of the present invention;

[0052] Figure 3 It is a flowchart showing the deeply optimized ectopic positional relationship in an embodiment of the present invention;

[0053] Figure 4 It is an example diagram showing the positions of dual cameras in an embodiment of the present invention;

[0054] Figure 5 It is a structural diagram of an intelligent vehicle speed measurement system with dual cameras based on deep optimization and ectopic in an embodiment of the present invention;

[0055] Figure 6 It is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed implementation manners

[0056] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0057] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanations of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0058] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0059] For the convenience of understanding the technical solution of the present invention, some technical terms related to the present invention will be introduced below.

[0060] HRNet (High-Resolution Network) is a deep learning model that focuses on maintaining high-resolution feature representations, avoiding the loss of spatial information caused by downsampling in traditional networks, and achieving high-precision feature learning through parallel multi-resolution branches.

[0061] SuperGlue is a feature matching algorithm based on deep learning, which solves the feature association problem between images by combining graph neural networks and optimal transport theory.

[0062] DeepLabV3 significantly improves the accuracy of dense prediction tasks by improving dilated convolution and multi-scale context modeling.

[0063] RANSAC (RANdom SAmple Consensus) is a robust model fitting algorithm used to estimate the parameters of a mathematical model from data containing a large amount of noise or outliers. Its core idea is to screen out inliers that are consistent with the true model through a random sampling + voting mechanism.

[0064] The eight-point algorithm is the core method in computer vision for calculating the fundamental matrix between two views. Its core goal is to establish an epipolar geometric constraint relationship through matching point pairs.

[0065] Based on this, this embodiment provides an intelligent vehicle speed measurement method based on depth optimization for two cameras with different positions, as Figure 1 shown, including:

[0066] Obtain the images collected by two cameras with different positions on the same side of the road. After processing, obtain the first image and the second image;

[0067] For each vehicle detected based on the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image;

[0068] When the target vehicle reaches the speed measurement point area, based on the depth estimation values of the two cameras and the three-dimensional geometric projection constraints of the target vehicle with respect to the two cameras, determine the three-dimensional geometric projection constraint residuals. Combine the depth consistency error to construct an objective function that minimizes the error, and optimize the depth estimation values of the two cameras to obtain the optimized depth estimation value of the first camera and the optimized depth estimation value of the second camera;

[0069] Based on the optimized depth estimation value of the first camera and the optimized depth estimation value of the second camera, convert the two-dimensional trajectory pixel coordinates of the target vehicle into three-dimensional world coordinates, calculate the first speed and the second speed of the target vehicle, and obtain the final speed of the target vehicle.

[0070] As Figure 2As shown in the figure, the present invention provides an intelligent vehicle speed measurement method based on deep optimization with an off - site dual - camera. Through off - site camera layout and automatic camera calibration, it realizes the off - site placement of dual cameras to form an independent observation perspective layout, constructs a right - angle constraint relationship for off - site dual cameras and an automatic calibration method based on the joint optimization of semantic segmentation filtering and geometric constraint; based on the stereo vehicle multi - threshold matching of off - site dual cameras, it uses the dual cameras with staggered layouts to form independent visual units for observation to achieve synchronous image capture and matching of vehicles; based on the depth optimization of off - site position relationships, through the double - loop iteration of geometric constraint verification and multi - variable non - linear optimization, it jointly optimizes the three - dimensional geometric projection constraint and depth consistency error to achieve the collaborative correction of off - site dual - camera depth estimation. It realizes and optimizes the speed measurement based on the detection and tracking of road vehicles by a deep - network model, and builds an intelligent speed - measurement hardware system for off - site dual cameras using the computer vision development board jetson Orin Nano.

[0071] In some embodiments, images collected by two off - site cameras arranged on the same side of the road are obtained and, after processing, a first image and a second image are obtained. The specific implementation process includes:

[0072] The off - site dual - camera layout and automatic camera calibration part is mainly realized through the following three steps:

[0073] 1) Construct an observation perspective for feature detection and matching: The two cameras are respectively installed at the front and rear positions on the same side of the road. The corner points of the vehicle are detected by YOLOv8 as key feature points, multi - scale scene features are extracted by HRNet, the cross - camera feature point matching is optimized by combining with the SuperGlue algorithm, and high - confidence matching point pairs are screened using DeepLabV3 + semantic segmentation (such as filtering noise regions like leaves and shadows).

[0074] The present invention uses two "off - site" monitoring cameras at two different positions with known position relationships that widely exist in traffic roads, forming a new method framework for off - site speed measurement based on camera position constraints.

[0075] 2) Initial calibration and geometric constraint fusion: Based on the matching point pairs, the fundamental matrix F is estimated using the RANSAC + eight - point algorithm, and then the fundamental matrix F is converted into the essential matrix E through the camera internal parameter matrices K1 and K2. The specific formula is as follows:

[0076]

[0077] Then, the essential matrix E is decomposed to obtain the initial external parameters (R, t); assuming the principal point is located at the center of the image Initialize the focal length parameter (intrinsic parameter K) according to the sensor size and resolution ratio. Construct dynamic weight constraints (adjust the weights of different feature points according to semantic information) through geometric priors (such as lane line parallelism, vehicle symmetry) to suppress noise interference.

[0078] 3) Semantic enhancement joint optimization: Compared with the application of traditional Bundle Adjustment, in the Bundle Adjustment framework of the present invention, a semantic enhancement constraint term is designed and added to jointly optimize the camera intrinsic parameter K (f x , f y , c x , c y ), and the extrinsic parameters (R, t). The optimization formula is as follows:

[0079]

[0080] Among them, represents the semantic constraint term (such as lane line parallelism, vehicle symmetry), λ represents the weight coefficient (default set to 0.1). K, R, t are the camera intrinsic and extrinsic parameters respectively, represents the coordinates of the i-th 3D point in the world coordinate system, As a projection function, it transforms the point in the world coordinate system to the camera coordinate system through the extrinsic parameters, and then projects it to the 2D image plane through the intrinsic parameters to obtain the projected coordinates. (u i , v i ) represents the actual observed coordinates of the i-th 3D point on the image plane. is used to measure the difference between the theoretical projection and the observed value.

[0081] The present invention proposes a camera automatic calibration method based on the joint optimization of semantic segmentation filtering and geometric constraints, which can make full use of the existing cameras on the road, improve the scene adaptability of the road speed measurement method, realize the filtering of noise points (such as leaves, shadows) to improve the calibration accuracy, and enhance the robustness to complex scenes.

[0082] In some embodiments, for each vehicle detected based on the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image; the specific implementation process includes:

[0083] The stereo vehicle coordinate matching is realized in the following three steps:

[0084] 1) Object Detection and Multi-Feature Extraction: The object detection algorithm YOLOv5 is used to process each frame of the video. For each detected target vehicle, its timestamp, bounding box coordinates, and unique identifier ID are extracted. The extracted vehicle appearance features (color, texture, contour) are stored in separate documents, corresponding to the vehicle trajectory information in the left and right videos respectively.

[0085] 2) Multi-Threshold Screening and Feature Fusion: First, candidate matching pairs are initially screened using a preset time difference threshold and Euclidean space distance threshold, and only the candidate matching pairs that satisfy the time difference (less than or equal to the preset time threshold) and spatial distance (less than or equal to the preset distance threshold) are retained. Then, multi-feature fusion is used to perform weighted calculations on color, texture, and contour features to generate a comprehensive similarity index. Finally, based on the criterion of the minimum comprehensive value, the global similarity of the candidate pairs is quantified, providing data support for subsequent optimal matching decisions. The Euclidean distance formula is:

[0086]

[0087] where (x1, y1) and (x2, y2) are the upper left and lower right coordinates of the recognition box.

[0088] 3) Optimal Matching Strategy and Determination of the Best Match: Based on the preliminary matching results, the candidate pair with the minimum comprehensive similarity is selected as the final match, ensuring that each target corresponds to only one best match result in the left and right camera trajectory data. This strategy significantly improves the accuracy and robustness of vehicle matching in the off-axis dual-camera scenario by integrating spatio-temporal constraints and multi-dimensional features.

[0089] In some embodiments, when the target vehicle reaches the speed measurement point area, based on the depth estimation values of the two cameras and the three-dimensional geometric projection constraints between the target vehicle and the two cameras, the three-dimensional geometric projection constraint residuals are determined. Combining the depth consistency error, an objective function for minimizing the error is constructed to optimize the depth estimation values of the two cameras, obtaining the optimized first camera depth estimation value and the optimized second camera depth estimation value. The specific implementation process includes:

[0090] The above-mentioned two off-axis cameras work together, where Camera 1 is vertically deployed at the speed measurement point and Camera 2 is flexibly deployed. After determining the same vehicle as the common observation point of the dual cameras under the vehicle multi-threshold matching module, when the observation point reaches the speed measurement point area, the depth optimization determination is triggered. Among them, using the three-dimensional geometric projection constraints between the two known monocular cameras, the depth information estimated by each monocular camera is corrected and optimized through depth optimization under a double loop. The whole process mainly includes pixel coordinate extraction and initial coordinate conversion, double loop iterative depth optimization based on geometric constraint verification and multi-variable non-linear optimization, and coordinate conversion again using the optimized depth information. The specific process is as Figure 3As shown, and the positional relationship for the camera to trigger depth optimization determination is as Figure 4 shown.

[0091] The present invention proposes a method for accurately measuring vehicle speed by using two non - collocated monocular cameras at known positions of road monitoring. The two cameras work together. Among them, camera 1 is vertically arranged at the speed measurement point, and camera 2 is flexibly deployed. By using the non - collocated vertical constraint relationship between the two known monocular cameras, the depth information estimated by each monocular camera is corrected and optimized, and the optimized depth information is used for vehicle speed measurement.

[0092] The goal of depth optimization is to jointly optimize the depth estimates K1 and K2 of camera 1 and camera 2 through a multi - variable non - linear optimization framework to satisfy three - dimensional geometric constraints and depth consistency. Among them, the three - dimensional geometric constraint is expressed as that the depth estimates of the two cameras need to satisfy the three - dimensional space geometric relationship under the same world coordinate for the observation point. The mathematical description is that assuming the camera baseline vector is L, within the orthogonal projection plane (the geometric plane where the line connecting the observation point and one of the cameras is perpendicular to the baseline), the depth estimate needs to satisfy multi - view geometric consistency:

[0093]

[0094] And the depth consistency error is defined as the deviation between the Euclidean distances A1, A2 after coordinate transformation and the monocular depth estimates B1, B2:

[0095] Depth error = λ×((A1 - B1) 2 +(A2 - B2) 2 )

[0096] Among them, λ is a weight coefficient (by default, λ is 0.1), which balances geometric constraints and depth consistency.

[0097] Therefore, the objective function under the overall framework can be expressed as:

[0098]

[0099] Among them, represents the three - dimensional geometric projection constraint residual, and λ×((A1 - B1) 2 +(A2 - B2) 2 ) represents the depth consistency error.

[0100] The specific double - loop iterative depth optimization process is as follows:

[0101] (1) The first loop - geometric constraint verification

[0102] Input: The initial depth estimates B1, B2 of camera 1 and camera 2, and assume that the baseline distance between the two cameras is a fixed value L.

[0103] Verification condition: Under the three-dimensional geometric projection constraint, the verification logic for reaching the first-level optimization: If (∈ = 0.1m, representing the first set value), then enter the second-level loop; otherwise, it is proved that the depth is reasonable, exit the loop, and the initial depth is the output result.

[0104] (2) The second-level loop - Multivariable nonlinear optimization

[0105] Initialize parameters:

[0106] Iterative optimization: Use the Levenberg-Marquardt algorithm to minimize the objective function, and iteratively update K1 and K2 within the framework of the second-level loop.

[0107] Residual vector design:

[0108] Convergence judgment: If ||r|| < δ (δ = 10 -3 , representing the second set value), stop the iteration and output the result at this time. The output result is the optimized depth; otherwise, loop for optimization until convergence.

[0109] The present invention proposes a heterotopic dual monocular camera based on known positional relationships, and a three-dimensional geometric projection constraint and depth consistency verification optimization method for optimizing the depth information of two cameras using their positional relationship constraints. Through a double-loop iteration based on geometric constraint verification and multivariable nonlinear optimization, collaborative correction of depth estimation is achieved, and the accuracy of depth information is improved, thereby improving the accuracy of speed measurement.

[0110] In some embodiments, based on the optimized values of the depth estimation of the first camera and the optimized values of the depth estimation of the second camera, the two-dimensional trajectory pixel coordinates of the target vehicle are converted into three-dimensional world coordinates, and the first speed and the second speed of the target vehicle are calculated to obtain the final speed of the target vehicle; the specific implementation process includes:

[0111] Based on the depth information obtained after depth optimization through the double-loop iteration of the previous module, the two-dimensional trajectory pixel coordinates can be accurately converted into actual three-dimensional world coordinates, providing accurate and reliable data support for the speed measurement module.

[0112] The vehicle detection, tracking, and speed calculation module implements this function through the following steps: 1) Vehicle detection and initial information acquisition: In video frames or images, the YOLOv5 model is used to detect vehicle candidate regions in real time and obtain bounding box information, which includes target coordinates, width, height, and confidence, etc. 2) Vehicle tracking and continuous monitoring: Based on the initial position, category, and confidence of the detected vehicles, the DeepSORT algorithm is combined to achieve multi-target continuous tracking. The Kalman filter is used to predict the target motion trajectory, and the Hungarian algorithm is used to solve the target association problem in occlusion scenarios, ensuring the continuity and correctness of tracking. 3) Vehicle speed calculation: Based on the optimized depth information, the changes in vehicle trajectories between consecutive frames and camera parameters are analyzed. The 3D reconstruction method is used to convert the two-dimensional trajectory coordinates into positions in the world coordinate system, and the real-time speed of the vehicle is accurately calculated. The coordinate conversion formula is as follows:

[0113]

[0114] where s is the depth, M is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation matrix, u and v are the abscissa and ordinate of the target vehicle trajectory, and X, Y, and Z are the converted world coordinates.

[0115] The coordinates of the two images are respectively converted to obtain the coordinates of the vehicle, the speeds of the two images are calculated, and the average value is taken to obtain the final speed.

[0116] The present invention constructs a hardware system for intelligent speed measurement with an off-site dual camera, including the following steps: 1) Hardware platform construction and algorithm optimization: The Jetson Orin Nano development board is selected as the hardware platform. Based on the 128-core GPU and 50 TOPS computing power characteristics of the Jetson Orin Nano development board, its GPU computing power and AI acceleration performance are utilized to improve the inference speed and ensure that the algorithm can run efficiently on the device. 2) Speed measurement and performance verification: Combining deep learning and computer vision technologies, vehicles are detected and tracked through the video streams captured by the dual cameras. Using the deployed YOLOv5 model and combining the TensorRT acceleration technology, FP16 / INT8 quantization and layer fusion optimization are adopted to increase the inference frame rate of the video stream and reduce resource occupancy at the same time, ensuring that the system can calculate the vehicle speed stably and accurately.

[0117] To verify the solution of this embodiment, video data was collected in Area A, covering different traffic flows (low, medium, and high), and the improvement effects of the depth optimization module and the automatic calibration method on the speed measurement accuracy and system robustness were mainly verified. First, by introducing the three-dimensional geometric projection constraint and the depth consistency double-loop iterative optimization algorithm, the accuracy of monocular depth estimation was significantly improved. Before optimization, the monocular depth estimation error was about 1.8 m, and after optimization, it dropped to 0.3 m, with a reduction rate of 83.3%. After optimization, the mean value of the geometric constraint residuals approaches 0 (residual range: ±0.05 m), verifying the effectiveness of the three-dimensional geometric projection constraint. The specific data is shown in Table 1.

[0118] Table 1 Comparison of depth optimization effects of off-site dual cameras

[0119]

[0120] Then, based on the automatic calibration technology that combines semantic segmentation filtering and geometric constraint joint optimization in this embodiment, in terms of calibration time consumption, manual calibration takes 20 - 30 minutes per point, while automatic calibration only takes 3 - 5 minutes per point, with a 6-fold efficiency improvement. Under the traditional Bundle Adjustment constraint, the calibration error (reprojection error) is 1.8 ± 0.5 pixels, while under the joint constraint with semantic enhancement of the present invention, the reprojection error is reduced to 0.7 ± 0.2 pixels, with a 61.1% accuracy improvement.

[0121] Moreover, the present invention introduces a stereo vehicle multi-threshold matching strategy for off-site dual cameras. By determining the observation points through triple thresholds and combining the continuity characteristics of the vehicle movement trajectory, it effectively suppresses the false matching interference in different shooting scenarios, and the vehicle matching success rate can reach 97.9%; the YOLOv5 model and DeepSORT algorithm are used to achieve continuous tracking, and the vehicle tracking rate can reach 98.3%. In the dual-camera collaborative solution, the mean absolute error in the vehicle speed measurement task is only 2.5 km / h, achieving high-confidence speed synchronization calculation and improving the speed accuracy. The specific data performance is shown in Table 2.

[0122] Table 2 Performance data table of off-site dual camera collaborative solution

[0123] Solution Ectopic dual-camera collaboration Mean absolute error (km / h) 2.5±0.8 Maximum instantaneous error (km / h) 7.1 Effective detection distance (m) 35-180 Multi-target tracking success rate 98.3％ Vehicle matching rate 97.9％

[0124] In practical applications, the vehicle speed measurement system needs to maintain high performance in various complex environments. Tests were conducted for complex environments such as low light and target occlusion. Taking the occlusion test as an example, an occlusion test was conducted near area B, simulating the vehicle's occlusion and light conditions in complex weather by blocking the glass. The test results show that when the occlusion rate is 30%, the system's speed measurement error is 3.5±1.4km / h, and the tracking success rate is 91.5%. This shows that under mild occlusion conditions, the system can still maintain a high speed measurement accuracy and tracking success rate. The specific data is shown in Table 3.

[0125] Table 3 System performance data table under simulated complex environment

[0126]

[0127] In terms of cost, compared with existing road cameras and speed measurement technologies, the present invention not only improves the speed measurement accuracy, but also has a lower cost. Specifically, using existing road monitoring cameras, the hardware cost is almost zero, the installation cost is also almost zero, and the maintenance cost is extremely low, about 10,000 to 20,000 yuan / year. In comparison, the equipment price of a monocular fixed camera is between 500-5,000 yuan, the installation cost is about 10,000 to 30,000 yuan, and the maintenance cost is about 10,000 to 20,000 yuan / year. The equipment price of the dual-camera (on both sides of the road) solution is about 50,000 to 150,000 yuan, the installation cost is about 30,000 to 80,000 yuan, and the maintenance cost is about 20,000 to 50,000 yuan / year. The equipment price of the radar speed measurement benchmark is about 7,900 yuan / unit, and a complete radar-assisted camera system costs about 100,000 to 500,000 yuan, the installation cost is about 50,000 to 100,000 yuan, and the maintenance cost is about 20,000 to 50,000 yuan / year.

[0128] Combination of the above Figure 1 , Figure 2 The ectopic dual-camera intelligent vehicle speed measurement method based on depth optimization provided in an embodiment of the present invention is introduced in detail. Next, the ectopic dual-camera intelligent vehicle speed measurement system based on depth optimization provided in an embodiment of the present invention will be introduced in conjunction with the accompanying drawings.

[0129] Figure 5 is a schematic diagram of the structure of the ectopic dual-camera intelligent vehicle speed measurement system based on deep optimization shown in an embodiment of the present invention, referring to Figure 5 , the system of the present invention comprises:

[0130] An ectopic dual-camera intelligent vehicle speed measurement system based on deep optimization, comprising:

[0131] A data acquisition and processing module is configured to: acquire images collected by two cameras arranged at different positions on the same side of the road, and obtain a first image and a second image through processing;

[0132] A feature matching module, which is configured to: for each vehicle detected in the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image;

[0133] A depth optimization module, which is configured to: when the target vehicle reaches the speed measurement point area, determine the three-dimensional geometric projection constraint residual according to the depth estimation values of two cameras and the three-dimensional geometric projection constraint of the target vehicle and the two cameras, combine the depth consistency error, construct an objective function for minimizing the error, and optimize the depth estimation values of the two cameras to obtain the optimized first camera depth estimation value and the optimized second camera depth estimation value;

[0134] A vehicle speed measurement module, which is configured to: based on the optimized first camera depth estimation value and the optimized second camera depth estimation value, convert the two-dimensional trajectory pixel coordinates of the target vehicle into three-dimensional world coordinates, calculate the first speed and the second speed of the target vehicle, and obtain the final speed of the target vehicle.

[0135] In some possible implementation manners, the data acquisition and processing module is specifically configured to: detect the vehicle corner points in each image as key feature points, extract multi-scale scene features, optimize the matching of the key feature points in the two images, and screen to obtain high-confidence matching point pairs; based on the matching point pairs, obtain the essential matrix; based on the essential matrix, obtain the external parameters of the camera; based on the external parameters and internal parameters of the camera, construct a dynamic weight constraint; based on the dynamic weight constraint and the semantic constraint, construct an objective function for minimizing the external parameters of the camera to optimize the two images to obtain the first image and the second image.

[0136] In some possible implementation manners, the feature matching module is specifically configured to: for each vehicle detected in the first image and the second image, extract the timestamp, the bounding box coordinates, the unique identifier ID, and the vehicle appearance features; based on the timestamps of the first image and the second image, select the first candidate matching pairs that meet the preset time threshold; based on the bounding box coordinates of the first image and the second image, select the second candidate matching pairs that meet the preset distance threshold from the first candidate matching pairs; based on the vehicle appearance features of the first image and the second image, select the final matching pair with the highest similarity from the second candidate matching pairs to determine the same target vehicle located in the first image and the second image.

[0137] In some embodiments, the three-dimensional geometric projection constraint is represented by the following formula:

[0138]

[0139] Wherein, K1 and K2 represent the depth estimation values of two cameras, and L represents the camera baseline vector between the two cameras;

[0140] The objective function for minimizing the error is expressed by the following formula:

[0141]

[0142] Wherein, represents the three-dimensional geometric projection constraint residual, and λ×((A1 - B1) 2 +(A2 - B2) 2 ) represents the depth consistency error, λ is the weight coefficient, A1 and A2 represent the Euclidean distances after coordinate transformation, and B1 and B2 represent the deviations of the depth estimation values of the two cameras.

[0143] In some possible implementation manners, the depth optimization module is specifically configured as: if then enter the non-linear optimization; otherwise, output the optimized value of the camera depth estimation; the non-linear optimization includes: initializing the deviation of the depth estimation values of the two cameras, optimizing the objective function for minimization, and iteratively updating the deviation of the depth estimation values of the two cameras; designing the residual vector, if the residual vector is less than the second set value, then stop the iteration and output the optimized value of the first camera depth estimation and the optimized value of the second camera depth estimation; otherwise, loop and optimize until convergence; wherein, ∈ represents the first set value.

[0144] In some possible implementation manners, the vehicle speed measurement module is specifically configured as: based on the three-dimensional world coordinates of the target vehicle, obtain the initial position and the end position of the target vehicle in the speed measurement point area, combine the passing time, calculate the first speed and the second speed of the target vehicle corresponding to the videos collected by the two cameras, and calculate the average value of the first speed and the second speed to obtain the final speed.

[0145] According to the embodiment of the present invention, the intelligent vehicle speed measurement system with non-synchronous dual cameras based on depth optimization can correspond to execute the method described in the embodiment of the present invention, and the above and other operations and / or functions of each module of the intelligent vehicle speed measurement system with non-synchronous dual cameras based on depth optimization are respectively for implementing Figure 1 the corresponding processes of each method in, for the sake of brevity, will not be elaborated herein.

[0146] See Figure 6The structural diagram of the computer device shown, the computer device includes a processor, a communication interface, and a computer-readable storage medium. Among them, the processor, the communication interface, and the computer-readable storage medium can be connected through a bus or other means. Among them, the communication interface is used to receive and send data. The computer-readable storage medium can be stored in the memory of the computer device, and the computer-readable storage medium is used to store a computer program, and the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer-readable storage medium. The processor (or CPU (Central Processing Unit, central processor)) is the computing core and control core of the computer device, and it is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding steps in the embodiment of the intelligent vehicle speed measurement method based on deeply optimized off-axis dual cameras.

[0147] This embodiment provides a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in the computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, and the processing system of the computer device is stored in this storage space.

[0148] And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory; optionally, it can also be at least one computer-readable storage medium located far from the aforementioned processor.

[0149] In one embodiment, one or more instructions are stored in the computer-readable storage medium; the processor loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the embodiment of the intelligent vehicle speed measurement method based on deeply optimized off-axis dual cameras.

[0150] This embodiment provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in the computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the corresponding steps in the embodiment of the intelligent vehicle speed measurement method based on deeply optimized off-axis dual cameras.

[0151] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0152] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0153] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0155] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0156] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An intelligent vehicle speed measurement method based on deeply optimized off-site dual cameras, characterized in that Including: Obtain the images collected by two cameras arranged at different positions on the same side of the road, and after processing, obtain the first image and the second image; For each vehicle detected based on the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image; When the target vehicle reaches the speed measurement point area, based on the depth estimation values of the two cameras and the three-dimensional geometric projection constraints of the target vehicle with the two cameras, determine the three-dimensional geometric projection constraint residuals, and combine the depth consistency error to construct an objective function for minimizing the error, and optimize the depth estimation values of the two cameras to obtain the optimized first camera depth estimation value and the optimized second camera depth estimation value; Based on the optimized first camera depth estimation value and the optimized second camera depth estimation value, convert the two-dimensional trajectory pixel coordinates of the target vehicle into three-dimensional world coordinates, calculate the first speed and the second speed of the target vehicle, and obtain the final speed of the target vehicle.

2. The method for intelligent vehicle speed measurement based on depth optimization with an off - axis dual - camera according to claim 1, wherein, After processing, obtain the first image and the second image; The method includes: Detect the vehicle corner points in each image as key feature points, extract multi-scale scene features, optimize the matching of the key feature points in the two images, and filter to obtain high-confidence matching point pairs; Based on the matching point pairs, obtain the essential matrix; based on the essential matrix, obtain the external parameters of the camera; based on the external parameters and internal parameters of the camera, construct a dynamic weight constraint; Based on the dynamic weight constraint and the semantic constraint, construct an objective function for minimizing the external parameters of the camera to optimize the two images and obtain the first image and the second image.

3. The method for intelligent vehicle speed measurement based on depth optimization with two cameras at different positions according to claim 1, characterized in that, For each vehicle detected based on the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image; The method includes: For each vehicle detected based on the first image and the second image, extract the timestamp, bounding box coordinates, unique identifier ID, and vehicle appearance features; Based on the timestamps of the first image and the second image, select the first candidate matching pairs that meet the preset time threshold; Based on the bounding box coordinates of the first image and the second image, select the second candidate matching pairs that meet the preset distance threshold from the first candidate matching pairs; Based on the vehicle appearance features of the first image and the second image, select the final matching pair with the highest similarity from the second candidate matching pairs to determine the same target vehicle located in the first image and the second image.

4. The method for intelligent vehicle speed measurement based on depth optimization with an off - axis dual - camera according to claim 1, wherein, The three-dimensional geometric projection constraint is expressed by the following formula: Where, K1 and K2 represent the depth estimation values of the two cameras, and L represents the camera baseline vector between the two cameras; The objective function for minimizing the error is expressed by the following formula: Among them, represents the three-dimensional geometric projection constraint residual, and λ×((A1 - B1) 2 +(A2 - B2) 2 ) represents the depth consistency error, λ is the weight coefficient, A1 and A2 represent the Euclidean distances after coordinate transformation, and B1 and B2 represent the deviations of the depth estimates of the two cameras.

5. The method for intelligent vehicle speed measurement based on depth optimization with two cameras at different positions according to claim 4, wherein Optimize the depth estimation values of the two cameras to obtain the optimized first camera depth estimation value and the optimized second camera depth estimation value; The method includes: If then enter non-linear optimization; otherwise, output the optimized value of camera depth estimation; Nonlinear optimization includes: initializing the deviation of the depth estimates of two cameras, optimizing the objective function to be minimized, and iteratively updating the deviation of the depth estimates of the two cameras; designing a residual vector, and if the residual vector is less than the second set value, stopping the iteration and outputting the optimized depth estimate value of the first camera and the optimized depth estimate value of the second camera; otherwise, looping for optimization until convergence; where ∈ represents the first set value.

6. The method for intelligent vehicle speed measurement based on depth optimization with an off-site dual camera according to claim 1, characterized in that, Calculating the first speed and the second speed of the target vehicle to obtain the final speed of the target vehicle; the method includes: based on the three-dimensional world coordinates of the target vehicle, obtaining the initial position and the end position of the target vehicle in the speed measurement point area, and combining the passing time, calculating the first speed and the second speed of the target vehicle corresponding to the videos collected by the two cameras, and calculating the average value of the first speed and the second speed to obtain the final speed.

7. An intelligent vehicle speed measurement system with an ectopic dual camera based on deep optimization, characterized in that, Including: A data acquisition and processing module, which is configured to: acquire the images collected by two cameras arranged at different positions on the same side of the road, and after processing, obtain a first image and a second image; A feature matching module, which is configured to: based on each vehicle detected in the first image and the second image, extract the features of each vehicle, and match the features of the vehicles in the first image and the second image to determine the same target vehicle located in the first image and the second image; A depth optimization module, which is configured to: when the target vehicle reaches the speed measurement point area, determine the three-dimensional geometric projection residual based on the depth estimates of the two cameras and the three-dimensional geometric projection constraints between the target vehicle and the two cameras, combine the depth consistency error, construct an objective function for minimizing the error, and optimize the depth estimates of the two cameras to obtain the optimized depth estimate value of the first camera and the optimized depth estimate value of the second camera; A vehicle speed measurement module, which is configured to: based on the optimized depth estimate value of the first camera and the optimized depth estimate value of the second camera, convert the two-dimensional trajectory pixel coordinates of the target vehicle into three-dimensional world coordinates, calculate the first speed and the second speed of the target vehicle, and obtain the final speed of the target vehicle.

8. A computer device, characterized in that a processor, adapted to execute a computer program; a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by the processor, the steps in the method for intelligent vehicle speed measurement using two cameras at different positions based on depth optimization according to any one of claims 1-6 are implemented.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by the processor to implement the steps in the method for intelligent vehicle speed measurement using two cameras at different positions based on depth optimization according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by the processor, the steps in the method for intelligent vehicle speed measurement using two cameras at different positions based on depth optimization according to any one of claims 1-6 are implemented.

Images