Vehicle speed measurement method and monocular camera

Abstract

The application discloses a vehicle speed measurement method and a monocular camera, and belongs to the technical field of intelligent transportation. The method comprises the following steps: performing three-dimensional feature prediction on a multi-scale feature map of a current frame monocular image to obtain three-dimensional coordinates of a center point of at least one target vehicle; predicting a coordinate position of a dangerous point of the target vehicle based on the multi-scale feature map and key attributes to obtain three-dimensional coordinates of the dangerous point; respectively processing the three-dimensional coordinates of the center point and the three-dimensional coordinates of the dangerous point to determine a center point speed and a dangerous point speed of the target vehicle; dynamically adjusting a weight distribution between the center point speed and the dangerous point speed based on a heading angle; and performing weighted fusion on the center point speed and the dangerous point speed based on a result of the weight distribution to obtain a driving speed of the target vehicle. The application can significantly improve the robustness and scene adaptability of vehicle speed estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, and in particular to a vehicle speed measurement method and a monocular camera. BACKGROUND

[0002] In intelligent driving, traffic monitoring and other application scenarios, accurately obtaining the motion state (such as speed) of other traffic participants on the road is the basis for ensuring driving safety and realizing autonomous decision-making.

[0003] For a monocular camera, its advantage lies in low cost and easy deployment, making it an important sensing component on a vehicle. However, due to its imaging principle, the image collected by the monocular camera loses depth information in the process of projecting from three-dimensional space to two-dimensional plane, which leads to the inability to directly calculate the real moving distance and direction of the target in the real world from the image displacement, thereby affecting the accuracy of speed calculation.

[0004] In related technologies, the speed estimation method based on monocular vision image mainly relies on the displacement change of the center point of the target detection frame between consecutive frames for derivation. However, in actual complex traffic environment, this method faces many challenges, for example, in the following or crossing driving scene, part or all of the target object area may be blocked by the front obstacle, resulting in the center point being invisible, and thus the speed estimation of the target object fails; due to the influence of factors such as target posture change, light fluctuation or detector instability, the center point is prone to drift or jitter, which makes the speed measurement result based on the displacement of the center point deviate significantly. Therefore, in the above-mentioned scenarios, how to accurately predict the speed of the target becomes a problem to be solved. SUMMARY

[0005] The embodiments of the present application provide a vehicle speed measurement method, which can solve the technical problem that part or all of the target object area is blocked in the driving scene, resulting in the center point being invisible, and making the speed prediction of the target object unstable.

[0006] In a first aspect, the embodiments of the present application provide a vehicle speed measurement method, which comprises:

[0007] performing feature extraction on a current frame monocular image to obtain a multi-scale feature map of the current frame monocular image;

[0008] performing three-dimensional feature prediction on the multi-scale feature map to obtain key attributes of at least one target vehicle in the current frame monocular image, wherein the key attributes include 3D scale, three-dimensional coordinates of the center point, depth value of the center point and heading angle;

[0009] for each target vehicle,

[0010] predict a coordinate position of the dangerous point of the target vehicle based on the multi-scale feature map and the key attributes, to obtain a three-dimensional coordinate of the dangerous point, wherein the dangerous point is a midpoint of a bottom side of a detection bounding box of the target vehicle;

[0011] respectively process the three-dimensional coordinate of the center point and the three-dimensional coordinate of the dangerous point, to determine a center point speed and a dangerous point speed of the target vehicle;

[0012] based on the heading angle, dynamically adjust a weight distribution between the center point speed and the dangerous point speed, and based on a result of the weight distribution, perform weighted fusion on the center point speed and the dangerous point speed, to obtain a driving speed of the target vehicle.

[0013] Optionally, the predicting of the coordinate position of the dangerous point of the target vehicle based on the multi-scale feature map and the key attributes, to obtain the three-dimensional coordinate of the dangerous point, comprises:

[0014] input the multi-scale feature map and the three-dimensional coordinate of the center point into an offset prediction head in an offset depth fusion network, to output a position offset of the dangerous point relative to the center point on a monocular image plane of a current frame, and based on the position offset, determine an x-axis coordinate and a y-axis coordinate of the dangerous point in a camera coordinate system, wherein the position offset comprises an offset in an x-axis direction and an offset in a y-axis direction;

[0015] input the multi-scale feature map and the depth value of the center point into a depth estimation network in the offset depth fusion network, to obtain a discrete representation of the depth of each pixel point in the multi-scale feature map, and further obtain a depth value of the dangerous point in the camera coordinate system;

[0016] based on the x-axis coordinate, the y-axis coordinate and the depth value of the dangerous point in the camera coordinate system, and a camera calibration parameter, calculate a three-dimensional coordinate of the dangerous point in a world coordinate system through a back projection function.

[0017] Optionally, the obtaining of the discrete representation of the depth of each pixel point in the multi-scale feature map comprises:

[0018] for each pixel point in the multi-scale feature map, predict a probability that the pixel point belongs to each preset depth discrete value through the depth estimation network, multiply each probability by a corresponding depth discrete value, and then sum the results, to obtain a depth value corresponding to the pixel point.

[0019] Optionally, the predicting of the coordinate position of the dangerous point of the target vehicle based on the multi-scale feature map and the key attributes, to obtain the three-dimensional coordinate of the dangerous point, comprises:

[0020] inputting the multi-scale feature map into a dangerous point heat map prediction branch in the dangerous point prediction network to obtain a dangerous point heat map, and determining a 2D image coordinate of the dangerous point based on the dangerous point heat map, wherein the dangerous point heat map is used to represent a probability that each spatial position in the multi-scale feature map belongs to a dangerous point;

[0021] based on a dangerous point depth prediction branch in the dangerous point prediction network, obtaining a depth value of the dangerous point from the multi-scale feature map through the 2D image coordinate of the dangerous point in the dangerous point heat map;

[0022] determining a three-dimensional coordinate of the dangerous point based on the 2D image coordinate and the depth value of the dangerous point.

[0023] Optionally, the three-dimensional coordinate of the dangerous point of the target vehicle is predicted based on the multi-scale feature map and the key attributes, including:

[0024] determining the length of the target vehicle based on the 3D scale of the target vehicle, wherein the 3D scale of the target vehicle includes the length, width and height of the target vehicle;

[0025] based on the three-dimensional coordinate of the center point of the target vehicle, the three-dimensional coordinate of the dangerous point is calculated by using the geometric relationship between the length of the target vehicle and the heading angle.

[0026] Optionally, the weight distribution between the center point speed and the dangerous point speed is dynamically adjusted based on the heading angle, and the driving speed of the target vehicle is obtained by weighted fusion based on the result of the weight distribution, including:

[0027] the driving speed of the target vehicle is obtained by the following formula:

[0028] ;

[0029] wherein, represents the heading angle, represents the dangerous point speed, represents the center point speed.

[0030] Optionally, the multi-scale feature map is subjected to three-dimensional image feature prediction to obtain the key attributes of at least one target vehicle in the current frame monocular image, including:

[0031] inputting the multi-scale feature map into a heat map prediction branch in the three-dimensional detection network for class prediction to obtain a class heat map, and determining at least one target center point position based on the class heat map, wherein the class heat map represents a probability that each spatial position in the multi-scale feature map belongs to a target center point;

[0032] input the multi-scale feature map into an offset prediction branch in the three-dimensional detection network for processing to obtain a center point offset of each spatial position in the multi-scale feature map;

[0033] based on a coordinate prediction branch in the three-dimensional detection network, the three-dimensional coordinates of the center point of the target vehicle are obtained from the multi-scale feature map through the index of the target center point position on the category heat map and the center point offset corresponding to the target center point position;

[0034] based on a size prediction branch in the three-dimensional detection network, the 3D scale of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the category heat map, wherein the 3D scale of the target vehicle includes the length, width and height of the target vehicle;

[0035] based on a depth prediction branch in the three-dimensional detection network, the depth value of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the category heat map;

[0036] based on a heading angle prediction branch in the three-dimensional detection network, the heading angle of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the category heat map.

[0037] Optionally, the three-dimensional coordinates of the center point and the three-dimensional coordinates of the dangerous point are processed respectively to determine the center point speed and the dangerous point speed of the target vehicle, including:

[0038] the three-dimensional coordinates of the center point of the current frame and the historical center point state vector of the last frame are updated by Kalman filtering to determine the center point speed of the target vehicle, wherein the center point state vector includes the three-dimensional coordinates (x, y, z) of the center point and the speed in x, y and z axis directions;

[0039] the three-dimensional coordinates of the dangerous point of the current frame and the historical dangerous point state vector of the last frame are updated by Kalman filtering to determine the dangerous point speed of the target vehicle, wherein the dangerous point state vector includes the three-dimensional coordinates (x, y, z) of the dangerous point and the speed in x, y and z axis directions.

[0040] Optionally, the method further comprises;

[0041] based on the driving speed of the target vehicle and the driving speed of the ego vehicle, the relative position relationship between the target vehicle and the monocular camera installed on the ego vehicle is determined;

[0042] based on the relative position relationship, the driving strategy of the ego vehicle is adjusted.

[0043] In a second aspect, the embodiments of the present application provide a monocular camera, comprising:

[0044] a photosensitive component configured to collect a monocular image;

[0045] an image processor configured to perform the steps of the vehicle speed measurement method as described above.

[0046] The present application can effectively improve the distance measurement and perception stability of the monocular camera in a complex environment by performing image processing on the monocular image to obtain the center point coordinates of the target vehicle and the three-dimensional coordinates of the dangerous point, and by analyzing the visibility features of the target object as an auxiliary judgment basis for the position of the dangerous point. The center point speed and the dangerous point speed of the target vehicle are obtained by using Kalman filtering, which can focus on a specific position point in the target vehicle that is most critical for safety evaluation, and can more accurately evaluate the relative position relationship and potential risk between the target vehicle and the ego vehicle. In the case of real-time changes in the traffic scene, the heading angle is used as a dynamic weight to adjust the weight distribution between the center point speed and the dangerous point speed, the dynamic complement of the two different source speeds is realized through the geometric prior of the heading angle, the dangerous point speed and the target center point speed information are adaptively fused to obtain the final speed of the target vehicle, the center point speed jitter problem of the target vehicle caused by occlusion and other reasons is suppressed, the geometric advantages (such as good angle stability) are retained, and the robustness and scene adaptability of vehicle speed estimation are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is a first flowchart of the vehicle speed measurement method provided by the embodiments of the present application;

[0049] Figure 2 is a second flowchart of the vehicle speed measurement method provided by the embodiments of the present application;

[0050] Figure 3 is a third flowchart of the vehicle speed measurement method provided by the embodiments of the present application;

[0051] Figure 4 is a fourth flowchart of the vehicle speed measurement method provided by the embodiments of the present application;

[0052] Figure 5 is a fifth flowchart of the vehicle speed measurement method provided by the embodiments of the present application;

[0053] Figure 6 is a system block diagram of a monocular camera provided by an embodiment of the present application;

[0054] Figure 7 is a system block diagram of a vehicle provided by an embodiment of the present application;

[0055] Figure 8 is a system block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] The present application will be described in detail below with reference to specific embodiments shown in the drawings, but these embodiments do not limit the present application, and the structural, method, or functional changes made by those of ordinary skill in the art based on these embodiments are included in the protection scope of the present application.

[0057] Referring to Figure 1 The embodiment of the present application provides a vehicle speed measurement method, which comprises steps S101-S105.

[0058] S101, feature extraction is performed on a current frame monocular image to obtain a multi-scale feature map of the current frame monocular image.

[0059] The monocular image refers to a two-dimensional image obtained by a single camera or a single camera. Exemplarily, the monocular image can be an image collected by a monocular camera.

[0060] Exemplarily, a CNN network is used to extract deep feature representation in the input current frame monocular image, and output a multi-scale feature map of the current frame monocular image. For example, the network architecture of the CNN network can adopt a network structure such as ResNet, DarkNet19, and the like. The input current frame monocular image is subjected to multiple convolution operations and down-sampling operations to obtain a plurality of first intermediate feature maps of different scales. The intermediate feature map of the smallest scale is subjected to convolution operation and up-sampling operation to obtain a plurality of second intermediate feature maps of different scales. The first intermediate feature map and the second intermediate feature map having the same scale are fused to obtain the multi-scale feature map of the current frame monocular image, so as to obtain the feature representation at multiple scales. The multi-scale feature map fuses the context relationship of semantic information and spatial information of different scales.

[0061] S102, three-dimensional image feature prediction is performed on the multi-scale feature map to obtain key attributes of at least one target vehicle in the current frame monocular image.

[0062] The key attributes include 3D dimensions, three-dimensional coordinates of the center point, a depth value of the center point, and a heading angle. The 3D dimensions refer to the length L, the width W, and the height H of the target vehicle. The three-dimensional coordinates of the center point refer to the three-dimensional coordinates (x, y, z) of the center point of the target vehicle in the world coordinate system. The depth value of the center point of the target vehicle can be understood as the z value. There can be multiple target vehicles in the current frame monocular image, and the key attributes of each target vehicle are obtained.

[0063] Exemplarily, a three-dimensional detection network such as a 3D prediction head (3D Head) or a CenterNet network can be used to process the input multi-scale feature map to obtain the 3D dimensions, the three-dimensional coordinates of the center point, the depth value of the center point, and the heading angle of the target vehicle.

[0064] In S103, for each target vehicle, the coordinate position of the dangerous point of the target vehicle is predicted based on the multi-scale feature map and the key attributes, and the three-dimensional coordinates of the dangerous point are obtained.

[0065] The dangerous point is the midpoint of the bottom side of the detection bounding box of the target vehicle. The dangerous point can be understood as the boundary region of the target vehicle close to the ego vehicle, which contains more information about the proximity and potential collision risk of the target vehicle. Therefore, in this embodiment, by predicting the coordinate position of the dangerous point of the target vehicle based on the multi-scale feature map and the key attributes, the three-dimensional coordinates of the dangerous point are obtained, which can focus on a specific position point that is most critical for safety evaluation in the target vehicle.

[0066] In S104, the three-dimensional coordinates of the center point and the three-dimensional coordinates of the dangerous point are processed respectively to determine the center point speed and the dangerous point speed of the target vehicle.

[0067] Exemplarily, the three-dimensional coordinates of each center point are subjected to Kalman filtering processing to obtain a smooth and accurate center point speed of the target vehicle, and the three-dimensional coordinates of each dangerous point are subjected to Kalman filtering processing to obtain a smooth and accurate dangerous point speed of the target vehicle.

[0068] In S105, the weight distribution between the center point speed and the dangerous point speed is dynamically adjusted based on the heading angle, and the center point speed and the dangerous point speed are weighted and fused based on the result of the weight distribution to obtain the driving speed of the target vehicle.

[0069] The heading angle is used as a dynamic weight to adjust the weight distribution between the center point speed and the dangerous point speed under different conditions, and the center point speed and the dangerous point speed are weighted and fused to obtain the final driving speed of the target vehicle.

[0070] In this embodiment, the monocular image is processed to obtain the three-dimensional feature attributes of the target vehicle in the image, and the three-dimensional coordinates of the center point of the target vehicle and the dangerous point are determined based on the three-dimensional feature attributes, without relying on accurate 3D reconstruction or complex scene prior knowledge. Instead, the visibility features of the target object are analyzed as auxiliary judgment basis for the position of the dangerous point, which can effectively improve the stability of the monocular camera in complex environments. The center point speed and the dangerous point speed of the target vehicle are obtained by Kalman filtering, and the speed of the target vehicle is evaluated based on the three-dimensional coordinates of the dangerous point. This can focus on the specific position point in the target vehicle that is most critical for safety evaluation, and can more accurately evaluate the relative position relationship and potential risk between the target vehicle and the ego vehicle. In the case of real-time changes in the scene, the heading angle is used as a dynamic weight to adjust the weight distribution between the center point speed and the dangerous point speed. Based on the effectiveness of the dangerous point and the center point of the target vehicle in different scene conditions, the weight proportion of the center point speed and the dangerous point speed in the final speed estimation is adjusted, and the dangerous point speed and the target center point speed information are adaptively fused to obtain the final speed of the target vehicle. This avoids the limitations of a single information source in specific scenarios (such as following or crossing driving scenarios), thereby obtaining a more accurate and robust speed of the target vehicle.

[0071] One embodiment of the present application, as shown in Figure 2 The three-dimensional image feature prediction is performed on the multi-scale feature map to obtain the key attributes of at least one target vehicle in the current frame monocular image, including:

[0072] S201, input the multi-scale feature map into the heat map prediction branch of the three-dimensional detection network for class prediction to obtain a class heat map, and determine at least one target center point position based on the class heat map;

[0073] S202, input the multi-scale feature map into the offset prediction branch of the three-dimensional detection network for processing to obtain the center point offset of each spatial position in the multi-scale feature map;

[0074] S203, based on the coordinate prediction branch in the three-dimensional detection network, the three-dimensional coordinates of the center point of the target vehicle are obtained from the multi-scale feature map through the index of the target center point position on the class heat map and the center point offset corresponding to the target center point position;

[0075] S204, based on the size prediction branch in the three-dimensional detection network, the 3D size of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the class heat map;

[0076] S205, based on the depth prediction branch in the three-dimensional detection network, the depth value of the center point of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the class heat map.

[0077] S206, based on the heading angle prediction branch in the three-dimensional detection network, obtaining the heading angle of the target vehicle from the multi-scale feature map through the index of the target center point position on the category heat map.

[0078] The three-dimensional detection network includes a heat map prediction branch, an offset prediction branch, a size prediction branch, a depth prediction branch, a coordinate prediction branch, and a heading angle prediction branch.

[0079] The multi-scale feature map is input into the heat map prediction branch for category prediction, and the outputs of all positions are compressed to 0-1 through a sigmoid activation function, so as to obtain a category heat map. The category heat map represents the probability of each spatial position in the multi-scale feature map belonging to the target center point. The category heat map is subjected to local maximum suppression processing to obtain multiple local maximum points of the category heat map. The Top-K target center points with the highest probability are selected from the obtained multiple local maximum points. For each Top-K target center point, the coordinate position of the target center point is obtained, which indicates the center point position of the target in the feature map coordinate system.

[0080] The multi-scale feature map is input into the offset prediction branch in the three-dimensional detection network for processing to obtain the center point offset of each spatial position in the multi-scale feature map. Through the offset prediction branch, the grid misplacement error caused by down-sampling in the category heat map processing is compensated, the target center point coordinate error predicted on the integer point (integer grid) in the category heat map is corrected, and the sub-pixel level positioning accuracy is improved.

[0081] Based on the coordinate prediction branch in the three-dimensional detection network, the center point offset corresponding to the target center point position is obtained. The three-dimensional coordinates of the target vehicle center point are obtained by adding the coordinate position of the target center point and the corresponding center point offset, and then multiplying the down-sampling ratio. The down-sampling ratio can be understood as how many times the original pixel is reduced in the output feature map. Through the down-sampling ratio, the position of the target center point in the feature map coordinate system is mapped to the position of the current frame monocular image, i.e. the three-dimensional coordinates in the world coordinate system.

[0082] Based on the size prediction branch in the three-dimensional detection network, the 3D scale of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the category heat map, and the 3D scale of the target vehicle includes the length, width and height of the target vehicle. The size prediction branch is used to predict the geometric size of the target vehicle in three-dimensional space. For example, the scale prediction branch predicts a scale feature map with the same spatial resolution as the category heat map, wherein each feature point corresponds to an independent scale feature representing length, width and height. The scale feature at the corresponding position is extracted from the scale feature map through the index of the target center point position on the category heat map, and the extracted scale feature is decoded into the 3D scale of the target vehicle, i.e. length, width and height, according to a preset scale regression decoding algorithm. For example, the scale regression decoding algorithm includes but is not limited to direct regression, residual regression and normalized regression.

[0083] Based on the depth prediction branch in the three-dimensional detection network, the corresponding depth value is obtained based on the target center point position, so as to obtain the depth value of the center point of the target vehicle. For example, the depth prediction branch outputs the probability that the target center point position belongs to each preset depth discrete value, which represents the probability that the target center point position is located at different depths. The corresponding depth discrete value is multiplied by each probability and then summed to obtain the depth value corresponding to the target center point position, so as to obtain the depth value of the center point of the target vehicle. For example, the depth discrete value can be in the range of 0-200m.

[0084] Based on the heading angle prediction branch in the three-dimensional detection network, the corresponding heading angle is obtained based on the target center point position on the category heat map, so as to obtain the heading angle of the target vehicle. For example, the feature dimension corresponding to the heading angle is 2D, representing the sin value and the cos value of the predicted heading angle respectively. The heading angle prediction branch outputs the sin value and the cos value of the heading angle, and the tan value is obtained by dividing the sin value by the cos value, and the heading angle is obtained by the inverse trigonometric function.

[0085] In this embodiment, the multi-branch of the three-dimensional detection network is used to predict the three-dimensional feature of the input multi-scale feature map, so as to obtain the three-dimensional feature attribute information of the target vehicle.

[0086] In one embodiment of the present application, when training the model of the three-dimensional detection network, the model training of the heat map prediction branch is performed, the error between the predicted heat map of the sample image and the true value heat map is taken as the loss error, and the Focal Loss loss function is used to weight and correct the loss error, so as to adjust the parameter adjustment of the heat map prediction branch, so as to alleviate the class imbalance problem of the sample. The true value heat map can be generated by drawing a Gaussian kernel at the target center point.

[0087] In an embodiment of the present application, when training the model of the offset prediction branch, the size prediction branch, the depth prediction branch, the coordinate prediction branch and the heading angle prediction branch of the three-dimensional detection network, the scale, the three-dimensional coordinates, the depth value and the heading angle of the target are obtained by predicting the sample image through each branch. A 3D bounding box of the target is constructed based on the predicted scale, three-dimensional coordinates, depth value and heading angle. The 3D bounding box is composed of eight corner points in three-dimensional space, so the 3D coordinates of the eight corner points can be obtained. When constructing the training set of the model, the scale sample, the three-dimensional coordinate sample, the depth value sample and the heading angle sample are used as sample elements of the training set. The sample data is combined in a combination mode in which the value of one sample element changes and the values of other sample elements remain unchanged, so as to construct multiple training sets. Each training set is trained respectively. For each training set, the pre-trained three-dimensional detection network is used to train the training set, the sample element with changing value is predicted to obtain the predicted value corresponding to the sample element, and the predicted eight corner point coordinates are constructed based on the predicted value and the values of the other sample elements with unchanged values. The error between the predicted eight corner point coordinates and the obtained 3D coordinates of the eight corner points is used as the loss, and the parameters of the model are adjusted to obtain the trained three-dimensional detection network.

[0088] For example, a first training set is constructed. The values of each scale sample in the first training set are different, and each three-dimensional coordinate sample, each depth value sample and each heading angle sample has a fixed value. The model is trained on the first training set. The three-dimensional detection network outputs each predicted scale value. Based on each predicted scale value and the fixed values of the three-dimensional coordinate sample, the depth value sample and the heading angle sample, the 3D coordinates of each predicted eight corner point are determined. The error between the predicted eight corner point coordinates and the obtained 3D coordinates of the eight corner points is used as the loss, and the parameters of the model are adjusted. Similarly, other training sets can be obtained.

[0089] The present embodiment constructs multiple training sets by decoupling the coupling relationship between different attributes (scale, three-dimensional coordinates, depth value and heading angle), i.e., one attribute changes and the other attributes remain unchanged, and introduces the eight corner points of the 3D bounding box for geometric constraint auxiliary training. The regression error of the model can be directly discarded, the learning difficulty of the model is reduced, the convergence of the model is accelerated, and the training stability of the model is improved.

[0090] The coordinate position of the dangerous point of the target vehicle is predicted based on the multi-scale feature map and the key attribute, and three-dimensional coordinates of the dangerous point are obtained. For the implementation of obtaining the three-dimensional coordinates of the dangerous point, the application provides three implementation manners of decoupled optimization strategies. According to the needs of different business scenarios, the calculation complexity and the accuracy requirements, etc., the corresponding implementation manner can be selected, so it has high flexibility, and different dangerous point strategies can be flexibly adopted according to different scene requirements. The flexibility not only enhances the practicability and adaptability of the system, but also reduces the adjustment cost when deploying in different scenes, so that the system can quickly adapt to various actual application environments. The following Figures 3-5 The three implementation manners are described in detail.

[0091] An embodiment of the application, as shown in Figure 3 The coordinate position of the dangerous point of the target vehicle is predicted based on the multi-scale feature map and the key attribute, and three-dimensional coordinates of the dangerous point are obtained. For the implementation of obtaining the three-dimensional coordinates of the dangerous point, the application provides three implementation manners of decoupled optimization strategies. According to the needs of different business scenarios, the calculation complexity and the accuracy requirements, etc., the corresponding implementation manner can be selected, so it has high flexibility, and different dangerous point strategies can be flexibly adopted according to different scene requirements. The flexibility not only enhances the practicability and adaptability of the system, but also reduces the adjustment cost when deploying in different scenes, so that the system can quickly adapt to various actual application environments. The following

[0092] S301, inputting the multi-scale feature map and the three-dimensional coordinates of the center point of the target vehicle into the offset prediction head in the offset depth fusion network, outputting the position offset of the dangerous point of the target vehicle relative to the center point on the current frame monocular image plane, and determining the x-axis coordinate and y-axis coordinate of the dangerous point in the camera coordinate system based on the position offset;

[0093] S302, inputting the multi-scale feature map and the depth value of the center point of the target vehicle into the depth estimation network in the offset depth fusion network, obtaining the discrete representation of the depth of each pixel point in the multi-scale feature map, and further obtaining the depth value of the dangerous point in the camera coordinate system;

[0094] S303, based on the x-axis coordinate, y-axis coordinate and depth value of the dangerous point in the camera coordinate system, and the camera calibration parameter, the three-dimensional coordinates of the dangerous point in the world coordinate system are calculated by the back projection function.

[0095] The position offset includes the offset in the x-axis direction and the offset in the y-axis direction.

[0096] Exemplarily, the offset depth fusion network includes the offset prediction head and the depth estimation network, and the position offset of the dangerous point of the target vehicle relative to the center point on the image plane and the depth value of the dangerous point can be obtained through the offset depth fusion network.

[0097] Exemplarily, the offset prediction head can be a 1*1 convolutional layer, inputting the multi-scale feature map and the three-dimensional coordinates of the center point of the target vehicle into the offset prediction head, and outputting a two-channel feature map, which respectively represents the offset amount of the output danger point in the x-axis direction and the offset amount in the y-axis direction relative to the center point on the monocular image plane of the current frame. The x-axis coordinate and the y-axis coordinate of the danger point in the camera coordinate system are determined based on the position offset amount. The x-axis coordinate of the danger point in the camera coordinate system is represented as follows: and the y-axis coordinate of the danger point in the camera coordinate system is represented as follows:

[0098] ;

[0099] ;

[0100] wherein, x represents the x-axis coordinate of the center point, y represents the y-axis coordinate of the center point, x_offset represents the offset amount in the x-axis direction relative to the center point, and y_offset represents the offset amount in the y-axis direction relative to the center point.

[0101] Exemplarily, the depth estimation network can adopt the depth estimation network in the CenterNet network mechanism. The multi-scale feature map and the depth value of the center point of the target vehicle are input into the depth estimation network, the image features and the depth features of the center point are fused, the discrete representation of the depth of each pixel point in the multi-scale feature map is obtained, the depth discrete feature representation of the corresponding danger point is obtained, and the depth value of the danger point is decoded through the index of the center point.

[0102] wherein, obtaining the discrete representation of the depth of each pixel point in the multi-scale feature map comprises: for each pixel point in the multi-scale feature map, obtaining the depth value corresponding to the pixel point by multiplying each probability with the corresponding depth discrete value and then summing after predicting the probability that the pixel point belongs to each depth discrete value through the depth estimation network; and taking the coordinates of the center point as the index, obtaining the depth value of the danger point in the camera coordinate system based on the depth value corresponding to each pixel point. The corresponding multiple depth discrete values can be obtained by uniformly discretizing the depth range.

[0103] Exemplarily, the depth of the danger point The calculation formula can be represented as:

[0104] ;

[0105] wherein, x_k represents the kth depth discrete value, and p_k represents the probability that the depth value of the danger point belongs to the kth depth discrete value.

[0106] ​​​An x-axis coordinate of the dangerous point in a camera coordinate system , a y-axis coordinate , and a depth value , and according to known camera calibration parameters, the image coordinates, depth and camera calibration parameters of the dangerous point in the camera coordinate system are combined to calculate the accurate position of the dangerous point in the world coordinate system through strict geometric transformation.

[0107] Exemplarily, the three-dimensional coordinates (x, y, z) of the dangerous point in the world coordinate system are represented as:

[0108] ;

[0109] wherein, represents a back projection function, represents a camera calibration parameter.

[0110] In this embodiment, the three-dimensional coordinates and depth information of the center point of the target vehicle obtained above are used to fuse the multi-scale features extracted from the image, to predict the translation amount and depth value of the dangerous point relative to the center point on the image plane through the offset depth fusion network, and to calculate the accurate position of the dangerous point in the world coordinate system through strict geometric transformation, so as to fully consider the influence factors such as camera distortion, internal and external parameters, and to ensure the accuracy of the spatial positioning of the dangerous point of the target vehicle.

[0111] One embodiment of the present application, as shown in Figure 4 , predicts the coordinate position of the dangerous point of the target vehicle based on the multi-scale feature map and the key attribute, to obtain the three-dimensional coordinates of the dangerous point, including:

[0112] S401, inputting the multi-scale feature map into the dangerous point heat map prediction branch in the dangerous point prediction network to obtain the dangerous point heat map, and determining the 2D image coordinates of the dangerous point based on the dangerous point heat map;

[0113] S402, based on the dangerous point depth prediction branch in the dangerous point prediction network, the depth value of the dangerous point is obtained from the multi-scale feature map through the 2D image coordinate index of the dangerous point in the dangerous point heat map;

[0114] S403, based on the 2D image coordinates and the depth value of the dangerous point, the three-dimensional coordinates of the dangerous point are determined.

[0115] Exemplarily, the dangerous point prediction network can be a lightweight network, and its structure is similar to the architecture of the CenterNet network. The dangerous point prediction network includes a dangerous point heat map prediction branch and a dangerous point depth prediction branch.

[0116] Exemplarily, the multi-scale feature map is input into a dangerous point heat map prediction branch for category prediction, and the outputs of all positions are compressed to 0-1 through a sigmoid activation function, so as to obtain a dangerous point heat map. The dangerous point heat map represents the probability that each spatial position in the multi-scale feature map belongs to a dangerous point. The dangerous point heat map is subjected to local maximum suppression processing to obtain a plurality of local maximum points of the dangerous point heat map. The position of a target dangerous point is selected from the plurality of local maximum points, and the 2D image coordinates of the dangerous point are determined based on the position of the target dangerous point.

[0117] For each pixel point in the multi-scale feature map, the probability that the pixel point belongs to each preset depth discrete value is predicted through the dangerous point depth prediction branch. The depth value corresponding to the pixel point is obtained by multiplying each probability with the corresponding depth discrete value and then summing. The position coordinates of the dangerous point are determined based on the 2D image coordinates of the dangerous point, and the depth value of the dangerous point is determined based on the depth values of the pixel points that have been obtained. The plurality of depth discrete values can be obtained by uniformly discretizing the depth range. For example, the discretized value range of the depth can be 0-200 m. The three-dimensional coordinates of the dangerous point are determined based on the 2D image coordinates and the depth value of the dangerous point.

[0118] In an embodiment of the present application, a dangerous point prediction network is added after the CNN network. During the model training of the CNN network and the dangerous point prediction network, the dangerous point prediction network can share the weight parameters with the CNN network, and the parameters of the dangerous point prediction network are completely independent, so as to ensure that the dangerous point coordinates predicted by the dangerous point prediction network are not disturbed by attributes such as category / heading angle, thereby enhancing the robustness of the model to the key safety position (dangerous point) of the target vehicle.

[0119] In the embodiment, a dangerous point prediction network is added after the CNN network. The dangerous point prediction network and the three-dimensional detection network are a parallel branch. Based on the multi-scale features output by the CNN network, the 2D image coordinates and the depth value of the dangerous point can be independently predicted, so as to ensure that the dangerous point coordinates are not disturbed by attributes such as category / heading angle, thereby enhancing the robustness of the model to the key safety position (dangerous point) of the target vehicle.

[0120] As shown in FIG. 1, in an embodiment of the present application, the dangerous point prediction network is added after the CNN network. The dangerous point prediction network and the three-dimensional detection network are a parallel branch. Based on the multi-scale features output by the CNN network, the 2D image coordinates and the depth value of the dangerous point can be independently predicted, so as to ensure that the dangerous point coordinates are not disturbed by attributes such as category / heading angle, thereby enhancing the robustness of the model to the key safety position (dangerous point) of the target vehicle. Figure 5 As shown in FIG. 1, in an embodiment of the present application, the dangerous point prediction network is added after the CNN network. The dangerous point prediction network and the three-dimensional detection network are a parallel branch. Based on the multi-scale features output by the CNN network, the 2D image coordinates and the depth value of the dangerous point can be independently predicted, so as to ensure that the dangerous point coordinates are not disturbed by attributes such as category / heading angle, thereby enhancing the robustness of the model to the key safety position (dangerous point) of the target vehicle.

[0121] S501, determining the length of the target vehicle based on the 3D scale of the target vehicle, wherein the 3D scale of the target vehicle includes the length, width and height of the target vehicle;

[0122] S502, based on the three-dimensional coordinates of the center point of the target vehicle, the three-dimensional coordinates of the dangerous point are calculated by using the geometric relationship between the length of the target vehicle and the heading angle.

[0123] In most traffic scenarios, vehicle collisions mainly occur in the length direction, and the width direction has relatively small influence. Therefore, in a scenario with limited computing resources, the three-dimensional coordinates of the dangerous point are directly approximated by using the three-dimensional coordinates of the center point obtained by geometric relationship, so as to simplify the model and improve the computing efficiency.

[0124] Exemplarily, the x-axis coordinate and the z-axis coordinate of the center point of the target vehicle are determined by re-projection based on the three-dimensional coordinates of the center point of the target vehicle; the x-axis coordinate and the z-axis coordinate of the dangerous point of the target vehicle are calculated by the following formula based on the heading angle of the target vehicle, the x-axis coordinate and the z-axis coordinate of the center point and the z-axis coordinate of the center point of the target vehicle:

[0125] ;

[0126] ;

[0127] wherein, the z-axis coordinate of the center point of the target vehicle, the x-axis coordinate of the center point of the target vehicle, and the heading angle of the target vehicle.

[0128] In the embodiment, in a scenario with limited computing resources, the width of the target vehicle is ignored, the three-dimensional coordinates of the dangerous point are directly approximated by using the three-dimensional coordinates of the center point of the target vehicle obtained by geometric relationship, so as to simplify the model calculation without adding any additional parameters, the computing complexity is almost negligible, the spatial perception ability of the center point of the target vehicle to the dangerous point is implicitly enhanced, and the method is suitable for rapid zero-cost deployment to business scenarios.

[0129] In an embodiment of the present application, the three-dimensional coordinates of the center point and the three-dimensional coordinates of the dangerous point are processed respectively to determine the center point speed and the dangerous point speed of the target vehicle, including: the state of the center point of the current frame and the historical center point state vector of the last frame are updated by Kalman filtering to determine the center point speed of the target vehicle, wherein the center point state vector includes the three-dimensional coordinates (x, y, z) of the center point and the speed in the x, y and z axis directions; the state of the dangerous point of the current frame and the historical dangerous point state vector of the last frame are updated by Kalman filtering to determine the dangerous point speed of the target vehicle, wherein the dangerous point state vector includes the three-dimensional coordinates (x, y, z) of the dangerous point and the speed in the x, y and z axis directions.

[0130] The embodiment fuses the three-dimensional coordinates of the center point in time sequence and the three-dimensional coordinates of the dangerous point in time slot by Kalman filtering. The Kalman filtering fuses the historical trajectory and the current detection result adaptively through a "prediction-update" cycle, suppresses single-frame noise, directly extracts the speed component from the optimized state vector, and obtains the smooth and accurate motion speed of the center point and the dangerous point of the visual vehicle. This method does not require additional parameters, and high-precision speed estimation can be achieved only by using geometric position information, which significantly improves the stability and trajectory prediction ability of target tracking.

[0131] The processing of Kalman filtering is described by taking the center point as an example, and the processing mode of the dangerous point is similar, which will not be described here. The specific steps are as follows:

[0132] 1. State vector initialization: constructing a center point state vector, which includes the three-dimensional coordinates (x, y, z) of the center point and the speed of the vehicle in the x, y, and z axis directions;

[0133] 2. Prediction cycle: including state prediction and covariance prediction. The state prediction is based on the uniform motion model, and the state transition matrix is used to predict the state at the next time. The state transition matrix considers the time interval and can extrapolate the current state to the next time. Covariance prediction: simultaneously calculate the uncertainty of the state, i.e. the reliability of the predicted state. This process includes the uncertainty propagation of the last time and the addition of process noise to reflect the difference between the motion model and the actual motion;

[0134] 3. Update cycle: when the three-dimensional coordinates of the center point of the new frame (current observation result) are obtained, the prediction result and the observation result are fused to obtain the corrected state. The residual calculation is to compare the observation result of the current frame with the predicted state, and calculate the difference between them. The Kalman gain calculation is to calculate a fusion weight according to the uncertainty of the predicted state and the uncertainty of the current observation. When the detection result is reliable, the weight is larger, and the current detection is more dependent. When the detection result noise is larger, the weight is smaller, and the predicted value is more dependent. The state prediction and covariance prediction are updated by residual calculation and Kalman gain calculation: the predicted state and the current observation result are weighted and fused by using the calculated fusion weight, the state is updated, and the optimized state estimation is obtained; at the same time, the covariance is updated to update the uncertainty estimation of the state, reflecting the reliability of the current state.

[0135] In an embodiment of the present application, the weight distribution between the center point speed and the dangerous point speed is dynamically adjusted based on the heading angle, and the center point speed and the dangerous point speed are weighted and fused based on the result of the weight distribution to obtain the driving speed of the target vehicle, which includes:

[0136] Driving speed of the target vehicle is obtained by the following formula:

[0137]

[0138] wherein, represents the heading angle, represents the dangerous point speed, represents the center point speed.

[0139] In the above formula, the visibility of the center point of the target vehicle is represented by the heading angle, for example, when θ is 0, the rear of the target vehicle is directly opposite the monocular camera on the ego vehicle, which means that the center point of the target vehicle is invisible, so when the weight distribution of the driving speed of the target vehicle is calculated, the weight tends to the dangerous point speed.

[0140] In the actual traffic application scenario, in the case that the center point of the target vehicle is blocked, causing the predicted vehicle speed to be unstable, for example, the predicted oscillation caused by the invisible center point in the following vehicle scene, the dynamic weighting mechanism of the heading angle is used to weight and fuse the geometric complete center point speed and the visually robust θ dangerous point speed, and the dynamic complement of the speeds from two different sources is realized through the geometric prior of the heading angle. While suppressing the center point speed jitter of the target vehicle caused by blocking and other reasons, the geometric advantages (such as good angle stability) are retained, and the robustness and scene adaptability of the vehicle speed estimation are significantly improved.

[0141] In one embodiment of the present application, the method further comprises: determining the relative position relationship between the target vehicle and the monocular camera mounted on the ego vehicle based on the driving speed of the target vehicle and the driving speed of the ego vehicle; and adjusting the driving strategy of the ego vehicle based on the relative position relationship.

[0142] As shown in Figure 6 The embodiment of the present application provides a monocular camera, which comprises:

[0143] The photosensitive component 601 is configured to collect a monocular image.

[0144] The image processor 602 is configured to execute the steps of the vehicle speed measurement method.

[0145] Optionally, the image processor 602 specifically executes:

[0146] feature extraction on the current frame monocular image to obtain a multi-scale feature map of the current frame monocular image;

[0147] three-dimensional feature prediction on the multi-scale feature map to obtain key attributes of at least one target vehicle in the current frame monocular image, wherein the key attributes include 3D scale, three-dimensional coordinates of the center point, depth value of the center point, and heading angle; ​

[0148] for each target vehicle,

[0149] predicting a coordinate position of a dangerous point of the target vehicle based on the multi-scale feature map and the key attribute, to obtain a three-dimensional coordinate of the dangerous point, wherein the dangerous point is a bottom edge midpoint of a detection bounding box of the target vehicle;

[0150] respectively processing the three-dimensional coordinate of the center point and the three-dimensional coordinate of the dangerous point to determine a center point speed and a dangerous point speed of the target vehicle;

[0151] based on the heading angle, dynamically adjusting the weight distribution between the center point speed and the dangerous point speed, and based on a result of the weight distribution, weighting and fusing the center point speed and the dangerous point speed to obtain a driving speed of the target vehicle.

[0152] Optionally, the image processor 602 specifically performs:

[0153] inputting the multi-scale feature map and the three-dimensional coordinate of the center point into an offset prediction head in the offset depth fusion network to output a position offset of the dangerous point relative to the center point on a current frame monocular image plane, and based on the position offset, determining an x-axis coordinate and a y-axis coordinate of the dangerous point in a camera coordinate system, wherein the position offset includes an offset in the x-axis direction and an offset in the y-axis direction;

[0154] inputting the multi-scale feature map and the depth value of the center point into a depth estimation network in the offset depth fusion network to obtain a discrete representation of the depth of each pixel point in the multi-scale feature map, and further obtain a depth value of the dangerous point in the camera coordinate system;

[0155] based on the x-axis coordinate, the y-axis coordinate and the depth value of the dangerous point in the camera coordinate system, and camera calibration parameters, calculating a three-dimensional coordinate of the dangerous point in a world coordinate system through a back projection function.

[0156] Optionally, the image processor 602 specifically performs:

[0157] for each pixel point in the multi-scale feature map, predicting a probability that the pixel point belongs to each preset depth discrete value through the depth estimation network, and based on summing each probability multiplied by the corresponding depth discrete value, obtaining a depth value corresponding to the pixel point.

[0158] Optionally, the image processor 602 specifically performs:

[0159] inputting the multi-scale feature map into a dangerous point heat map prediction branch in the dangerous point prediction network to obtain a dangerous point heat map, and based on the dangerous point heat map, determining a 2D image coordinate of the dangerous point, wherein the dangerous point heat map is used to represent a probability that each spatial position in the multi-scale feature map belongs to the dangerous point;

[0160] Based on the dangerous point depth prediction branch in the dangerous point prediction network, the depth value of the dangerous point is obtained from the multi-scale feature map through the 2D image coordinate index of the dangerous point in the dangerous point heat map;

[0161] Based on the 2D image coordinate and the depth value of the dangerous point, the three-dimensional coordinate of the dangerous point is determined.

[0162] Optionally, the image processor 602 specifically performs:

[0163] Based on the 3D scale of the target vehicle, the length of the target vehicle is determined, wherein the 3D scale of the target vehicle includes the length, width and height of the target vehicle;

[0164] Based on the three-dimensional coordinate of the center point of the target vehicle, the three-dimensional coordinate of the dangerous point is calculated using the geometric relationship between the length of the target vehicle and the heading angle.

[0165] Optionally, the image processor 602 specifically performs:

[0166] The driving speed of the target vehicle It is obtained by the following formula:

[0167] ;

[0168] Wherein, The heading angle is represented by, The dangerous point speed is represented by, The center point speed is represented by.

[0169] Optionally, the image processor 602 specifically performs:

[0170] The multi-scale feature map is input into the heat map prediction branch in the three-dimensional detection network for class prediction to obtain a class heat map, and at least one target center point position is determined based on the class heat map, wherein the class heat map represents the probability that each spatial position in the multi-scale feature map belongs to the target center point;

[0171] The multi-scale feature map is input into the offset prediction branch in the three-dimensional detection network for processing to obtain the center point offset of each spatial position in the multi-scale feature map;

[0172] Based on the coordinate prediction branch in the three-dimensional detection network, the three-dimensional coordinate of the center point of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the class heat map and the center point offset corresponding to the target center point position;

[0173] Based on the size prediction branch in the three-dimensional detection network, the 3D scale of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the category heat map, wherein the 3D scale of the target vehicle includes the length, width and height of the target vehicle.

[0174] Based on the depth prediction branch in the three-dimensional detection network, the depth value of the center point of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the category heat map.

[0175] Based on the heading angle prediction branch in the three-dimensional detection network, the heading angle of the target vehicle is obtained from the multi-scale feature map through the index of the target center point position on the category heat map.

[0176] Optionally, the image processor 602 specifically performs:

[0177] The three-dimensional coordinates of the center point of the current frame and the historical center point state vector of the last frame are updated by Kalman filtering to determine the center point speed of the target vehicle, wherein the center point state vector includes the three-dimensional coordinates (x, y, z) of the center point and the speeds in the x, y and z axis directions.

[0178] The three-dimensional coordinates of the dangerous point of the current frame and the historical dangerous point state vector of the last frame are updated by Kalman filtering to determine the dangerous point speed of the target vehicle, wherein the dangerous point state vector includes the three-dimensional coordinates (x, y, z) of the dangerous point and the speeds in the x, y and z axis directions.

[0179] Optionally, the image processor 602 specifically performs:

[0180] Based on the driving speed of the target vehicle and the driving speed of the ego vehicle, the relative position relationship between the target vehicle and the monocular camera installed on the ego vehicle is determined.

[0181] Based on the relative position relationship, the driving strategy of the ego vehicle is adjusted.

[0182] As shown in Figure 7 The embodiment of the present application provides a vehicle, which comprises:

[0183] A monocular camera 701 is configured to collect a monocular image.

[0184] A processor 702 is configured to execute the steps of the vehicle speed measurement method.

[0185] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the vehicle speed measurement methods.

[0186] Figure 8Fig. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. Figure 8 The electronic device shown includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, and the processor 801, the communication interface 802, and the memory 803 complete communication with each other through the communication bus 804. Among them, Figure 8 The connection mode between the processor 801, the communication interface 802, and the memory 803 shown is only exemplary, and in the implementation process, the processor 801, the communication interface 802, and the memory 803 can also be connected in communication with each other in other connection modes other than the communication bus 804.

[0187] The memory 803 can be used to store a computer program, and the computer program can include instructions and data, and implement the steps of any one of the vehicle speed measurement methods described above. In the embodiments of the present application, the memory 803 can be various types of storage media, such as random access memory (RAM), read only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, and registers, etc. The memory 803 can include a hard disk and / or an internal memory.

[0188] The processor 801 can be a general-purpose processor, which can be a processor that reads and executes a computer program (such as a computer program) stored in a memory (such as the memory 803) to perform specific steps and / or operations, and the general-purpose processor can use data stored in the memory (such as the memory 803) in the process of executing the steps and / or operations. The general-purpose processor can be, for example but not limited to, a central processing unit (CPU). In addition, the processor 801 can also be a special-purpose processor, which can be a processor specially designed to perform specific steps and / or operations, and the special-purpose processor can be, for example but not limited to, ASIC and FPGA, etc. In addition, the processor 801 can also be a combination of multiple processors, such as a multi-core processor.

[0189] The communication interface 802 can include an input / output (I / O) interface, a physical interface, and a logical interface, etc. for realizing interconnection of devices inside the network device, and an interface for realizing interconnection of the network device and other devices (e.g. network devices). The communication network can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 802 can be a module, a circuit, a transceiver, or any device capable of realizing communication.

[0190] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor 801 or instructions in the form of software. The method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution completion, or executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art, such as random access memory flash, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory 803, and the processor 801 reads the information in the memory 803, and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0191] It should be understood that the term "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after are an "or" relationship, but can also represent an "and / or" relationship, which can be understood in combination with the context.

[0192] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0193] It is to be noted that, as used in this document, the terminology "first", "second", etc. is merely used to differentiate one entity or action from another, and does not necessarily imply or require any actual relationship or order between or among the entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0194] Each of the embodiments in the present specification is described in a related manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments.

[0195] The preferred embodiments of the present application are described above in detail. The above description is only for the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for measuring vehicle speed, characterized in that, The method includes: Feature extraction is performed on the current frame monocular image to obtain the multi-scale feature map of the current frame monocular image; Three-dimensional feature prediction is performed on the multi-scale feature map to obtain key attributes of at least one target vehicle in the current frame monocular image, wherein the key attributes include 3D scale, three-dimensional coordinates of the center point, depth value of the center point, and heading angle; For each target vehicle. Based on the multi-scale feature map and key attributes, the coordinates of the danger points of the target vehicle are predicted to obtain the three-dimensional coordinates of the danger points, wherein the danger points are the midpoints of the bottom edges of the detection bounding box of the target vehicle. The three-dimensional coordinates of the center point and the three-dimensional coordinates of the danger point are processed separately to determine the center point speed and the danger point speed of the target vehicle. The weight distribution between the center point speed and the danger point speed is dynamically adjusted based on the heading angle. The center point speed and the danger point speed are then weighted and fused based on the weight distribution result to obtain the driving speed of the target vehicle. Specifically, based on the multi-scale feature map and key attributes, the coordinates of the danger points of the target vehicle are predicted to obtain the three-dimensional coordinates of the danger points, including: The multi-scale feature map and the three-dimensional coordinates of the center point are input into the offset prediction head in the offset depth fusion network, and the position offset of the danger point relative to the center point on the monocular image plane of the current frame is output. Based on the position offset, the x-axis coordinates and y-axis coordinates of the danger point in the camera coordinate system are determined, wherein the position offset includes the offset in the x-axis direction and the offset in the y-axis direction. The depth values ​​of the multi-scale feature map and the center point are input into the depth estimation network in the offset depth fusion network to obtain a discretized representation of the depth of each pixel in the multi-scale feature map, and then the depth value of the danger point in the camera coordinate system is obtained. Based on the x-axis coordinates, y-axis coordinates, and depth value of the hazard point in the camera coordinate system, as well as the camera calibration parameters, the three-dimensional coordinates of the hazard point in the world coordinate system are calculated using the back projection function. or, Based on the multi-scale feature map and key attributes, the coordinates of the danger points of the target vehicle are predicted to obtain the three-dimensional coordinates of the danger points, including: The multi-scale feature map is input into the hazard heatmap prediction branch of the hazard prediction network to obtain the hazard heatmap. The 2D image coordinates of the hazard points are determined based on the hazard heatmap. The hazard heatmap is used to characterize the probability that each spatial location in the multi-scale feature map belongs to a hazard point. Based on the hazard depth prediction branch in the hazard prediction network, the depth value of the hazard point is obtained from the multi-scale feature map by using the 2D image coordinate index of the hazard point in the hazard heat map; Based on the 2D image coordinates and depth values ​​of the hazard point, the 3D coordinates of the hazard point are determined.

2. The vehicle speed measurement method as described in claim 1, characterized in that, The discretized representation of the depth of each pixel in the multi-scale feature map is obtained, including: For each pixel in the multi-scale feature map, the probability of the pixel belonging to each preset depth discrete value is predicted by the depth estimation network. The probability is multiplied by the corresponding depth discrete value and then summed to obtain the depth value corresponding to the pixel.

3. The vehicle speed measurement method as described in claim 1, characterized in that, Based on the multi-scale feature map and key attributes, the coordinates of the danger points of the target vehicle are predicted to obtain the three-dimensional coordinates of the danger points, including: The length of the target vehicle is determined based on the 3D dimensions of the target vehicle, wherein the 3D dimensions of the target vehicle include the length, width and height of the target vehicle; Based on the three-dimensional coordinates of the center point of the target vehicle, the three-dimensional coordinates of the danger point are calculated using the geometric relationship between the vehicle length and heading angle of the target vehicle.

4. The vehicle speed measurement method as described in claim 1, characterized in that, The weight allocation between the center point speed and the danger point speed is dynamically adjusted based on the heading angle. Based on the weight allocation result, the center point speed and the danger point speed are weighted and fused to obtain the target vehicle's travel speed, including: target vehicle speed Obtain it using the following formula: ； in, Characterizing the heading angle, Characterizing the velocity at the danger point, Characterizes the velocity at the center point.

5. The vehicle speed measurement method as described in claim 1, characterized in that, Perform 3D image feature prediction on the multi-scale feature map to obtain key attributes of at least one target vehicle in the current frame monocular image, including: The multi-scale feature map is input into the heatmap prediction branch of the 3D detection network for category prediction to obtain a category heatmap. Based on the category heatmap, at least one target center point location is determined, wherein the category heatmap represents the probability that each spatial location in the multi-scale feature map belongs to the target center point. The multi-scale feature map is input into the offset prediction branch of the 3D detection network for processing to obtain the center point offset of each spatial location in the multi-scale feature map. Based on the coordinate prediction branch in the 3D detection network, the 3D coordinates of the center point of the target vehicle are obtained from the multi-scale feature map by using the index of the target center point position on the category heatmap and the center point offset corresponding to the target center point position. Based on the size prediction branch in the 3D detection network, the 3D scale of the target vehicle is obtained from the multi-scale feature map by the index of the target center point position on the category heatmap, wherein the 3D scale of the target vehicle includes the length, width and height of the target vehicle. Based on the depth prediction branch in the 3D detection network, the depth value of the target vehicle is obtained from the multi-scale feature map by using the index of the target center point position on the category heatmap; Based on the heading angle prediction branch in the 3D detection network, the heading angle of the target vehicle is obtained from the multi-scale feature map by indexing the target center point position on the category heatmap.

6. The vehicle speed measurement method as described in claim 1, characterized in that, The three-dimensional coordinates of the center point and the danger point are processed separately to determine the center point velocity and danger point velocity of the target vehicle, including: Kalman filtering is used to update the state of the center point of the current frame using the three-dimensional coordinates and the historical center point state vector of the previous frame, thereby determining the center point velocity of the target vehicle. The center point state vector includes the three-dimensional coordinates (x, y, z) of the center point and the velocities in the three axes x, y, and z. Kalman filtering is used to update the state of the hazard points in the current frame and the historical hazard point state vectors in the previous frame to determine the hazard point speed of the target vehicle. The hazard point state vector includes the hazard point's three-dimensional coordinates (x, y, z) and the speed in the x, y, z axes.

7. The vehicle speed measurement method as described in claim 1, characterized in that, The method further includes; Based on the speed of the target vehicle and the speed of the vehicle itself, the relative positional relationship between the target vehicle and the monocular camera mounted on the vehicle itself is determined. Based on the relative positional relationship, the driving strategy of the vehicle is adjusted.

8. A monocular camera, characterized in that, include: Photosensitive components are used to acquire monocular images; An image processor for performing the steps of the vehicle speed measurement method according to any one of claims 1 to 7.

Images