Method and system for predicting and tracking pitch angle of camera

Abstract

The invention provides a camera pitch angle prediction and tracking method and system, and belongs to the technical field of camera external parameter estimation. The method comprises the following steps: acquiring vehicle state information and camera basic parameters; constructing a rough transformation matrix according to the vehicle state information and the basic parameters of the camera; determining a current frame prediction image according to the rough transformation matrix; combining the current frame prediction image and the current frame actual image along the channel dimension and zooming to a preset size to obtain a multi-channel image; reasoning the multi-channel image to obtain a multi-channel to-be-decoded feature map; decoding the multi-channel to-be-decoded feature map to obtain a vanishing line height and a matching point pair coordinate set; calculating a current pitch angle of the camera according to the vanishing line height; calculating the change rate of the pitch angle of the camera according to the matching point pair coordinate set; and according to the current pitch angle of the camera and the pitch angle variable quantity, generating a camera pitch angle tracking sequence through a Kalman filtering algorithm so as to track the pitch angle of the camera.

Description

Technical Field

[0001] The present invention relates to the technical field of camera extrinsic parameter estimation, and in particular to a camera pitch angle prediction and tracking method and system. Background Art

[0002] In the automated driving assistance system (ADAS), the monocular ranging algorithm is one of the commonly used target detection technologies. Projection ranging technology estimates the target distance based on the ordinate of the target ground point in the image, the camera's internal parameters and external parameters. Specifically, the pixel ordinate is usually provided by the target detection model, while the camera's internal parameters are generally considered to be fixed, and the external parameters mainly include the camera's height above the ground and the pitch angle. Since the vehicle is often affected by factors such as road conditions, vehicle acceleration and deceleration, and uphill and downhill driving during driving, the camera's pitch angle is prone to change. Therefore, accurately estimating the camera's pitch angle becomes a key factor in the accuracy of projection ranging.

[0003] In the prior art, the existing camera pitch angle prediction method mainly relies on the parallel lane lines in the image. It is necessary to find a pair of straight lane lines that are parallel in the physical world in the image, use their intersection in the image as the vanishing point, and predict the camera pitch angle by calculating the coordinates of the vanishing point.

[0004] However, the above method mainly relies on the parallel lane lines in the image and cannot predict the camera pitch angle in scenes with curves or no lane lines.

[0005] In addition, the above method has high requirements on the accuracy of the vanishing point coordinates. A slight error may lead to a large range of ranging errors. Especially when the vehicle is driving at high speed or in complex road conditions, the prediction accuracy of the camera pitch angle is low and the reliability is poor. Summary of the invention

[0006] The main purpose of the present invention is to provide a camera pitch angle prediction and tracking method and system to solve the problem that the camera pitch angle prediction method in the prior art mainly relies on parallel lane lines in the image, and the camera pitch angle cannot be predicted in a curved road or in a scene without lane lines. In addition, the accuracy of the vanishing point coordinates is high, and a slight error may lead to a large range of ranging errors. In particular, when the vehicle is driving at high speed or in complex road conditions, the camera pitch angle prediction accuracy is low and the reliability is poor.

[0007] In order to achieve the above object, according to one aspect of the present invention, a camera pitch angle prediction and tracking method is provided, comprising:

[0008] S1: Get vehicle status information and basic camera parameters;

[0009] S2: constructing a rough transformation matrix according to the vehicle state information and the basic parameters of the camera;

[0010] S3: Determine a current frame prediction image according to the rough transformation matrix;

[0011] S4: merging the current frame prediction image and the current frame actual image along the channel dimension and scaling them to a preset size to obtain a multi-channel image;

[0012] S5: performing inference on the multi-channel image to obtain a multi-channel feature map to be decoded;

[0013] S6: Decode the multi-channel feature map to be decoded to obtain the vanishing line height and the coordinate set of the matching point pair;

[0014] S7: Calculate the current pitch angle of the camera according to the height of the vanishing line;

[0015] S8: Calculate the camera pitch angle change rate according to the matching point pair coordinate set;

[0016] S9: Generate a camera pitch angle tracking sequence according to the current pitch angle of the camera and the pitch angle change through a Kalman filter algorithm to perform camera pitch angle tracking.

[0017] Further, the vehicle status information includes: a vehicle speed sequence and a heading rate sequence;

[0018] The basic parameters of the camera include: the initial pitch angle of the camera, the internal parameters of the camera and the installation height of the camera.

[0019] Furthermore, the S2 specifically includes:

[0020] S201: Calculate the vehicle displacement according to the vehicle speed sequence:

[0021] d=∑v i

[0022] Where d represents the vehicle displacement, v i represents the speed of the vehicle at time i;

[0023] S202: Calculate the vehicle heading angle change according to the heading rate sequence:

[0024]

[0025] Among them, ΔΨ represents the change of vehicle heading angle, represents the heading rate of the vehicle at time i;

[0026] S203: Constructing a heading angle rotation matrix according to the heading angle change of the vehicle:

[0027]

[0028] Among them, R z () represents the heading angle rotation matrix, cos() represents the cosine operation, and sin() represents the sine operation;

[0029] S204: constructing a pitch angle rotation matrix according to the initial pitch angle of the camera:

[0030]

[0031] Among them, R y () represents the pitch angle rotation matrix, θ0 represents the initial pitch angle of the camera;

[0032] S205: Fusing the heading angle rotation matrix and the pitch angle rotation matrix to construct a fused rotation matrix:

[0033]

[0034] Among them, R1 represents the fusion rotation matrix;

[0035] S206: Construct a translation matrix according to the initial pitch angle of the camera, the installation height of the camera, the displacement of the vehicle, and the change in the heading angle of the vehicle:

[0036]

[0037] Where t represents the translation matrix, and H represents the camera installation height;

[0038] S207: Construct the rough transformation matrix according to the camera intrinsic parameters, the fused rotation matrix and the translation matrix:

[0039] T1=K·[R1|t]

[0040]

[0041] Among them, T1 represents the rough transformation matrix, K represents the camera internal parameter, | represents the concatenate operation, and f x Indicates the focal length of the camera in the x-axis direction, f y Indicates the focal length of the camera in the y-axis direction, c x Indicates the coordinate of the camera's principal point in the x-axis direction, c y Indicates the coordinate of the camera's principal point in the y-axis direction.

[0042] Furthermore, the S3 is specifically:

[0043] According to the rough transformation matrix, the current frame prediction image is determined by the warpPerspective function of OpenCV:

[0044]

[0045] in, Represents the predicted image of the current frame, cv2.warpPerspective() represents the warpPerspective function of OpenCV, f(i-1) represents the previous frame image, T1 represents the coarse transformation matrix, and size() represents the image size.

[0046] Furthermore, the S5 specifically includes:

[0047] S501: extracting multi-scale features of the multi-channel image through a ResNet network;

[0048] S502: Perform feature fusion on the multi-scale features through a BiFPN network to obtain a fused feature map;

[0049] S503: Processing the fused feature map through the FCN network to obtain a multi-channel feature map to be decoded.

[0050] Furthermore, each channel in the multi-channel feature map to be decoded includes: a vanishing line channel, a horizontal offset channel, a vertical offset channel and a mask channel.

[0051] Furthermore, the S6 specifically includes:

[0052] S601: Activate the vanishing line channel through the hyperbolic tangent function to obtain the vanishing line height:

[0053] tanh(O[0,row,col])=(h[row,col]-4*row+2) / h img

[0054] h[row,col]=tanh(O[0,row,col])*h img +4*row+2)

[0055] h = mean(h[row,col])

[0056] Among them, tanh() represents the hyperbolic tangent function, O[0,row,col] represents the value of the pixel in the vanishing line channel, row represents the row label of the pixel, col represents the column label of the pixel, h[row,col] represents the vanishing line height predicted at the current row and column position, and h imgIt represents the height of the multi-channel feature map to be decoded, h represents the vanishing line height, and mean() represents the mean function;

[0057] S602: Selecting a pixel point whose element value in the mask channel is greater than a preset value as a target pixel point;

[0058] S603: Calculate the pixel coordinates of the target pixel point in the actual image of the current frame:

[0059] x[row,col]=4*col-2

[0060] y[row,col]=4*col-2

[0061] Among them, x[row,col] represents the pixel horizontal coordinate of the target pixel point in the actual image of the current frame, and y[row,col] represents the pixel vertical coordinate of the target pixel point in the actual image of the current frame;

[0062] S604: Calculate the pixel coordinates of the target pixel point in the current frame prediction image according to the horizontal offset channel and the vertical offset channel:

[0063]

[0064] in, Represents the pixel horizontal coordinate of the target pixel in the predicted image of the current frame, O[1,row,col] represents the value of the pixel in the horizontal offset channel, Represents the pixel vertical coordinate of the target pixel in the predicted image of the current frame, and O[2,row,col] represents the value of the pixel in the vertical offset channel;

[0065] S605: Convert the pixel coordinates of the target pixel point in the actual image of the current frame and the pixel coordinates of the target pixel point in the predicted image of the current frame into a plurality of camera coordinates:

[0066] P camera =K -1 ·P pix

[0067] Among them, P camera represents the camera coordinates, P pix represents pixel coordinates, K represents the camera internal parameters, -1 represents the inverse operation;

[0068] S606: Merge the camera coordinates to obtain a set of matching point pair coordinates:

[0069]

[0070] Among them, P setRepresents the coordinate set of matching point pairs, x camera [row,col] represents the camera horizontal coordinate of the target pixel in the actual image of the current frame, camera [row,col] represents the camera ordinate of the target pixel in the actual image of the current frame. Represents the camera horizontal coordinate of the target pixel in the predicted image of the current frame, Represents the camera ordinate of the target pixel in the predicted image of the current frame, and O[3,row,col] represents the value of the pixel in the mask channel.

[0071] Further, the S7 is specifically:

[0072] Calculate the current pitch angle of the camera according to the following formula:

[0073]

[0074] Among them, θ represents the current pitch angle of the camera, c y Indicates the coordinate of the camera's principal point in the y-axis direction, f y It represents the focal length of the camera in the y-axis direction, and h represents the height of the vanishing line.

[0075] Furthermore, the S8 specifically includes:

[0076] S801: Based on the coordinate set of the matching point pairs, a refined transformation matrix is ​​constructed by the least squares method:

[0077]

[0078] Wherein, T2 represents the fine transformation matrix.

[0079] S802: Construct a projection matrix according to the refined transformation matrix:

[0080] P=T2·T1

[0081] Among them, P represents the projection matrix, T1 represents the rough transformation matrix;

[0082] S803: Using the cv2.decomposeProjectionMatrix function of OpenCV, the projection matrix is ​​restored to obtain a restored rotation matrix:

[0083]

[0084] Among them, R2 represents the restored rotation matrix, r ij Represents the element in the i-th row and j-th column of the restored rotation matrix, i = 1, 2, 3, j = 1, 2, 3;

[0085] S804: Calculate the camera pitch angle change rate according to the restored rotation matrix:

[0086] Δθ=arcsin(-r 31 )

[0087] Among them, Δθ represents the rate of change of the camera pitch angle, arcsin() represents the inverse sine function, and r 31 Represents restoring the element in the 3rd row and 1st column of the rotation matrix.

[0088] According to one aspect of the present invention, there is provided a camera pitch angle prediction and tracking system, comprising: a memory and one or more processors;

[0089] One or more applications are stored in the memory, and the one or more applications are suitable for being executed by the one or more processors to implement the camera pitch angle prediction and tracking method according to any one of claims 1 to 9.

[0090] By applying the technical solution of the present invention, a rough transformation matrix can be constructed according to vehicle state information and basic camera parameters. According to the rough transformation matrix, a predicted image of the current frame is determined. The predicted image of the current frame and the actual image of the current frame are merged along the channel dimension and scaled to a preset size to obtain a multi-channel image. The multi-channel image is no longer dependent on parallel lane lines in the image. The camera pitch angle can be predicted in a curved road or in a scene without lane lines. By inferring the multi-channel image, a multi-channel feature map to be decoded is obtained. The multi-channel feature map to be decoded is decoded to obtain the vanishing line height and a set of matching point pair coordinates. The vanishing point coordinates are no longer dependent, and the phenomenon of a large range of ranging errors caused by a slight error in the vanishing point coordinates is avoided. When the vehicle is driving at high speed or in complex road conditions, the camera pitch angle prediction accuracy is high and the reliability is strong.

[0091] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0093] Figure 1 A schematic diagram of a process flow of a camera pitch angle prediction and tracking method provided by an embodiment of the present invention is shown;

[0094] Figure 2 A schematic structural diagram of a camera pitch angle prediction and tracking system provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0095] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0096] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0097] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0098] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0099] Reference Manual Attached Figure 1 , shows a flow chart of a camera pitch angle prediction and tracking method provided by an embodiment of the present invention.

[0100] An embodiment of the present invention provides a camera pitch angle prediction and tracking method, comprising:

[0101] S1: Get vehicle status information and basic camera parameters.

[0102] Optionally, the vehicle status information includes: a vehicle speed sequence and a heading rate sequence.

[0103] It should be noted that the method for acquiring the vehicle speed sequence and the heading rate sequence is specifically as follows: within one image acquisition cycle (from acquiring the previous frame image to acquiring the current frame image), multiple vehicle speeds and heading rates are collected from the vehicle host (through CAN signal transmission), and the collected multiple vehicle speed data and multiple heading rates are stored as a vehicle speed sequence and a heading rate sequence, respectively.

[0104] The basic parameters of the camera include: initial pitch angle of the camera, internal parameters of the camera, and installation height of the camera.

[0105] It should be noted that the camera initial pitch angle uses a calibration value when it is first calculated, and the camera pitch angle obtained from the previous calculation is used in subsequent calculations.

[0106] In the present invention, by storing the vehicle speed sequence and the heading rate sequence as continuous data within a period, the dynamic changes of the vehicle within the image acquisition period can be fully reflected, and the accuracy of the displacement and heading angle change calculation is enhanced. At the same time, by using the calibration value for the first calculation and the previous calculation result for the subsequent calculation, not only the accuracy of the initial state is guaranteed, but also the continuity and stability of the pitch angle are achieved by using historical data, thereby improving the real-time and robustness of the pitch angle prediction and tracking.

[0107] S2: Construct a rough transformation matrix based on the vehicle status information and basic camera parameters.

[0108] In the present invention, by constructing a rough transformation matrix based on vehicle state information and basic camera parameters, the vehicle dynamic information and the camera's geometric parameters can be combined to quickly generate a preliminary transformation result of the current frame prediction image, providing a basis for subsequent image feature extraction and accurate pitch angle calculation. This method effectively improves the calculation efficiency, while ensuring the initial accuracy of the prediction, and helps to achieve real-time tracking of the camera pitch angle.

[0109] In a possible implementation, S2 specifically includes sub-steps S201 to S207:

[0110] S201: Calculate vehicle displacement according to vehicle speed sequence:

[0111] d=Σv i

[0112] Where d represents the vehicle displacement, v i represents the speed of the vehicle at time i.

[0113] S202: Calculate the vehicle heading angle change according to the heading rate sequence:

[0114]

[0115] Among them, ΔΨ represents the change of vehicle heading angle, Represents the heading rate of the vehicle at time i.

[0116] S203: Construct a heading angle rotation matrix according to the heading angle change of the vehicle:

[0117]

[0118] Among them, R z () represents the heading angle rotation matrix, cos() represents the cosine operation, and sin() represents the sine operation.

[0119] S204: Construct a pitch angle rotation matrix according to the initial pitch angle of the camera:

[0120]

[0121] Among them, R y () represents the pitch angle rotation matrix, and θ0 represents the initial pitch angle of the camera.

[0122] S205: Fuse the heading angle rotation matrix and the pitch angle rotation matrix to construct a fused rotation matrix:

[0123]

[0124] Among them, R1 represents the fused rotation matrix.

[0125] S206: Construct a translation matrix according to the initial pitch angle of the camera, the camera installation height, the vehicle displacement, and the change in the vehicle heading angle:

[0126]

[0127] Where t represents the translation matrix and H represents the camera installation height.

[0128] S207: Construct a rough transformation matrix based on the camera intrinsic parameters, the fused rotation matrix and the translation matrix:

[0129] T1=K·[R1|t]

[0130]

[0131] Among them, T1 represents the rough transformation matrix, K represents the camera internal parameter, | represents the concatenate operation, and f x Indicates the focal length of the camera in the x-axis direction, f y Indicates the focal length of the camera in the y-axis direction, c x Indicates the coordinate of the camera's principal point in the x-axis direction, c y Indicates the coordinate of the camera's principal point in the y-axis direction.

[0132] In the present invention, by calculating the displacement and heading angle change of the vehicle based on the vehicle speed sequence and heading rate sequence, the motion state of the vehicle during the image acquisition cycle is accurately reflected, providing reliable displacement and angle data for subsequent image transformation. By constructing the heading angle rotation matrix and the pitch angle rotation matrix respectively and then fusing them, the changes in vehicle motion and camera posture can be accurately described, and the accuracy of the transformation matrix can be improved. By constructing the translation matrix, the actual displacement of the vehicle, the camera installation height and the heading angle change are combined to ensure the accuracy of the spatial transformation, thereby reducing the errors that may be introduced due to simplified assumptions. Finally, a rough transformation matrix is ​​constructed by combining the camera internal parameters and the rotation and translation matrices, providing an accurate basis for subsequent image transformation and pitch angle calculation.

[0133] S3: Determine the predicted image of the current frame according to the rough transformation matrix.

[0134] In a possible implementation, S3 is specifically:

[0135] According to the rough transformation matrix, the current frame prediction image is determined through the warpPerspective function of OpenCV:

[0136]

[0137] in, Represents the predicted image of the current frame, cv2.warpPerspective() represents the warpPerspective function of OpenCV, f(i-1) represents the previous frame image, T1 represents the coarse transformation matrix, and size() represents the image size.

[0138] Specifically, through the warpPerspective function of OpenCV, a rough transformation matrix is ​​used to perform a projection transformation on the input previous frame image and scale it to a preset size to obtain a predicted image of the current frame.

[0139] It should be noted that OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library designed to provide basic functions for real-time computer vision. OpenCV provides a wealth of image and video processing tools, supporting image reading, processing, analysis, recognition, and tracking functions. It is widely used in various computer vision tasks, such as object recognition, face detection, image restoration, motion analysis, etc.

[0140] It should be noted that warpPerspective is a function in OpenCV that performs perspective transformation. Perspective transformation is an image transformation operation that maps an image through a given transformation matrix (usually a 3x3 or 3x4 matrix) to achieve effects such as image rotation, scaling, translation, or distortion. In camera pitch angle prediction and tracking, warpPerspective is used to transform the previous frame image according to the coarse transformation matrix and scale the image to a preset size so that it is aligned with the current frame image.

[0141] In the present invention, by using the warpPerspective function of OpenCV, combined with the coarse transformation matrix, the perspective transformation of the previous frame image is performed and scaled to a preset size, the predicted image of the current frame can be quickly and accurately generated. This method can effectively reflect the dynamic changes of the vehicle and the adjustment of the camera posture into the image, thereby ensuring the alignment of the current frame with the actual scene. At the same time, by utilizing the efficient image processing capabilities of OpenCV, the computational burden is reduced, the real-time performance and processing speed are improved, and efficient performance in real-time applications is ensured.

[0142] S4: The predicted image of the current frame and the actual image of the current frame are merged along the channel dimension and scaled to a preset size to obtain a multi-channel image.

[0143] Specifically, the predicted image of the current frame and the actual image of the current frame are merged along the channel dimension and scaled to a preset size to obtain a 6-channel image.

[0144] It should be noted that those skilled in the art can set the size of the preset size according to actual needs, and the present invention is not limited thereto.

[0145] In the present invention, by merging the predicted image of the current frame with the actual image of the current frame along the channel dimension, the feature information of the predicted image and the actual image can be fully integrated to provide richer input for subsequent image processing. This not only helps to improve the diversity and feature expression ability of the image, but also enhances the model's perception of image changes and improves the accuracy of pitch angle prediction and tracking.

[0146] S5: Perform inference on the multi-channel image to obtain a multi-channel feature map to be decoded.

[0147] In the present invention, by inferring the multi-channel image to obtain the multi-channel feature map to be decoded, richer feature information can be effectively extracted from the fused image.

[0148] In a possible implementation, S5 specifically includes sub-steps S501 to S503:

[0149] S501: Extract multi-scale features of multi-channel images through the ResNet network.

[0150] Specifically, the ResNet network is used to extract the basic features of the above 6-channel images and generate multi-scale features (F4, F8, F 16 ,F 32 ), F represents Feature, and i represents the downsampling ratio.

[0151] It should be noted that ResNet (Residual Network) is a deep convolutional neural network architecture designed to solve the problems of gradient vanishing and training difficulties that arise as the network deepens. ResNet introduces skip connections, which allows information to skip certain layers directly, thereby achieving deeper network training without losing information. This structure can greatly improve the network's expressiveness and training effects, and is widely used in computer vision tasks such as image classification and object detection.

[0152] S502: Perform feature fusion on multi-scale features through the BiFPN network to obtain a fused feature map.

[0153] Specifically, the multi-scale features are fused through the BiFPN network to obtain a 4-fold downsampled fused feature map.

[0154] It should be noted that BiFPN (Bidirectional Feature Pyramid Network) is a network structure for feature fusion, which is specially designed to process multi-scale features. BiFPN uses bidirectional information flow when fusing features, and more effectively captures image information at different scales by weighted fusion of feature maps at different levels. Unlike traditional feature pyramid networks (FPN), BiFPN effectively improves the detection accuracy of small objects by performing multiple weighted fusions between layers, and has good computational efficiency, and is often used in object detection tasks.

[0155] S503: Process the fused feature map through the FCN network to obtain a multi-channel feature map to be decoded.

[0156] Specifically, the fused feature map is processed through the FCN network to obtain a 4-fold downsampled multi-channel feature map to be decoded (O).

[0157] It should be noted that FCN (Fully Convolutional Network) is a neural network composed entirely of convolutional layers, which is particularly suitable for pixel-level prediction tasks, such as semantic segmentation. Unlike traditional convolutional neural networks (CNNs), FCN does not have a fully connected layer, but directly outputs a prediction map of the same size as the input image through the convolutional layer. FCN can classify each pixel, so it is particularly suitable for tasks that require predictions for each pixel, such as image segmentation, edge detection, etc.

[0158] In the present invention, the multi-scale features of the multi-channel image are extracted through the ResNet network, which can fully capture the different scale information in the image, improve the recognition ability of the image details, and provide rich information for subsequent feature fusion and prediction. The BiFPN network is used for feature fusion, and the feature maps of different scales are weighted and fused through the bidirectional information flow, which not only improves the detection accuracy of small objects, but also optimizes the calculation efficiency, ensures that better fusion results can be obtained under multi-scale features, and thus enhances the accuracy of pitch angle prediction. The fused feature map is processed by the FCN network to generate a multi-channel feature map to be decoded suitable for pixel-level prediction, which can retain the spatial structure information of the image, ensure that the data of each pixel can be accurately processed in the pitch angle calculation, and improve the granularity and accuracy of the prediction.

[0159] Among them, each channel in the multi-channel feature map to be decoded includes: a vanishing line channel, a horizontal offset channel, a vertical offset channel and a mask channel.

[0160] It should be noted that the vanishing line channel O[0] is used to predict the position of the vanishing line in the image.

[0161] It should be noted that the horizontal offset channel O[1] and the vertical offset channel O[2] respectively represent the pixel position offsets in the horizontal direction and the vertical direction, and are used to associate the pixel correspondence between the current frame and the predicted frame.

[0162] It should be noted that the mask channel O[3] is used to determine whether the pixel of each cell is a road surface area, and to check whether the corresponding point of the pixel can be found in the previous frame.

[0163] In the present invention, the vanishing line channel is used to accurately predict the position of the vanishing line in the image, which helps to infer the camera pitch angle. The horizontal and vertical offset channels represent the pixel position offsets in the horizontal and vertical directions, respectively, and can associate the pixel correspondence between the current frame and the predicted frame, thereby improving the image alignment accuracy. The mask channel is used to distinguish the road area and check whether the pixel can find a corresponding point in the previous frame, ensuring that only valid pixel information is used for further calculations.

[0164] S6: Decode the multi-channel feature map to be decoded to obtain the vanishing line height and the coordinate set of the matching point pair.

[0165] In the present invention, the vanishing line height provides a basis for estimating the pitch angle of the camera, and the matching point pair coordinate set helps to establish the spatial relationship between the current frame and the predicted frame. Through this decoding process, the geometric changes in the image can be accurately calculated, thereby improving the prediction accuracy and tracking stability of the pitch angle. In addition, this method can effectively reduce the impact of noise and error on the prediction results, thereby enhancing the robustness of the system.

[0166] In a possible implementation, S6 specifically includes sub-steps S601 to S606:

[0167] S601: Activate the vanishing line channel through the hyperbolic tangent function to obtain the vanishing line height:

[0168] tanh(O[0,row,col])=(h[row,col]-4*row+2) / h img

[0169] h[row,col]=tanh(O[0,row,col])*h img +4*row+2)

[0170] h = mean(h[row,col])

[0171] Among them, tanh() represents the hyperbolic tangent function, O[0,row,col] represents the value of the pixel in the vanishing line channel, row represents the row label of the pixel, col represents the column label of the pixel, h[row,col] represents the vanishing line height predicted at the current row and column position, and h img It represents the height of the multi-channel feature map to be decoded, h represents the vanishing line height, and mean() represents the mean function.

[0172] It should be noted that the hyperbolic tangent function (tanh) is a mathematical function defined as the ratio of the sine and cosine of the input value. In computer vision and deep learning, the hyperbolic tangent function is often used as an activation function to normalize the input data so that the output value of the network is limited to a reasonable range, thereby accelerating convergence and reducing the problem of gradient explosion. Specifically for the application in this step, tanh is used to activate the value in the vanishing line channel, convert it into a physically meaningful vanishing line height, and perform subsequent calculations through this process.

[0173] S602: Selecting pixels in the mask channel whose element values ​​are greater than a preset value as target pixels.

[0174] It should be noted that those skilled in the art can set the size of the preset value according to actual needs, and the present invention is not limited here.

[0175] S603: Calculate the pixel coordinates of the target pixel in the actual image of the current frame:

[0176] x[row,col]=4*col-2

[0177] y[row,col]=4*col-2

[0178] Among them, x[row,col] represents the pixel horizontal coordinate of the target pixel point in the actual image of the current frame, and y[row,col] represents the pixel vertical coordinate of the target pixel point in the actual image of the current frame.

[0179] S604: Calculate the pixel coordinates of the target pixel in the current frame prediction image according to the horizontal offset channel and the vertical offset channel:

[0180]

[0181]

[0182] in, Represents the pixel horizontal coordinate of the target pixel in the predicted image of the current frame, O[1,row,col] represents the value of the pixel in the horizontal offset channel, Represents the pixel vertical coordinate of the target pixel in the predicted image of the current frame, and O[2,row,col] represents the value of the pixel in the vertical offset channel.

[0183] S605: Convert the pixel coordinates of the target pixel point in the actual image of the current frame and the pixel coordinates in the predicted image of the current frame into multiple camera coordinates:

[0184] P camera =K -1 ·P pix

[0185] Among them, P camera represents the camera coordinates, P pix represents pixel coordinates, K represents the camera internal parameters, -1 Represents the inverse operation.

[0186] For example, the pixel coordinates of the target pixel point in the actual image of the current frame and the pixel coordinates in the predicted image of the current frame are converted into multiple camera coordinates, specifically:

[0187] (x camera ,y camera ,1) T =K -1 ·(xpix ,y pix ,1) T

[0188] Among them, x camera Indicates the horizontal coordinate of the camera, y camera represents the camera ordinate, T represents the transpose operation, x pix Indicates the horizontal coordinate of the pixel, y pix Indicates the pixel vertical coordinate.

[0189] S606: Merge the coordinates of each camera to obtain a set of matching point pair coordinates:

[0190]

[0191] Among them, P set Represents the coordinate set of matching point pairs, x camera [row,col] represents the camera horizontal coordinate of the target pixel in the actual image of the current frame, camera [row,col] represents the camera ordinate of the target pixel in the actual image of the current frame. Represents the camera horizontal coordinate of the target pixel in the predicted image of the current frame, Represents the camera ordinate of the target pixel in the predicted image of the current frame, and O[3,row,col] represents the value of the pixel in the mask channel.

[0192] In the present invention, by activating the vanishing line channel through the hyperbolic tangent function, the vanishing line position can be converted into a vanishing line height with physical significance, providing an accurate basis for the calculation of the camera pitch angle. The target pixel points related to the road surface are screened out through the mask channel to ensure that only valid pixel information is used for subsequent calculations, thereby improving the accuracy and robustness of the system. By calculating the coordinates of the target pixel points in the actual image and the predicted image of the current frame, the pixel correspondence between the two frames of images can be accurately established, providing reliable data support for subsequent matching and transformation calculations. By converting pixel coordinates into camera coordinates, the image space coordinates can be accurately mapped to three-dimensional space, ensuring accurate docking between the image and the actual scene, and providing an accurate basis for subsequent geometric calculations. By merging the coordinates of each camera, a set of matching point pair coordinates is generated, which provides accurate geometric data support for the calculation and tracking of the pitch angle, and improves the accuracy of image transformation and camera parameter estimation.

[0193] S7: Calculate the current pitch angle of the camera according to the height of the vanishing line.

[0194] In a possible implementation manner, S7 specifically includes:

[0195] Calculate the current pitch angle of the camera according to the following formula:

[0196]

[0197] Among them, θ represents the current pitch angle of the camera, c y Indicates the coordinate of the camera's principal point in the y-axis direction, f y It represents the focal length of the camera in the y-axis direction, and h represents the height of the vanishing line.

[0198] In the present invention, by calculating the current pitch angle of the camera according to the vanishing line height, the geometric position of the vanishing line reflected in the image can be directly converted into the actual pitch angle of the camera. By combining the relationship between the camera internal parameters (such as focal length and principal point coordinates) and the vanishing line height, the pitch angle is quickly calculated, avoiding the complex iterative solution process, which not only improves the calculation efficiency, but also ensures the accuracy of angle prediction, thereby providing reliable technical support for real-time correction and tracking of camera posture.

[0199] S8: Calculate the camera pitch angle change rate based on the matching point pair coordinate set.

[0200] In the present invention, by calculating the camera pitch angle change rate based on the matching point pair coordinate set, the change of the camera posture over time can be effectively captured. This method provides accurate angle change information by analyzing the matching point pairs between the current frame and the predicted frame, which helps to dynamically adjust the camera pitch angle and ensure more stable and accurate tracking.

[0201] In a possible implementation, S8 specifically includes sub-steps S801 to S804:

[0202] S801: Based on the coordinate set of the matching point pairs, a refined transformation matrix is ​​constructed by the least squares method:

[0203]

[0204] Wherein, T2 represents the fine transformation matrix.

[0205] It should be noted that the least squares method is a mathematical optimization technique used to find the best fitting model in the data, and is usually used in regression analysis. The basic idea is to solve the unknown parameters by minimizing the sum of squares of the errors between the model predictions and the actual observed values. Specifically, the least squares method calculates the sum of squares of the vertical distances between all data points and the fitting curve (or straight line), and minimizes this sum of squares by adjusting the model parameters to obtain the optimal fitting result. It is widely used in linear regression, curve fitting and other problems.

[0206] S802: Construct a projection matrix based on the refined transformation matrix:

[0207] P=T2·T1

[0208] Among them, P represents the projection matrix and T1 represents the rough transformation matrix.

[0209] S803: Use the cv2.decomposeProjectionMatrix function of OpenCV to restore the projection matrix and obtain the restored rotation matrix:

[0210]

[0211] Among them, R2 represents the restored rotation matrix, r ij Represents the element in the i-th row and j-th column of the restored rotation matrix, i=1,2,3, j=1,2,3.

[0212] It should be noted that cv2.decomposeProjectionMatrix is ​​a function in the OpenCV library that is used to decompose the projection matrix. The function is to decompose a 3x4 projection matrix into multiple matrices, usually including rotation matrices, translation vectors, intrinsic matrix, etc., to restore the internal and external parameters of the camera. The projection matrix is ​​usually obtained from the camera's 3D to 2D projection conversion, and by decomposing the projection matrix, the camera matrix, rotation matrix, and translation matrix of the camera can be extracted. This information is very important for tasks such as camera pose estimation and 3D reconstruction.

[0213] S804: Calculate the camera pitch angle change rate based on the restored rotation matrix:

[0214] Δθ=arcsin(-r 31 )

[0215] Among them, Δθ represents the rate of change of the camera pitch angle, arcsin() represents the inverse sine function, and r 31 Represents restoring the element in the 3rd row and 1st column of the rotation matrix.

[0216] In the present invention, a refined transformation matrix is ​​constructed by the least squares method, which can effectively reduce matching errors, ensure the accuracy of image transformation, and provide high-quality data for subsequent pitch angle change rate calculations. By constructing a projection matrix based on the refined transformation matrix, the two-dimensional information in the image can be converted into three-dimensional data, thereby more accurately describing the geometric relationship of the camera. By decomposing the projection matrix through the cv2.decomposeProjectionMatrix function of OpenCV, the rotation matrix of the camera can be extracted, thereby accurately restoring the camera posture and ensuring the reliability of the pitch angle change rate calculation. Using the restored rotation matrix, the rate of change of the camera pitch angle is calculated by the inverse sine function, which can accurately quantify the change of the camera posture and provide an accurate basis for dynamically adjusting the camera pitch angle.

[0217] S9: Generate a camera pitch angle tracking sequence according to the current pitch angle of the camera and the pitch angle change through a Kalman filter algorithm to perform camera pitch angle tracking.

[0218] It should be noted that the Kalman filter algorithm is a recursive algorithm for estimating the state of a linear dynamic system. It combines the system's prediction model and actual observation data to provide the best state estimate in the presence of noise. Kalman filtering works in two main steps: prediction and update. In the prediction step, the algorithm predicts the current state based on the state at the previous moment and the system model. In the update step, the prediction is corrected using the actual measurement value to improve the estimation accuracy. Kalman filtering is widely used in fields such as automatic control, navigation, and signal processing. In particular, it can effectively smooth data, eliminate noise, and improve tracking accuracy in tasks such as camera tracking and target tracking.

[0219] Specifically, take (θ, Δθ) as the state variable and define the state variable X:

[0220] Furthermore, the state transfer matrix F' is defined as: f represents the acquisition frequency of the camera.

[0221] Furthermore, the measurement matrix H of the second-order unit matrix is ​​defined as:

[0222] Furthermore, through the state transfer matrix F' and the state variable X of the previous frame i-1 Predict the state variable X of the current frame i :X i =F′·X i-1 .

[0223] Furthermore, the predicted state is corrected by the θ and Δθ obtained by actual measurement: K represents the Kalman gain, P i-1 represents the covariance matrix of the previous frame, and R represents the covariance matrix of the observation noise.

[0224] Further, the updated status is: represents the updated state variable, z i Indicates θ and Δθ actually measured.

[0225] Further, update the covariance matrix P i :P i =(IK·H)·P i-1 , I represents the identity matrix.

[0226] Furthermore, the output result is: generating a stable camera pitch angle tracking sequence to perform camera pitch angle tracking.

[0227] In the present invention, by combining the prediction model of the system and the actual observation data through the Kalman filter algorithm, noise can be effectively eliminated, the accuracy of state estimation can be improved, and the camera pitch angle tracking can be made more stable and reliable. The prediction and update mechanism of the Kalman filter can process each frame of data in real time, and by recursively updating the state variables, the real-time and high efficiency of the camera pitch angle tracking are guaranteed to adapt to changes in the dynamic environment. Through the update of the covariance matrix and the adjustment of the Kalman gain, the system can automatically optimize the weights according to the observed noise and prediction error, maintain the accuracy and stability of the camera pitch angle tracking process, and avoid error accumulation and tracking drift. By combining the current pitch angle and the change amount, the Kalman filter algorithm can accurately estimate the posture change of the camera in a dynamic environment, ensuring the high accuracy of the camera pitch angle prediction and tracking.

[0228] From the above description, it can be seen that the camera pitch angle prediction and tracking method provided by the present invention can construct a rough transformation matrix according to the vehicle state information and the basic parameters of the camera, determine the current frame prediction image according to the rough transformation matrix, merge the current frame prediction image and the current frame actual image along the channel dimension and scale them to a preset size to obtain a multi-channel image, which no longer depends on the parallel lane lines in the image, and can predict the camera pitch angle in a curved road or in a scene without lane lines. By reasoning the multi-channel image, a multi-channel feature map to be decoded is obtained, and the multi-channel feature map to be decoded is decoded to obtain the vanishing line height and the matching point pair coordinate set, which no longer depends on the vanishing point coordinates, and avoids the phenomenon of large-scale ranging errors caused by slight errors in the vanishing point coordinates. When the vehicle is driving at high speed or in complex road conditions, the camera pitch angle prediction accuracy is high and the reliability is strong.

[0229] Reference Manual Attached Figure 2 , showing a structural schematic diagram of a camera pitch angle prediction and tracking system provided by an embodiment of the present invention.

[0230] The embodiment of the present invention provides a camera pitch angle prediction and tracking system 30 , including: a memory 303 and one or more processors 301 .

[0231] One or more application programs are stored in the memory 303 , and the one or more application programs are suitable for being executed by the one or more processors 301 to implement the camera pitch angle prediction and tracking method described in the method embodiment.

[0232] The camera pitch angle prediction and tracking system 30 includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302.

[0233] The structure of the camera pitch angle prediction and tracking system 30 does not constitute a limitation on the embodiment of the present invention.

[0234] Processor 301 may be a CPU, a general purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0235] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI bus or an EISA bus, etc. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0236] The memory 303 can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compressed optical disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0237] It should be noted that the camera pitch angle prediction and tracking system 30 can implement the above-mentioned camera pitch angle prediction and tracking method, and can achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0238] From the above description, it can be seen that the camera pitch angle prediction and tracking system provided by the present invention can construct a rough transformation matrix according to the vehicle state information and the basic parameters of the camera, determine the current frame prediction image according to the rough transformation matrix, merge the current frame prediction image and the current frame actual image along the channel dimension and scale them to a preset size to obtain a multi-channel image, which no longer depends on the parallel lane lines in the image, and can predict the camera pitch angle in a curved road or in a scene without lane lines. By reasoning the multi-channel image, a multi-channel feature map to be decoded is obtained, and the multi-channel feature map to be decoded is decoded to obtain the vanishing line height and the matching point pair coordinate set, which no longer depends on the vanishing point coordinates, and avoids the phenomenon of large-scale ranging errors caused by slight errors in the vanishing point coordinates. When the vehicle is driving at high speed or in complex road conditions, the camera pitch angle prediction accuracy is high and the reliability is strong.

[0239] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0240] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0241] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the devices or elements referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention. The directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0242] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A camera pitch angle prediction and tracking method, characterized in that: include: S1: Get vehicle status information and basic camera parameters; S2: constructing a rough transformation matrix according to the vehicle state information and the basic parameters of the camera; S3: Determine a current frame prediction image according to the rough transformation matrix; S4: merging the current frame prediction image and the current frame actual image along the channel dimension and scaling them to a preset size to obtain a multi-channel image; S5: performing inference on the multi-channel image to obtain a multi-channel feature map to be decoded; S6: Decode the multi-channel feature map to be decoded to obtain the vanishing line height and the coordinate set of the matching point pair; S7: Calculate the current pitch angle of the camera according to the height of the vanishing line; S8: Calculate the camera pitch angle change rate according to the matching point pair coordinate set; S9: Generate a camera pitch angle tracking sequence according to the current pitch angle of the camera and the pitch angle change through a Kalman filter algorithm to perform camera pitch angle tracking.

2. The camera pitch angle prediction and tracking method according to claim 1, characterized in that: The vehicle status information includes: a vehicle speed sequence and a heading rate sequence; The basic parameters of the camera include: the initial pitch angle of the camera, the internal parameters of the camera and the installation height of the camera.

3. The camera pitch angle prediction and tracking method according to claim 2, characterized in that: The S2 specifically includes: S201: Calculate the vehicle displacement according to the vehicle speed sequence: d=Σv i Where d represents the vehicle displacement, v i represents the speed of the vehicle at time i; S202: Calculate the vehicle heading angle change according to the heading rate sequence: Among them, ΔΨ represents the change of vehicle heading angle, represents the heading rate of the vehicle at time i; S203: Constructing a heading angle rotation matrix according to the heading angle change of the vehicle: Among them, R z () represents the heading angle rotation matrix, cos() represents the cosine operation, and sin() represents the sine operation; S204: constructing a pitch angle rotation matrix according to the initial pitch angle of the camera: Among them, R y () represents the pitch angle rotation matrix, θ0 represents the initial pitch angle of the camera; S205: Fusing the heading angle rotation matrix and the pitch angle rotation matrix to construct a fused rotation matrix: Among them, R1 represents the fusion rotation matrix; S206: Construct a translation matrix according to the initial pitch angle of the camera, the installation height of the camera, the displacement of the vehicle, and the change in the heading angle of the vehicle: Where t represents the translation matrix, and H represents the camera installation height; S207: Construct the rough transformation matrix according to the camera intrinsic parameters, the fused rotation matrix and the translation matrix: T1=K·[R1|t] Among them, T1 represents the rough transformation matrix, K represents the camera internal parameter, | represents the concatenate operation, and f x Indicates the focal length of the camera in the x-axis direction, f y Indicates the focal length of the camera in the y-axis direction, c x Indicates the coordinate of the camera's principal point in the x-axis direction, c y Indicates the coordinate of the camera's principal point in the y-axis direction.

4. The camera pitch angle prediction and tracking method according to claim 1, characterized in that: The S3 is specifically: According to the rough transformation matrix, the current frame prediction image is determined by the warpPerspective function of OpenCV: in, Represents the predicted image of the current frame, cv2.warpPerspective() represents the warpPerspective function of OpenCV, f(i-1) represents the previous frame image, T1 represents the coarse transformation matrix, and size() represents the image size.

5. The camera pitch angle prediction and tracking method according to claim 1, characterized in that: The S5 specifically includes: S501: extracting multi-scale features of the multi-channel image through a ResNet network; S502: Perform feature fusion on the multi-scale features through a BiFPN network to obtain a fused feature map; S503: Processing the fused feature map through the FCN network to obtain a multi-channel feature map to be decoded.

6. The camera pitch angle prediction and tracking method according to claim 5, characterized in that: Each channel in the multi-channel feature map to be decoded includes: a vanishing line channel, a horizontal offset channel, a vertical offset channel and a mask channel.

7. The camera pitch angle prediction and tracking method according to claim 6, characterized in that: The S6 specifically includes: S601: Activate the vanishing line channel through the hyperbolic tangent function to obtain the vanishing line height: tanh(O[0,row,col])=(h[row,col]-4*row+2) / h img h[row,col]=tanh(O[0,row,col])*h img +4*row+2) h = mean(h[row,col]) Among them, tanh() represents the hyperbolic tangent function, O[0,row,col] represents the value of the pixel in the vanishing line channel, row represents the row label of the pixel, col represents the column label of the pixel, h[row,col] represents the vanishing line height predicted at the current row and column position, and h img It represents the height of the multi-channel feature map to be decoded, h represents the vanishing line height, and mean() represents the mean function; S602: Selecting a pixel point whose element value in the mask channel is greater than a preset value as a target pixel point; S603: Calculate the pixel coordinates of the target pixel point in the actual image of the current frame: x[row,col]=4*col-2 y[row,col]=4*col-2 Among them, x[row,col] represents the pixel horizontal coordinate of the target pixel point in the actual image of the current frame, and y[row,col] represents the pixel vertical coordinate of the target pixel point in the actual image of the current frame; S604: Calculate the pixel coordinates of the target pixel point in the current frame prediction image according to the horizontal offset channel and the vertical offset channel: in, Represents the pixel horizontal coordinate of the target pixel in the predicted image of the current frame, O[1,row,col] represents the value of the pixel in the horizontal offset channel, Represents the pixel vertical coordinate of the target pixel in the predicted image of the current frame, and O[2,row,col] represents the value of the pixel in the vertical offset channel; S605: Convert the pixel coordinates of the target pixel point in the actual image of the current frame and the pixel coordinates of the target pixel point in the predicted image of the current frame into a plurality of camera coordinates: P camera =K -1 ·P pix Among them, P camera represents the camera coordinates, P pix represents pixel coordinates, K represents the camera internal parameters, -1 represents the inverse operation; S606: Merge the camera coordinates to obtain a set of matching point pair coordinates: Among them, P set Represents the coordinate set of matching point pairs, x camera [row,col] represents the camera horizontal coordinate of the target pixel in the actual image of the current frame, camera [row,col] represents the camera ordinate of the target pixel in the actual image of the current frame. Represents the camera horizontal coordinate of the target pixel in the predicted image of the current frame, Represents the camera ordinate of the target pixel in the predicted image of the current frame, and O[3,row,col] represents the value of the pixel in the mask channel.

8. The camera pitch angle prediction and tracking method according to claim 1, characterized in that: The S7 is specifically: Calculate the current pitch angle of the camera according to the following formula: Among them, θ represents the current pitch angle of the camera, c y Indicates the coordinate of the camera's principal point in the y-axis direction, f y It represents the focal length of the camera in the y-axis direction, and h represents the height of the vanishing line.

9. The camera pitch angle prediction and tracking method according to claim 1, characterized in that: The S8 specifically includes: S801: Based on the coordinate set of the matching point pairs, a refined transformation matrix is ​​constructed by the least squares method: Wherein, T2 represents the fine transformation matrix. S802: Construct a projection matrix according to the refined transformation matrix: P=T2·T1 Where P represents the projection matrix, T1 represents the rough transformation matrix; S803: Using the cv2.decomposeProjectionMatrix function of OpenCV, the projection matrix is ​​restored to obtain a restored rotation matrix: Among them, R2 represents the restored rotation matrix, r ij Represents the element in the i-th row and j-th column of the restored rotation matrix, i = 1, 2, 3, j = 1, 2, 3; S804: Calculate the camera pitch angle change rate according to the restored rotation matrix: Δθ=arcsin(-r 31 ) Among them, Δθ represents the rate of change of the camera pitch angle, arcsin() represents the inverse sine function, and r 31 Represents restoring the element in the 3rd row and 1st column of the rotation matrix.

10. A camera pitch angle prediction and tracking system, characterized in that: include: memory and one or more processors; One or more applications are stored in the memory, and the one or more applications are suitable for being executed by the one or more processors to implement the camera pitch angle prediction and tracking method according to any one of claims 1 to 9.

Images