Vehicle steering angle determination method, device and equipment

Abstract

The invention provides a vehicle steering angle determination method, device and equipment, and the method comprises the steps: obtaining a plurality of candidate angular points based on a current frame of panoramic image, and for any pixel point in the current frame of panoramic image, obtaining a plurality of candidate angular points on a circle which takes the pixel point as a circle center and has a radius R; if the difference value between the brightness values of N continuous pixel points and the brightness value of the pixel point is greater than a threshold value, taking the pixel point as a candidate angular point; determining a characteristic intensity response value of the candidate corner point based on the brightness value of each pixel point in the neighborhood window of the candidate corner point; generating a feature point set; wherein on the basis of the feature intensity response value of each candidate corner point, a part of candidate corner points are screened from the plurality of candidate corner points to serve as feature points, and the feature point set comprises a plurality of feature points; wherein the feature point set is used for determining a first parameter, and the collected steering angle of the vehicle is converted into the actual steering angle of the vehicle through the first parameter. According to the scheme, the first parameter calculation accuracy can be improved, and errors of the first parameter are avoided.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation, and in particular to a method, apparatus and equipment for determining vehicle steering angle. Background Technology

[0002] With the continuous development of vehicle driver assistance technology and vehicle autonomous driving technology, many vehicles have automatic parking (automatic parking) function. Automatic parking refers to the vehicle automatically parking into a space without the need for manual control by the user. It can help users park automatically, avoid the need for manual parking, and improve the user experience.

[0003] To achieve automatic parking, the vehicle's steering angle needs to be determined. For example, vehicle sensors collect the vehicle's steering angle and provide it to the vehicle's driver assistance system or automatic driving system. The driver assistance system or automatic driving system then plans the vehicle's driving trajectory (i.e., the driving path during the parking process) based on the vehicle's steering angle and automatically parks the vehicle in the parking space based on the driving trajectory.

[0004] However, when vehicle sensors collect the vehicle's steering angle, there may be errors in the steering angle, meaning that an accurate and reliable vehicle steering angle cannot be obtained. As a result, when planning the vehicle's driving trajectory based on the vehicle's steering angle, an accurate and reliable vehicle driving trajectory cannot be obtained, and the safety of the vehicle driving trajectory is low. In other words, when automatically parking the vehicle based on the vehicle driving trajectory, there may be a risk of collision. Summary of the Invention

[0005] This application provides a method for determining the steering angle of a vehicle, the method comprising: Multiple candidate corner points are obtained based on the current frame panoramic image. Each candidate corner point is a pixel point located at the edge of an object in the current frame panoramic image. Specifically, for any pixel point in the current frame panoramic image, if the difference between the brightness value of N consecutive pixels and the brightness value of the pixel point is greater than a threshold on a circle with radius R centered at the pixel point, then the pixel point is considered as a candidate corner point, where both R and N are greater than 1. Based on the brightness values ​​of each pixel within the neighborhood window of a candidate corner point, the feature intensity response value of the candidate corner point is determined, and the feature intensity response value is used to quantify the intensity of the candidate corner point. A set of feature points is generated; wherein, based on the feature intensity response value of each candidate corner point, a portion of candidate corner points are selected from the plurality of candidate corner points as feature points, and the set of feature points includes a plurality of feature points. The feature point set is used to determine the first parameter, which represents the conversion relationship between the actual vehicle steering angle and the vehicle's acquired steering angle. After calibrating the first parameter for the vehicle, the vehicle's acquired steering angle is collected by the vehicle sensor, and the acquired steering angle is converted into the actual vehicle steering angle using the first parameter.

[0006] This application provides a vehicle steering angle determination device, the device comprising: The acquisition module is used to acquire multiple candidate corner points based on the current frame panoramic image. Each candidate corner point is a pixel point located at the edge of an object in the current frame panoramic image. Specifically, for any pixel point in the current frame panoramic image, if the difference between the brightness value of N consecutive pixels and the brightness value of the pixel point is greater than a threshold on a circle with the pixel point as the center, then the pixel point is regarded as a candidate corner point. The determination module is used to determine the feature intensity response value of the candidate corner point based on the brightness value of each pixel point in the neighborhood window of the candidate corner point. The feature intensity response value is used to quantify the intensity of the candidate corner point. A generation module is used to generate a set of feature points; wherein, based on the feature intensity response value of each candidate corner point, a portion of candidate corner points are selected as feature points from the plurality of candidate corner points, and the set of feature points includes a plurality of feature points; wherein, the set of feature points is used to determine a first parameter, the first parameter representing the conversion relationship between the actual steering angle of the vehicle and the collected steering angle of the vehicle; Specifically, after calibrating the first parameter for the vehicle, the vehicle's steering angle is collected by the vehicle's sensors, and the collected steering angle is converted into the vehicle's actual steering angle using the first parameter.

[0007] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the vehicle steering angle determination method of the above example of this application.

[0008] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the vehicle steering angle determination method of the above example of this application.

[0009] This application provides a machine-readable storage medium storing machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions to implement the vehicle steering angle determination method of the above example of this application.

[0010] As can be seen from the above technical solutions, in this embodiment, multiple candidate corner points are obtained based on the current frame panoramic image. Each candidate corner point is a pixel point located at the edge of an object within the current frame panoramic image. Based on the brightness value of each pixel point within the neighborhood window of the candidate corner point, the feature intensity response value of the candidate corner point is determined. A feature point set is generated based on the feature intensity response values ​​of each candidate corner point. This allows for the selection of feature points with obvious corner characteristics to form a feature point set. When determining the first parameter based on the feature point set, an accurate and reliable first parameter can be obtained. That is, when calculating the first parameter based on feature points with obvious corner characteristics, the accuracy of parameter calculation can be improved, and errors in the parameter can be avoided. Based on this, when converting the vehicle's collected steering angle into the vehicle's actual steering angle using the first parameter, the actual vehicle steering angle has no error or minimal error, resulting in an accurate and reliable actual vehicle steering angle. When planning the vehicle's driving trajectory based on the actual vehicle steering angle, an accurate and reliable vehicle driving trajectory can be obtained, resulting in a high level of safety for the vehicle driving trajectory. For example, when automatically parking the vehicle based on the vehicle driving trajectory, there is no risk of collision.

[0011] When acquiring multiple candidate corner points based on the current frame panoramic image, for any pixel in the current frame panoramic image, if the difference between the brightness value of N consecutive pixels and the brightness value of the current pixel is greater than a threshold, then the pixel is considered a candidate corner point. This fast detection method based on circumferential continuity effectively eliminates a large number of non-corner areas and significantly improves the efficiency of subsequent processing.

[0012] By determining the feature intensity response values ​​of candidate corner points, a set of feature points is generated based on these values. The feature intensity response values ​​effectively distinguish between corner points, edge regions, and flat regions, ensuring that the selected feature points possess significant local structural information and providing stable support for subsequent tracking and matching. This results in feature points being concentrated in areas of the image with obvious corner characteristics, such as vehicle contour transitions, window edges, and headlight structures. Based on this processing, accurate and reliable first parameters can be obtained. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a method for determining the vehicle steering angle in one embodiment of this application. Figure 2A This is a schematic diagram of a fisheye image in one embodiment of this application; Figure 2B This is an example of mapping an image in the camera coordinate system to an image in the world coordinate system; Figure 2C This is an example of stitching top-down sub-images into a panoramic image in one embodiment of this application; Figure 3AThis is a schematic diagram illustrating the generation of a set of feature points in one embodiment of this application; Figure 3B This is a schematic diagram of feature point detection in one embodiment of this application; Figure 3C This is a schematic diagram of feature points in a filtered image edge blind area according to one embodiment of this application; Figure 3D This is a schematic diagram of feature point following in one embodiment of this application; Figure 4 This is a schematic diagram of the parameter acquisition process in one embodiment of this application; Figure 5 This is a schematic diagram of the vehicle steering angle determining device in one embodiment of this application; Figure 6 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0014] This application proposes a method for determining the vehicle steering angle, which can be applied to electronic devices. See [link to relevant documentation]. Figure 1 The diagram shown is a flowchart illustrating the method for determining the vehicle's steering angle. This method may include: Step 101: Obtain multiple candidate corner points based on the current frame panoramic image. Each candidate corner point is a pixel point located at the edge of an object in the current frame panoramic image. Specifically, for any pixel point in the current frame panoramic image, if the difference between the brightness value of N consecutive pixels and the brightness value of the pixel point is greater than a threshold on a circle with radius R centered at the pixel point, then the pixel point is considered as a candidate corner point, where both R and N are greater than 1.

[0015] Step 102: Based on the brightness values ​​of each pixel in the neighborhood window of the candidate corner point, determine the feature intensity response value of the candidate corner point. This feature intensity response value is used to quantify the intensity of the candidate corner point.

[0016] Step 103: Generate a feature point set; wherein, based on the feature intensity response values ​​of each candidate corner point, a portion of candidate corner points are selected as feature points from multiple candidate corner points, and the feature point set may include multiple feature points. The feature point set is used to determine a first parameter, which represents the conversion relationship between the actual vehicle steering angle and the vehicle's acquired steering angle; after calibrating the first parameter for the vehicle, the vehicle's acquired steering angle is acquired through vehicle sensors, and the acquired steering angle is converted into the actual vehicle steering angle using the first parameter.

[0017] For example, determining the feature intensity response value of a candidate corner point based on the brightness values ​​of each pixel within its neighborhood window may include, but is not limited to: for any pixel within the neighborhood window, determining a first gradient value in the horizontal direction and a second gradient value in the vertical direction based on the brightness values ​​of each pixel within the neighborhood window; determining the structure tensor of the pixel based on the first and second gradient values, wherein the structure tensor is used to represent the distribution characteristics of the gradient within the neighborhood window; determining the weight value of the pixel based on the offset between the pixel and the candidate corner point, wherein the weight value is used to represent the gradient contribution near the candidate corner point; determining the gradient covariance matrix of the candidate corner point based on the weight values ​​of each pixel within the neighborhood window and the structure tensor; and determining the feature intensity response value of the candidate corner point based on the gradient covariance matrix.

[0018] For example, if the feature intensity response value of a candidate corner point is the largest feature intensity response value in its neighborhood, then the candidate corner point is used as a feature point. After generating the feature point set, target feature points can be determined based on the feature point set; wherein, multiple feature points in the feature point set are all used as target feature points; or, feature points in the feature point set located in the image edge blind zone and / or vehicle body area are filtered, and the remaining multiple feature points are used as target feature points. Pixel pairs corresponding to the target feature points are generated, and the pixel pairs include the pixel coordinates of the target feature point in the current frame panoramic image and the predicted coordinates of the target feature point in the next frame panoramic image. A first parameter is determined based on the pixel pairs corresponding to multiple target feature points.

[0019] For example, for any feature point within the feature point set, if the pixel coordinates of the feature point satisfy the following expression, then the feature point is not in the image edge blind zone; otherwise, the feature point is in the image edge blind zone: For any feature point within the feature point set, if the pixel coordinates of the feature point satisfy the following expression, then the feature point is located within the vehicle body region; otherwise, the feature point is not located within the vehicle body region: ;in, This represents the horizontal pixel coordinates of the feature point. This represents the vertical pixel coordinate of the feature point. This indicates the configured security boundary values. This indicates the width of the panoramic image in the current frame. Indicates the height of the panoramic image in the current frame. This indicates the pixel coordinates of the vehicle's center within the current frame of the panoramic image. This represents half the length of the vehicle. This indicates half the width of the vehicle.

[0020] For example, determining the first parameter based on pixel pairs corresponding to multiple target feature points may include: determining a second parameter based on pixel pairs corresponding to multiple target feature points, wherein the second parameter represents the transformation relationship between pixel coordinates in the current frame panoramic image and predicted coordinates in the next frame panoramic image; determining the actual vehicle steering angle corresponding to the current frame panoramic image based on the second parameter; generating a steering angle pair, wherein the steering angle pair includes the actual vehicle steering angle and the vehicle acquisition steering angle, wherein the vehicle acquisition steering angle is the steering angle acquired by the vehicle sensor corresponding to the current frame panoramic image; and determining the first parameter based on steering angle pairs corresponding to multiple panoramic images.

[0021] For example, determining the second parameter based on pixel pairs corresponding to multiple target feature points may include: determining the second parameter based on pixel pairs corresponding to multiple target feature points using the following formula: ;in, This indicates the predicted coordinates of the target feature point in the next frame of the panoramic image. This indicates the pixel coordinates of the target feature point in the current frame of the panoramic image. Indicates the second parameter; This represents the rotation matrix of the vehicle between the current frame and the next frame of the panoramic image. This represents the translation vector of the vehicle between the current frame panoramic image and the next frame panoramic image.

[0022] For example, determining the actual vehicle steering angle corresponding to the current frame panoramic image based on the second parameter may include: determining the actual vehicle steering angle using the following formula: ;in, Indicates the actual steering angle of the vehicle. Represents the arctangent function in the four quadrants. and This indicates the second parameter.

[0023] For example, determining the first parameter based on the steering angle pairs corresponding to multiple panoramic images may include: determining the first parameter using the following formula based on the steering angle pairs corresponding to multiple panoramic images: ;in, This indicates the actual steering angle of the vehicle at the center of the steering angle. This indicates that the steering angle of the vehicle in the steering angle pair is collected, and m represents the configured value. Represents the polynomial coefficients. This represents the fitting residual; where the first parameter includes the polynomial coefficients and the fitting residual.

[0024] As can be seen from the above technical solutions, in this embodiment, multiple candidate corner points are obtained based on the current frame panoramic image. Each candidate corner point is a pixel point located at the edge of an object within the current frame panoramic image. Based on the brightness value of each pixel point within the neighborhood window of the candidate corner point, the feature intensity response value of the candidate corner point is determined. A feature point set is generated based on the feature intensity response values ​​of each candidate corner point. This allows for the selection of feature points with obvious corner characteristics to form a feature point set. When determining the first parameter based on the feature point set, an accurate and reliable first parameter can be obtained. That is, when calculating the first parameter based on feature points with obvious corner characteristics, the accuracy of parameter calculation can be improved, and errors in the parameter can be avoided. Based on this, when converting the vehicle's collected steering angle into the vehicle's actual steering angle using the first parameter, the actual vehicle steering angle has no error or minimal error, resulting in an accurate and reliable actual vehicle steering angle. When planning the vehicle's driving trajectory based on the actual vehicle steering angle, an accurate and reliable vehicle driving trajectory can be obtained, resulting in a high level of safety for the vehicle driving trajectory. For example, when automatically parking the vehicle based on the vehicle driving trajectory, there is no risk of collision.

[0025] When acquiring multiple candidate corner points based on the current frame panoramic image, for any pixel in the current frame panoramic image, if the difference between the brightness value of N consecutive pixels and the brightness value of the current pixel is greater than a threshold, then the pixel is considered a candidate corner point. This fast detection method based on circumferential continuity effectively eliminates a large number of non-corner areas and significantly improves the efficiency of subsequent processing.

[0026] By determining the feature intensity response values ​​of candidate corner points, a set of feature points is generated based on these values. The feature intensity response values ​​effectively distinguish between corner points, edge regions, and flat regions, ensuring that the selected feature points possess significant local structural information and providing stable support for subsequent tracking and matching. This results in feature points being concentrated in areas of the image with obvious corner characteristics, such as vehicle contour transitions, window edges, and headlight structures. Based on this processing, accurate and reliable first parameters can be obtained.

[0027] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0028] This application proposes a method for determining a vehicle steering angle, which can be applied to electronic devices. These electronic devices can be in-vehicle devices or control devices for in-vehicle devices (such as personal computers, terminal devices, etc.). Taking an in-vehicle device as an example, the in-vehicle device can be an autonomous driving device (an in-vehicle device supporting an autonomous driving system) or an assisted driving device (an in-vehicle device supporting an assisted driving system). The autonomous driving device or the assisted driving device can determine the vehicle's driving trajectory based on the vehicle steering angle. The vehicle driving trajectory can be the trajectory during parking, the trajectory when leaving a parking space, or other driving trajectories. In summary, the vehicle steering angle can be applied to driving trajectories in any scenario.

[0029] Autonomous driving or driver assistance systems have automatic parking functions, such as AVP (Automated Valet Parking) or PAVP (Public Automated Valet Parking). Through these functions, the autonomous driving or driver assistance systems can automatically park the vehicle without requiring manual control from the user.

[0030] The vehicle in this embodiment is an intelligent vehicle capable of implementing assisted driving or autonomous driving functions. For example, if the vehicle is equipped with an autonomous driving device, then the vehicle is an intelligent vehicle capable of autonomous driving, and the vehicle steering angle determination method is implemented through the autonomous driving device. Alternatively, if the vehicle is equipped with an assisted driving device, then the vehicle is an intelligent vehicle capable of assisted driving, and the vehicle steering angle determination method is implemented through the assisted driving device. The vehicle may also be equipped with cameras and ultrasonic radar; in addition to cameras and ultrasonic radar, the vehicle may also be equipped with millimeter-wave radar and / or lidar, without limitation.

[0031] This application proposes a vehicle steering angle determination method, which is a vehicle steering angle calibration method based on feature point matching. The vehicle steering angle determination method involves processes such as multi-channel image stitching, feature point set generation, feature point filtering, feature point following, and parameter acquisition. These processes are described below.

[0032] First, regarding the multi-channel image stitching process.

[0033] In the process of multi-channel image stitching, multiple images (environmental images around the vehicle) from different cameras (such as multiple cameras around the vehicle) can be stitched together to obtain a panoramic view (also called a stitched image or panoramic stitched image). From the panoramic view, all-round environmental information of the vehicle can be obtained, which is beneficial for subsequent image recognition and environmental perception algorithms.

[0034] The cameras around the vehicle can be fisheye cameras, and the images captured by the fisheye cameras are fisheye images. See [link / reference]. Figure 2A The image shown is a schematic diagram of a fisheye image. For example, a fisheye image can be an image with distortion, and it can also be an image without positional and depth information.

[0035] For example, for a fisheye image, distortion correction processing can be performed to obtain a distortion-corrected image. For instance, distortion correction processing can be performed on the fisheye image based on the obtained distortion coefficients, converting the distorted locations within the fisheye image into distortion-free locations in the distortion-corrected image. For example, the following formula (1) can be used to convert the distorted locations within the fisheye image into distortion-free locations in the distortion-corrected image: Formula (1) In the above formula, This indicates the location without distortion. Indicates the location of the distortion. This represents the distortion coefficient, which can be determined based on the distance distortion coefficient and the angle distortion coefficient. and This refers to the position of the camera's optical center in the x and y directions. The following explains this process.

[0036] For example, a coefficient table can be generated in advance, which includes distance distortion coefficients. and angular distortion coefficient The mapping relationship between them. For example, distance distortion coefficient. With angle distortion coefficient The following expression applies between them: , =1, The distortion parameters are intrinsic camera parameters and can be known values. Based on this, the angular distortion coefficients can be iterated over. Multiple values ​​for the angle distortion coefficient For each value, the distance distortion coefficient is obtained through the above expression. The value of can be obtained in this way, thus yielding the distance distortion coefficient. and angular distortion coefficient The coefficient table between them.

[0037] The distance distortion coefficient can be determined in the following way. : , , , ( , This can be a camera intrinsic parameter. The distance distortion coefficient is obtained after... Then, by consulting the coefficient table, the distance distortion coefficient can be obtained. Corresponding angular distortion coefficient Based on this, the distortion coefficient can be determined using the following formula. : .

[0038] For example, after obtaining the distortion-corrected image, it can be converted to an image in the world coordinate system, that is, the image in the camera coordinate system can be mapped to an image in the world coordinate system. See [link to documentation]. Figure 2B The image shown is an example of mapping an image in the camera coordinate system to an image in the world coordinate system.

[0039] For example, to map an image from the camera coordinate system to the world coordinate system, this can be achieved through a transformation using the camera's intrinsic and extrinsic parameters. These parameters can be obtained through camera calibration, and the calibration process is not restricted. For instance, camera intrinsic parameters may include distortion parameters. , , , Center point parameters , Focal length parameters , A checkerboard calibration scheme can be used. By creating a calibration board, using images captured by the camera and obtaining the corner coordinates of the calibration board, and by minimizing the projection error, the camera intrinsic parameters can be obtained by calculating the feature points of multiple images.

[0040] For example, camera extrinsic parameters may include camera position parameters. and the camera's pose angle parameters The camera extrinsic parameters are calibrated using a joint calibration scheme. The vehicle is driven in a specific field, and a corner point extraction algorithm is used to perform preliminary calibration of the single camera. Then, an optical flow fine-tuning method is used to perform fusion calibration of the camera extrinsic parameters to obtain the optimal value with the minimum reprojection error.

[0041] Based on the camera intrinsic and extrinsic parameters, the image after distortion removal in the camera coordinate system can be converted into an image in the world coordinate system using the following formula (2), which is denoted as the projected image in the world coordinate system.

[0042] Formula (2) K represents the camera intrinsic parameters. T represents the camera position parameters. R represents the attitude angle parameter. The camera position parameters and attitude angle parameters represent the camera's extrinsic parameters. This refers to the camera's depth parameters. When projecting the image onto the ground, It is 0. , ) represents the pixel coordinates within the image after distortion correction. , () represents the pixel coordinates within the projected image in the world coordinate system.

[0043] In addition, the homography matrix H can be obtained. The homography matrix H is determined based on the camera intrinsic and extrinsic parameters. Based on the homography matrix H, the image after distortion in the camera coordinate system can be converted into a projected image in the world coordinate system using the following formula (3).

[0044] Formula (3) For example, after converting the distortion-corrected image into a projected image in the world coordinate system, multiple projected images can be stitched together to obtain a stitched panoramic image. For instance, the above operation can be repeated for four cameras (front, left, right, and rear) to obtain their respective top-view sub-images (i.e., projected images). These top-view sub-images are then placed together according to their positions in the world coordinate system to stitch together a panoramic top-view view, hereinafter referred to as the panoramic image. See also... Figure 2C The image shown is an example of stitching together top-down sub-images into a panoramic image.

[0045] Second, regarding the process of generating feature point sets.

[0046] During the feature point set generation process, multiple feature points can be extracted from the current frame panoramic image (the currently acquired panoramic image is denoted as the current frame panoramic image). Feature points are pixels with significant visual characteristics in the current frame panoramic image, which can be stably detected and reliably matched between different panoramic images.

[0047] For example, see Figure 3A The diagram shown illustrates the generation of a set of feature points, a process that may include: Step 301: Detect corner points based on the current frame panoramic image to obtain multiple candidate corner points; where each candidate corner point can be a pixel point located at the edge of an object in the current frame panoramic image.

[0048] For example, for any pixel point (denoted as pixel point) within the panoramic image of the current frame. In a circle with radius R centered at pixel p, if the difference between the brightness values ​​of N consecutive pixels and the brightness value of pixel p is greater than a threshold (configurable according to actual needs), then pixel p is considered a candidate corner point, where R and N are both positive integers greater than 1. Otherwise, pixel p is not considered a candidate corner point.

[0049] For example, on a circle with a radius of 3 centered at pixel p, if continuous If the difference between the brightness values ​​of 9, 10, 11, 12, etc., pixels and the brightness value of pixel p is greater than the threshold t, then pixel p is considered a candidate corner point, which can be expressed by the following formula: In the above formula, This represents the brightness value of pixel p. This represents the brightness value of the i-th pixel on the circle. It represents the i-th pixel on a circle with a radius of 3, excluding pixels inside the circle.

[0050] For example, the corner detection method mentioned above is a fast detection method based on circumferential continuity. By using fast detection based on circumferential continuity, a large number of non-corner areas can be effectively eliminated, significantly improving the efficiency of subsequent processing.

[0051] Step 302: For any candidate corner point, based on the brightness values ​​of each pixel within the neighborhood window of the candidate corner point, determine the feature intensity response value of the candidate corner point. This feature intensity response value is used to quantify the intensity of the candidate corner point. For example, this feature intensity response value can effectively distinguish corner points, edges, and flat areas, ensuring that the selected feature points have significant local structural information, providing stable support for subsequent tracking and matching.

[0052] For example, the characteristic intensity response value of the candidate corner point can be determined using the following steps: Step S11: Determine the neighborhood window (local window) of the candidate corner point. This neighborhood window is a rectangular or circular window centered on the candidate corner point, hereinafter referred to as the neighborhood window W. The size of the neighborhood window W is much smaller than the size of the current frame panoramic image, and it is used to calculate gradient statistics.

[0053] Step S12: For any pixel within the neighborhood window W, based on the brightness values ​​of each pixel within the neighborhood window W, determine the first gradient value in the horizontal direction and the second gradient value in the vertical direction for that pixel.

[0054] For example, taking a pixel (x, y) within a neighborhood window W, based on the brightness values ​​of each pixel within the neighborhood window W, the first gradient value of pixel (x, y) in the horizontal direction (x direction) can be determined. And determine the second gradient value of the pixel (x, y) in the vertical direction (y direction). .

[0055] Step S13: For any pixel within the neighborhood window W, determine the structure tensor of the pixel based on the first gradient value and the second gradient value. The structure tensor is used to represent the distribution characteristics of the gradient within the neighborhood window.

[0056] For example, based on the first gradient value and the second gradient value of pixel (x, y), the structure tensor of pixel (x, y) can be determined using the following formula: In the above formula, It can represent a structure tensor. It can represent the square of the horizontal gradient. It can represent the square of the vertical gradient. It can represent the product of the horizontal and vertical gradients.

[0057] Step S14: For any pixel within the neighborhood window W, determine the weight value of the pixel based on the offset between the pixel and the candidate corner point. The weight value is used to represent the gradient contribution near the candidate corner point.

[0058] Compare the pixel (x, y) with the candidate corner point The offset is denoted as Assuming pixel (x, y) is (10, 10), this candidate corner point If it is (9, 9), then the offset is... It is (1, 1). Assuming the pixel (x, y) is (8, 8), then the offset is... It is (-1, -1), and so on.

[0059] For example, a Gaussian weighting function can be pre-configured. There are no restrictions on this Gaussian weighting function; its input is the offset, and its output is the weight value, which is used to represent the candidate corner point. The gradient contribution from the vicinity. Based on this, by adjusting the offset... Substituting the values ​​into the Gaussian weighting function, we can obtain the weight value of the pixel (x, y), denoted as the weight value. .

[0060] Step S15: Determine the gradient covariance matrix of the candidate corner point based on the weight values ​​of each pixel in the neighborhood window and the structure tensor of each pixel in the neighborhood window.

[0061] For example, the following formula (4) can be used to determine the candidate corner points. Gradient covariance matrix: Formula (4) In the above formula, (x, y) represents any pixel within the neighborhood window. This represents the weight value of the pixel (x, y). The structure tensor representing the pixel (x, y) Indicates candidate corner points The gradient covariance matrix is ​​used to represent the superposition of weight tensions.

[0062] Step S16: Determine the feature intensity response value of the candidate corner point based on the gradient covariance matrix.

[0063] For example, candidate corner points can be determined using the following formula (5). Characteristic intensity response values: Formula (5) In formula (5), Indicates candidate corner points The characteristic intensity response value is used to quantify the corner intensity. The determinant of the gradient covariance matrix is ​​used to reflect the degree of dispersion of the gradient distribution. The trace represents the gradient covariance matrix, used to reflect the total gradient strength. This represents a configured constant value, which can be configured empirically, such as 0.04~0.06, to balance the response of corners and edges.

[0064] In summary, for each candidate corner point, the above steps can be used to obtain the feature intensity response value of that candidate corner point, that is, to obtain the feature intensity response value of each candidate corner point in the current frame panoramic image.

[0065] Step 303: Based on the feature intensity response values ​​of each candidate corner point, select a portion of candidate corner points as feature points from multiple candidate corner points. For any candidate corner point, if the feature intensity response value of the candidate corner point is the maximum feature intensity response value in its neighborhood, then the candidate corner point is selected as a feature point. If the feature intensity response value of the candidate corner point is not the maximum feature intensity response value in its neighborhood, then the candidate corner point is not selected as a feature point.

[0066] For example, after obtaining the feature intensity response values ​​of all candidate corner points, non-maximum suppression can be performed to avoid feature point clustering and improve distribution uniformity. For each candidate corner point, a neighborhood (such as a rectangular or circular region centered on the candidate corner point) can be determined, which may include multiple candidate corner points. Based on this, if the feature intensity response value of the candidate corner point is the largest feature intensity response value in the neighborhood, then the candidate corner point is selected as a feature point. If the feature intensity response value of the candidate corner point is not the largest feature intensity response value in the neighborhood, then the candidate corner point is not selected as a feature point. Based on the above processing, a portion of the candidate corner points can be selected as feature points from all candidate corner points.

[0067] For example, after obtaining the feature intensity response values ​​of all candidate corner points, only within the candidate point set... Local maxima suppression is used to obtain the final feature points. Assume the candidate point set is... The response function is The neighborhood function is (express The set of all points in the neighborhood of the local maximum suppression (Local maximum suppression) can be expressed as: .

[0068] For example, Represents the set of all candidate corner points, i.e. Represents n candidate corner points, This represents the characteristic intensity response value of the candidate corner point p. Indicates candidate corner points The set of all candidate corner points in the neighborhood of a given point, such as candidate corner points The set of all candidate corner points within a 10x10 window. This represents the final set of feature points retained after nonmaximum suppression, requiring that the response value of each point is the maximum in its neighborhood.

[0069] Step 304: Generate a set of feature points, which may include multiple feature points.

[0070] For example, see Figure 3B As shown, this is a schematic diagram of feature point detection. After corner detection, feature intensity response value calculation and non-maximum suppression, the final feature points are concentrated in areas with obvious corner characteristics in the image, such as key locations like vehicle outline turning points, window edges, and headlight structures.

[0071] Third, the feature point filtering process.

[0072] After obtaining the feature point set, target feature points can be determined based on the feature point set. For example, all feature points in the feature point set can be used as target feature points. Alternatively, feature points in the feature point set located in the image edge blind zone can be filtered, and the remaining feature points can be used as target feature points. Or, feature points in the feature point set located in the vehicle body area can be filtered, and the remaining feature points can be used as target feature points. Or, feature points in both the image edge blind zone and the vehicle body area can be filtered, and the remaining feature points can be used as target feature points.

[0073] For example, for filtering blind spots at image edges, see [link to relevant documentation]. Figure 3C The diagram illustrates the filtering of feature points in blind spots at image edges. Feature points that are close to the image edge need to be removed.

[0074] For example, for image edge blind spot filtering, feature points near the image edge are filtered out by setting a safety boundary value M. The safety boundary value M is configured according to actual needs, such as determining the safety boundary value M based on the width and height of the current frame panoramic image. For instance, the safety boundary value M can be determined based on the minimum value of the width and height of the current frame panoramic image. .

[0075] For example, for any feature point in the feature point set, if the pixel coordinates of the feature point satisfy the following expression, it means that the feature point is not in the image edge blind zone (i.e., the feature point is in the effective image area), and the feature point needs to be retained; otherwise, if the pixel coordinates of the feature point do not satisfy the following expression, it means that the feature point is in the image edge blind zone, and the feature point needs to be filtered.

[0076] ; Within the above expression, This represents the horizontal pixel coordinates of the feature point. This represents the vertical pixel coordinate of the feature point. This indicates the configured security boundary values. This indicates the width (in pixels) of the panoramic image in the current frame. This represents the height (in pixels) of the current frame's panoramic image. In summary, for the effective region condition, feature points need to satisfy the following inequalities to avoid feature points being located in the image edge blind zone: The feature point is at least [distance] from the left boundary of the current frame panoramic image. Pixel. The feature point is at least [distance] from the right boundary of the current frame panoramic image. Pixel. The feature point is at least [distance] from the upper boundary of the current frame panoramic image. Pixel. The feature point is at least [distance] from the lower boundary of the current frame panoramic image. Pixel.

[0077] For example, for vehicle body region filtering, the projection of the vehicle body in the current frame panoramic image can be approximated as a shape centered at the image center. Given a polygon (such as a rectangle), let the half-length of the rectangle be... The half-width of the rectangle is If a feature point falls within the rectangle, it means that the feature point falls within the vehicle body area and needs to be filtered out. If a feature point does not fall within the rectangle, it means that the feature point does not fall within the vehicle body area.

[0078] For example, for any feature point in the feature point set, if the pixel coordinates of the feature point satisfy the following expression, it means that the feature point is in the vehicle body area and needs to be filtered; otherwise, if the pixel coordinates of the feature point do not satisfy the following expression, it means that the feature point is not in the vehicle body area.

[0079] ; Within the above expression, This represents the horizontal pixel coordinates of the feature point. This represents the vertical pixel coordinate of the feature point. This indicates the pixel coordinates of the vehicle's center within the current frame of the panoramic image. This represents half the length of the vehicle. This represents half the width of the vehicle. Feature points that satisfy the above expression are filtered out, while feature points that do not satisfy the above expression are retained. , and / or .

[0080] Fourth, regarding the feature point following process. In the feature point following process, a correspondence is established between feature points in consecutive panoramic images, following the positional changes of the feature points over time. In this embodiment, after obtaining multiple target feature points, for any given target feature point, a corresponding pixel pair is generated. This pixel pair may include the pixel coordinates of the target feature point in the current frame of the panoramic image and the predicted coordinates of the target feature point in the next frame of the panoramic image. Obviously, the pixel coordinates of the target feature point in the current frame of the panoramic image are known. Regarding the predicted coordinates of the target feature point in the next frame of the panoramic image, convolutional optical flow can be used to follow the feature points to obtain the predicted coordinates, or other methods can be used to obtain these predicted coordinates; there are no restrictions on this.

[0081] For example, the optical flow corresponding to the target feature point can be determined (the optical flow between the pixel coordinates of the target feature point in the current frame panoramic image and the predicted coordinates of the target feature point in the next frame panoramic image), and the predicted coordinates of the target feature point can be determined based on this optical flow and the pixel coordinates of the target feature point.

[0082] For example, see the following formula for an example of determining this optical flow: In the above formula, Representing target feature points The pixel coordinates corresponding to the panoramic image in the current frame. This represents the optical flow corresponding to the target feature point. Representing target feature points The neighborhood window (local window) is a window that uses the target feature point as its reference. A rectangular or circular window centered on the object. For example, a neighborhood window. The size of the neighborhood window is much smaller than the size of the current frame's panoramic image. It is the radius The circular window , It is a configured scale factor, such as 2, 3, etc., used to control the ratio between the window size and the feature scale. It is the pre-configured feature point scale, which can be configured according to requirements.

[0083] For neighborhood windows Any pixel (x, y) within the target feature point, and the pixel (x, y) within the target feature point. pixel coordinates The offset is denoted as , This represents the weight value corresponding to the pixel (x, y). For example, a weight function can be pre-configured; there are no restrictions on this function. The input to the weight function is the offset, and the output is the weight value. Based on this, by adjusting the offset... Substituting the values ​​into the weight function, we can obtain the weight value of the pixel (x, y), denoted as the weight value. .

[0084] For example, an example of this weighting function could be: . It can be a Gaussian standard deviation, used to control the rate of weight decay, and can be set to... or It can be configured according to actual needs. Represents the neighborhood window The radius. This indicates the offset.

[0085] For example, The gradient structure tensor representing the pixel (x, y) ,Right now This represents the horizontal pixel coordinates of a pixel (x, y). Represents the vertical pixel coordinates of a pixel (x, y). , It can represent the square of the horizontal gradient. It can represent the square of the vertical gradient. It can represent the product of the horizontal and vertical gradients. This represents the first gradient value of the pixel (x, y) in the horizontal direction. This represents the second gradient value of the pixel (x, y) in the vertical direction.

[0086] In determining the gradient structure tensor of pixel (x, y) The calculation is performed based on the next frame of the panoramic image, which is the current frame's panoramic image. This represents the first gradient value of pixel (x, y) in the next frame of the panoramic image. This represents the second gradient value of pixel (x, y) in the next frame of the panoramic image.

[0087] For example, The gradient parameters of pixel (x, y) , . This represents the first gradient value of pixel (x, y) in the next frame of the panoramic image. This represents the second gradient value of pixel (x, y) in the next frame of the panoramic image. This represents the gradient value of the brightness difference of pixel (x, y) between the current frame and the next frame of the panoramic image. For example, it can be based on a neighborhood window centered on pixel (x, y) within the current frame of the panoramic image. The brightness value of each pixel within the image, and the neighborhood window centered at pixel (x, y) in the next frame of the panoramic image. The brightness values ​​of each pixel within the range are used to calculate the gradient value of the difference between these brightness values. This gradient value is then used as the gradient of the (x, y) pixel. .

[0088] In summary, the optical flow corresponding to the target feature point can be calculated using the above formula. Then, the following formula can be used to calculate. Obviously, Representing target feature points The pixel coordinates corresponding to the panoramic image in the current frame. Representing target feature points The pixel coordinates corresponding to the next frame of the panoramic image. Each time the current frame of the panoramic image is obtained, feature point tracking can be performed based on the above method, see [link to relevant documentation]. Figure 3D The diagram shown is a schematic of feature point following.

[0089] Fifth, regarding the parameter acquisition process. In the parameter acquisition process, the first and second parameters can be obtained based on multiple pixel pairs. The acquisition process for the first and second parameters is explained below.

[0090] For example, see Figure 4 The diagram shown illustrates the parameter acquisition process, which may include: Step 401: Determine the second parameter based on the pixel pairs corresponding to multiple target feature points. The second parameter represents the transformation relationship between the pixel coordinates in the current frame panoramic image and the predicted coordinates in the next frame panoramic image.

[0091] For example, based on multiple pixel pairs, the second parameter can be determined using the following formula (6): Formula (6) In formula (6), This indicates the predicted coordinates of the target feature point in the next frame of the panoramic image. This indicates the pixel coordinates of the target feature point in the current frame of the panoramic image. Indicates the second parameter; This represents the rotation matrix of the vehicle between the current frame and the next frame of the panoramic image. This represents the translation vector of the vehicle between the current frame panoramic image and the next frame panoramic image.

[0092] The definition of formula (6) and the solution process of formula (6) will be explained below.

[0093] Affine transformation models are matrices used to describe affine transformations between images, including linear transformations such as translation, rotation, scaling, and shearing. Let the homogeneous coordinates of the target feature point in the current frame of the panoramic image be... The homogeneous coordinates of the target feature point in the next frame of the panoramic image are: The homogeneous terms of homogeneous coordinates are used to facilitate matrix operations. The relationship between the two is defined by a... affine transformation matrix Given: Based on this, the relationship shown in formula (6) can be obtained. For rigid body motion, the upper left corner... The submatrix is ​​a rotation matrix : Thus, the complete Affine matrix for: In addition, affine matrices It can also be expressed as: . affine transformation matrix , used to describe rigid body motion between two frames.

[0094] Regarding the parameter solving process, for a set of pixel pairs , , represents the index of the target feature point, and each pixel pair represents the positional change of the same feature point between two frames.

[0095] The affine matrix can be solved using the least squares method. The transformation equation has 6 parameters. The expansion is shown below: , So, for Target feature points, based on The N pixel pairs corresponding to each target feature point can be used to construct an overdetermined system of linear equations: ; , is a set of pixel coordinates within the current frame of the panoramic image. , is the parameter vector to be solved, and is the second parameter. It is the observation vector, which represents the pixel coordinates within the next frame of the panoramic image.

[0096] Solve using the least squares method: , yes The matrix, determined by the pixel coordinates within the current frame of the panoramic image in the pixel pair, is used for linear regression, and each two rows correspond to a target feature point. and The direction equation. It is an observation vector, determined by the pixel coordinates within the next frame of the panoramic image in the pixel pair. The parameter vector to be determined can be solved based on X and Y. The parameter vector to be determined Includes all six parameters of the affine transformation matrix. This represents the least squares solution. express Information matrix, Represents the inverse of the information matrix. express The projection vector. Step 402: Determine the actual vehicle steering angle corresponding to the current frame panoramic image based on the second parameter.

[0097] For example, the second parameter may include Based on the second parameter, the actual steering angle of the vehicle is determined using the following formula: .in, Indicates the actual steering angle of the vehicle. This represents the arctangent function in the four quadrants, and its output range is... , and This indicates the second parameter.

[0098] For example, when solving for the affine transformation matrix Then, from the affine transformation matrix The top left corner submatrix The actual steering angle of the vehicle, also known as the actual yaw angle, is extracted from this. The actual steering angle is used to represent the vehicle's motion attitude. For example, due to the affine transformation matrix... The top left corner The submatrix is ​​a standard rotation matrix, and the actual steering angle of the vehicle can be calculated using the arctangent function. Furthermore, This represents the translation vector, derived from the affine transformation matrix. The last two columns represent the actual translational motion of the vehicle.

[0099] Step 403: Generate a steering angle pair, which includes the vehicle's actual steering angle and the vehicle's acquired steering angle. The vehicle's acquired steering angle is the steering angle collected by the vehicle's sensors that corresponds to the current frame panoramic image.

[0100] For example, when the second panoramic image is acquired, the second panoramic image is used as the current panoramic image, and the actual vehicle steering angle corresponding to the second panoramic image is determined based on the above steps. Furthermore, the vehicle's steering angle corresponds to the second frame of the panoramic image acquired by the vehicle's sensors at the current moment. Vehicle steering angle collection It can also be called the vehicle signal angle, or the vehicle's actual steering angle. and vehicle steering angle Forming a steering angle pair. When the third frame of the panoramic image is acquired, the third frame of the panoramic image is used as the current frame of the panoramic image. Based on the above steps, the actual vehicle steering angle corresponding to the third frame of the panoramic image is determined. Furthermore, the vehicle's steering angle corresponds to the third frame of the panoramic image acquired by the vehicle's sensors at the current moment. Actual steering angle of the vehicle and vehicle steering angle This forms a steering angle pair. By analogy, multiple steering angle pairs can be obtained, for example, the vehicle's actual steering angle: Vehicle steering angle data collection: n represents the frame number (panoramic image starting from the second frame). .

[0101] Step 404: Determine a first parameter based on the steering angle pairs corresponding to multiple panoramic images. This first parameter can represent the conversion relationship between the actual steering angle of the vehicle and the steering angle collected by the vehicle.

[0102] For example, based on multiple steering angle pairs, the first parameter can be determined using the following formula (7): Formula (7) In formula (7), This indicates the actual steering angle of the vehicle at the center of the steering angle. This indicates that the steering angle of the vehicle in the steering angle pair is collected, and m represents the configured value, such as 2, 3, 4, 5, 6, etc. Represents the polynomial coefficients. This represents the fitting residual. The polynomial coefficients and fitting residuals represent the first parameter.

[0103] The definition of formula (7) and the solution process of formula (7) will be explained below.

[0104] For example, a polynomial can be used to fit the actual steering angle of a vehicle. Vehicle steering angle The conversion relationship between them, such as using Polynomial fitting of the actual vehicle steering angle Vehicle steering angle The conversion relationship between them is shown in formula (7), which is: An example of a polynomial fitting transformation relationship.

[0105] In formula (7), Denotes the coefficients of the polynomial to be solved. This represents the fitting residual to be solved. The following explains the process of solving for the polynomial coefficients and the fitting residual.

[0106] For example, substituting the data points into the matrix equation yields: , In this way, a matrix can be designed. for: , , Based on this, the coefficient vector is solved using the least squares method. , .

[0107] In summary, the polynomial coefficients can be obtained. and fitting residuals polynomial coefficients and fitting residuals The first parameter represents the conversion relationship between the actual steering angle of the vehicle and the collected steering angle of the vehicle, i.e., the polynomial coefficients in formula (7). and fitting residuals Given the known values, we obtain the conversion relationship between the actual steering angle of the vehicle and the collected steering angle of the vehicle.

[0108] For example, see Table 1, which shows the actual steering angle of the vehicle. and vehicle steering angle Using polynomial fitting to fit the actual steering angle of the vehicle Vehicle steering angle The transformation relationship is used to obtain the polynomial coefficients. and fitting residuals The first-order linear fitting result is as follows: .

[0109] Table 1 For example, after obtaining the first parameter, the first parameter can be calibrated for the vehicle, such as calibrating the first-order linear fitting result: Based on this, each time the vehicle's steering angle is collected by the vehicle sensors, it can be converted into the actual vehicle steering angle using the first parameter. For example, the actual vehicle steering angle can be obtained by substituting the collected steering angle into the first-order linear fitting result.

[0110] For example, after calibrating the first parameter for the vehicle, it is not necessary to repeat the calibration of the first parameter, or the above steps can be used periodically to calibrate the first parameter for the vehicle, without any restriction.

[0111] In one possible implementation, all data involved in this embodiment is obtained and used only with the knowledge and authorization of the relevant users, and this embodiment does not impose any restrictions on this data.

[0112] As can be seen from the above technical solutions, in this embodiment, feature points with obvious corner characteristics can be selected to form a feature point set. When determining the first parameter based on the feature point set, an accurate and reliable first parameter can be obtained. That is, when calculating the first parameter based on feature points with obvious corner characteristics, the accuracy of parameter calculation can be improved, and parameter errors can be avoided. When converting the vehicle's acquired steering angle into the vehicle's actual steering angle through the first parameter, the actual vehicle steering angle has no error or the error is minimal, and an accurate and reliable actual vehicle steering angle can be obtained. The above method is a lightweight, robust, and highly interpretable vision method used to estimate the vehicle's actual turning behavior and establish a mapping relationship with the on-board signal system (used to output the vehicle's acquired steering angle), improving system safety and verifiability. Feature-based matching and following are organically combined to form a complementary vehicle motion analysis system; a joint calibration model of vehicle motion physical quantities (vehicle's actual steering angle) and vehicle electronic signals (CAN bus data, vehicle's acquired steering angle) is established.

[0113] Based on the same concept as the above method, this application proposes a vehicle steering angle determination device, see [link to relevant documentation]. Figure 5 The diagram shown is a structural schematic of the device, which may include: The acquisition module 51 is used to acquire multiple candidate corner points based on the current frame panoramic image. Each candidate corner point is a pixel point located at the edge of an object in the current frame panoramic image. Specifically, for any pixel point in the current frame panoramic image, if the difference between the brightness value of N consecutive pixels and the brightness value of the pixel point is greater than a threshold on a circle with radius R centered on the pixel point, then the pixel point is considered as a candidate corner point. The determination module 52 is used to determine the feature intensity response value of the candidate corner point based on the brightness value of each pixel point in the neighborhood window of the candidate corner point. The feature intensity response value is used to quantify the intensity of the candidate corner point. The generation module 53 is used to generate a feature point set; wherein, based on the feature intensity response value of each candidate corner point, a portion of the candidate corner points are selected as feature points from the plurality of candidate corner points, and the feature point set includes a plurality of feature points; wherein, the feature point set is used to determine a first parameter, the first parameter representing the conversion relationship between the actual steering angle of the vehicle and the collected steering angle of the vehicle; Specifically, after calibrating the first parameter for the vehicle, the vehicle's steering angle is collected by the vehicle's sensors, and the collected steering angle is converted into the vehicle's actual steering angle using the first parameter.

[0114] For example, when determining the feature intensity response value of a candidate corner point based on the brightness values ​​of each pixel within the neighborhood window, the determining module 52 specifically performs the following steps: for any pixel within the neighborhood window, based on the brightness values ​​of each pixel within the neighborhood window, determine the first gradient value in the horizontal direction and the second gradient value in the vertical direction of the pixel; determine the structure tensor of the pixel based on the first gradient value and the second gradient value, the structure tensor being used to represent the distribution characteristics of the gradient within the neighborhood window; determine the weight value of the pixel based on the offset between the pixel and the candidate corner point, the weight value being used to represent the gradient contribution near the candidate corner point; determine the gradient covariance matrix of the candidate corner point based on the weight values ​​of each pixel within the neighborhood window and the structure tensor, and determine the feature intensity response value of the candidate corner point based on the gradient covariance matrix.

[0115] For example, if the feature intensity response value of a candidate corner point is the maximum feature intensity response value in its neighborhood, then the candidate corner point is used as a feature point; the determining module 52 is further configured to determine target feature points based on the feature point set; wherein, multiple feature points in the feature point set are all used as target feature points; or, feature points in the feature point set located in the image edge blind zone and / or vehicle body area are filtered, and the remaining multiple feature points are used as target feature points; a pixel pair corresponding to the target feature point is generated, the pixel pair including the pixel coordinates of the target feature point in the current frame panoramic image and the predicted coordinates of the target feature point in the next frame panoramic image; a first parameter is determined based on the pixel pairs corresponding to multiple target feature points.

[0116] For example, for any feature point within the set of feature points, if the pixel coordinates of the feature point satisfy the following expression, then the feature point is not in the image edge blind zone; otherwise, the feature point is in the image edge blind zone: For any feature point within the set of feature points, if the pixel coordinates of the feature point satisfy the following expression, then the feature point is located in the vehicle body area; otherwise, the feature point is not located in the vehicle body area: ;in, This represents the horizontal pixel coordinates of the feature point. This represents the vertical pixel coordinate of the feature point. This indicates the configured security boundary values. This indicates the width of the panoramic image in the current frame. This indicates the height of the current frame panoramic image. This indicates the pixel coordinates of the vehicle's center within the current frame of the panoramic image. This represents half the length of the vehicle. This indicates half the width of the vehicle.

[0117] For example, when the determining module 52 determines the first parameter based on pixel pairs corresponding to multiple target feature points, it is specifically used to: determine a second parameter based on the pixel pairs corresponding to the multiple target feature points, wherein the second parameter represents the transformation relationship between pixel coordinates in the current frame panoramic image and predicted coordinates in the next frame panoramic image; determine the actual vehicle steering angle corresponding to the current frame panoramic image based on the second parameter; generate a steering angle pair, wherein the steering angle pair includes the actual vehicle steering angle and the vehicle acquisition steering angle, wherein the vehicle acquisition steering angle is the steering angle acquired by the vehicle sensor corresponding to the current frame panoramic image; and determine the first parameter based on the steering angle pairs corresponding to multiple panoramic images.

[0118] For example, when determining the second parameter based on pixel pairs corresponding to multiple target feature points, the determining module 52 is specifically used to: determine the second parameter based on multiple pixel pairs using the following formula: ;in, This indicates the predicted coordinates of the target feature point in the next frame of the panoramic image. This indicates the pixel coordinates of the target feature point in the current frame of the panoramic image. Indicates the second parameter; This represents the rotation matrix of the vehicle between the current frame and the next frame of the panoramic image. This represents the translation vector of the vehicle between the current frame panoramic image and the next frame panoramic image.

[0119] When determining the actual vehicle steering angle corresponding to the current frame panoramic image based on the second parameter, the determining module 52 specifically uses the following formula to determine the actual vehicle steering angle: ; Indicates the actual steering angle of the vehicle. Represents the arctangent function in the four quadrants. and This indicates the second parameter.

[0120] For example, when determining the first parameter based on the steering angle pairs corresponding to multiple panoramic images, the determining module 52 is specifically used to: determine the first parameter based on the steering angle pairs corresponding to the multiple panoramic images using the following formula: ;in, This indicates the actual steering angle of the vehicle in the steering angle pair. This indicates the vehicle's collected steering angle in the steering angle pair, where m represents the configured value. Represents the polynomial coefficients. The first parameter represents the fitting residual; wherein the first parameter includes the polynomial coefficients and the fitting residual.

[0121] Based on the same application concept as the above method, this application proposes an electronic device, see [link to application]. Figure 6 As shown, the electronic device includes a processor 61 and a machine-readable storage medium 62, the machine-readable storage medium 62 storing machine-executable instructions that can be executed by the processor 61; the processor 61 is used to execute the machine-executable instructions to implement the vehicle steering angle determination method disclosed in the above example of this application.

[0122] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the vehicle steering angle determination method disclosed in the above examples of this application.

[0123] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0124] Based on the same concept as the method described above, this application also provides a computer program product, which may include a computer program. When executed by a processor, the computer program implements the vehicle steering angle determination method disclosed in the above examples of this application.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A vehicle steering angle determination method characterized by, The method comprises: obtaining a plurality of candidate corner points based on a current frame panoramic image, each candidate corner point being a pixel point at an object edge in the current frame panoramic image; wherein, for any pixel point in the current frame panoramic image, if the difference between the luminance value of the pixel point and the luminance value of the continuous N pixel points on the circle with the pixel point as the center and a radius R is greater than a threshold value, the pixel point is a candidate corner point, R and N are both greater than 1; determining the feature intensity response value of the candidate corner point based on the luminance value of each pixel point in the neighborhood window of the candidate corner point, the feature intensity response value being used to quantify the intensity of the candidate corner point; generating a feature point set; wherein, based on the feature intensity response value of each candidate corner point, part of the candidate corner points are selected from the plurality of candidate corner points as feature points, and the feature point set comprises a plurality of feature points; wherein, the feature point set is used to determine a first parameter, the first parameter representing the conversion relationship between the actual steering angle of the vehicle and the collection steering angle of the vehicle; after calibrating the first parameter for the vehicle, the collection steering angle of the vehicle is collected through the vehicle sensor, and the collection steering angle of the vehicle is converted into the actual steering angle of the vehicle through the first parameter.

2. The method of claim 1, wherein, The method comprises: determining the feature intensity response value of the candidate corner point based on the luminance value of each pixel point in the neighborhood window of the candidate corner point, the feature intensity response value being used to quantify the intensity of the candidate corner point; for any pixel point in the neighborhood window, determining the first gradient value in the horizontal direction and the second gradient value in the vertical direction of the pixel point based on the luminance value of each pixel point in the neighborhood window; determining the structure tensor of the pixel point based on the first gradient value and the second gradient value, the structure tensor being used to represent the distribution characteristics of the gradient in the neighborhood window; determining the weight value of the pixel point based on the offset between the pixel point and the candidate corner point, the weight value being used to represent the gradient contribution near the candidate corner point; 3. The method of claim 1, wherein, determining the gradient covariance matrix of the candidate corner point based on the weight value and the structure tensor of each pixel point in the neighborhood window, and determining the feature intensity response value of the candidate corner point based on the gradient covariance matrix. If the feature intensity response value of the candidate corner point is the maximum feature intensity response value in the neighborhood, the candidate corner point is a feature point. After generating the feature point set, the method further comprises: determining a target feature point based on the feature point set; wherein, each of the plurality of feature points in the feature point set is a target feature point; or, the feature points in the feature point set that are in the image edge blind area and / or the vehicle body area are filtered, and the remaining plurality of feature points after filtering are target feature points; generating a pixel pair corresponding to the target feature point, the pixel pair comprising the pixel coordinates of the target feature point in the current frame panoramic image and the predicted coordinates of the target feature point in the next frame panoramic image; determining the first parameter based on the pixel pairs corresponding to the plurality of target feature points.

4. The method of claim 3, wherein, for any feature point in the feature point set, if the pixel coordinates of the feature point satisfy the following expression, the feature point is not in the image edge blind area, otherwise, the feature point is in the image edge blind area: For any feature point in the feature point set, if the pixel coordinates of the feature point satisfy the following expression, the feature point is in the vehicle body region, otherwise, the feature point is not in the vehicle body region: wherein, represents a horizontal pixel coordinate of the feature point, represents a vertical pixel coordinate of the feature point, represents a configured safety margin value, represents a width of the current frame panoramic image, represents a height of the current frame panoramic image, represents a corresponding pixel coordinate of the vehicle body center within the current frame panoramic image, represents a half of the vehicle length, represents a half of the vehicle width.

5. The method of claim 3, wherein, The first parameter is determined based on the pixel pairs corresponding to the plurality of target feature points, including: A second parameter is determined based on the pixel pairs corresponding to the plurality of target feature points, the second parameter representing a conversion relationship between a pixel coordinate in the current panoramic image and a predicted coordinate in the next panoramic image; A vehicle actual steering angle corresponding to the current panoramic image is determined based on the second parameter; A steering angle pair is generated, the steering angle pair including the vehicle actual steering angle and a vehicle collection steering angle, the vehicle collection steering angle being a steering angle corresponding to the current panoramic image collected by a vehicle sensor; The first parameter is determined based on the steering angle pairs corresponding to the plurality of panoramic images.

6. The method of claim 5, wherein, The second parameter is determined based on the pixel pairs corresponding to the plurality of target feature points, and the second parameter is determined based on the pixel pairs corresponding to the plurality of target feature points by using the following formula: ; wherein, represents a predicted coordinate of the target feature point corresponding to a next frame panoramic image, represents a pixel coordinate of the target feature point corresponding to a current frame panoramic image, represents the second parameter; represents a rotation matrix of the vehicle between the current frame panoramic image and the next frame panoramic image, represents a translation vector of the vehicle between the current frame panoramic image and the next frame panoramic image. The determining the actual turning angle of the vehicle corresponding to the panoramic image pair of the current frame based on the second parameter comprises: determining the actual turning angle of the vehicle by using the following formula: ; wherein, represents the actual turning angle of the vehicle, represents a four-quadrant inverse tangent function, and represents the second parameter.

7. The method of claim 5, wherein, The first parameter is determined based on the steering angle pairs corresponding to the plurality of panoramic images, including: The first parameter is determined based on the steering angle pairs corresponding to the plurality of panoramic images using the following formula: wherein represents the actual steering angle of the vehicle in the pair of steering angles, represents the collected steering angle of the vehicle in the pair of steering angles, m represents a configured number of values, represents a polynomial coefficient, represents a fitting residual; wherein the first parameters comprise the polynomial coefficient and the fitting residual.

8. A vehicle steering angle determination device characterized by comprising: The apparatus includes: An acquisition module configured to acquire a plurality of candidate corner points based on a current panoramic image, each candidate corner point being a pixel point at an edge of an object in the current panoramic image; wherein, for any pixel point in the current panoramic image, if a difference between a luminance value of a continuous N pixel point and a luminance value of the pixel point is greater than a threshold value, the pixel point is a candidate corner point; A determination module configured to determine a feature intensity response value of a candidate corner point based on luminance values of pixel points in a neighborhood window of the candidate corner point, the feature intensity response value quantifying an intensity of the candidate corner point; A generation module configured to generate a feature point set; wherein, based on the feature intensity response value of each candidate corner point, a part of the candidate corner points are selected as feature points from the plurality of candidate corner points, the feature point set including a plurality of feature points; wherein, the feature point set is used to determine a first parameter, the first parameter representing a conversion relationship between a vehicle actual steering angle and a vehicle collection steering angle; Wherein, after the first parameter is calibrated for the vehicle, a vehicle collection steering angle is collected by a vehicle sensor, and the vehicle collection steering angle is converted into a vehicle actual steering angle by the first parameter.

9. The apparatus of claim 8, wherein, When the determination module determines the feature intensity response value of the candidate corner point based on the luminance values of the pixel points in the neighborhood window of the candidate corner point, it is specifically configured to: for any pixel point in the neighborhood window, determine a first gradient value in a horizontal direction and a second gradient value in a vertical direction of the pixel point based on the luminance values of the pixel points in the neighborhood window; determine a structure tensor of the pixel point based on the first gradient value and the second gradient value, the structure tensor representing a distribution characteristic of gradients in the neighborhood window; determine a weight value of the pixel point based on an offset between the pixel point and the candidate corner point, the weight value representing a gradient contribution near the candidate corner point. determine a gradient covariance matrix of the candidate corner point based on the weight value and the structure tensor of each pixel point in the neighborhood window, and determine a feature intensity response value of the candidate corner point based on the gradient covariance matrix; Or, if the feature intensity response value of the candidate corner point is the maximum feature intensity response value in the neighborhood, the candidate corner point is taken as a feature point; the determination module is further configured to determine a target feature point based on the feature point set; wherein, each of the feature points in the feature point set is taken as the target feature point; or, the feature points in the feature point set located in the image edge blind area and / or the vehicle body area are filtered, and the remaining feature points are taken as the target feature points; a pixel pair corresponding to the target feature point is generated, the pixel pair including a pixel coordinate of the target feature point corresponding to a current frame panoramic image and a predicted coordinate of the target feature point corresponding to a next frame panoramic image; the first parameter is determined based on the pixel pairs corresponding to the target feature points; Or, for any feature point in the feature point set, if the pixel coordinates of the feature point satisfy the following expression, the feature point is not in the image edge blind area, otherwise, the feature point is in the image edge blind area: For any feature point in the feature point set, if the pixel coordinates of the feature point satisfy the following expression, the feature point is in the vehicle body area, otherwise, the feature point is not in the vehicle body area: Wherein, represents the horizontal pixel coordinates of the feature point, represents the vertical pixel coordinates of the feature point, represents the configured safety boundary value, represents the width of the current frame panoramic image, represents the height of the current frame panoramic image, represents the corresponding pixel coordinates of the vehicle body center in the current frame panoramic image, represents half of the vehicle length, represents half of the vehicle width; Or, the determination module is configured to determine the first parameter based on the pixel pairs corresponding to the target feature points, and specifically configured to: determine a second parameter based on the pixel pairs corresponding to the target feature points, the second parameter representing a conversion relationship between the pixel coordinate in the current frame panoramic image and the predicted coordinate in the next frame panoramic image; determine a vehicle actual steering angle corresponding to the current frame panoramic image based on the second parameter; generate a steering angle pair, the steering angle pair including the vehicle actual steering angle and a vehicle collection steering angle, the vehicle collection steering angle being a steering angle collected by a vehicle sensor and corresponding to the current frame panoramic image; and determine the first parameter based on the steering angle pairs corresponding to the panoramic images. Or, the determining module is specifically configured to determine the second parameter based on the pixel pairs corresponding to the plurality of target feature points, by using the following formula: ; represents the predicted coordinate of the target feature point corresponding to the next frame panoramic image, represents the pixel coordinate of the target feature point corresponding to the current frame panoramic image, represents the second parameter; represents the rotation matrix of the vehicle between the current frame panoramic image and the next frame panoramic image, represents the translation vector of the vehicle between the current frame panoramic image and the next frame panoramic image; The determining module determines the actual turning angle of the vehicle corresponding to the panoramic image of the current frame based on the second parameter, specifically: determining the actual turning angle of the vehicle corresponding to the panoramic image of the current frame by using the following formula: ; denotes the actual turning angle of the vehicle, denotes a four-quadrant inverse tangent function, and denotes the second parameter; Alternatively, the determining module is specifically configured to determine the first parameter based on the steering angle pairs corresponding to the plurality of panoramic images, by using the following formula: ; wherein, represents the actual steering angle of the vehicle in the steering angle pair, represents the collection steering angle of the vehicle in the steering angle pair, and m represents a configured numerical value, represents a polynomial coefficient, represents a fitting residual; wherein the first parameter comprises the polynomial coefficient and the fitting residual.

10. An electronic device, comprising: comprising: a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method of any one of claims 1-7.

Images