Vehicle chassis area image generation method, device and equipment and vehicle

Abstract

The invention provides a vehicle chassis area image generation method, device and equipment and a vehicle. Relates to the technical field of auxiliary driving image processing. The method comprises the following steps: determining an angle change observation value and a distance change observation value of a vehicle according to a chassis area panoramic image of the vehicle at a previous moment and a bird's-eye view image of the vehicle at a current moment, and determining a distance change observation value of the vehicle according to a first motion state parameter of the vehicle at the previous moment and a second motion state parameter of the vehicle at the current moment; determining an angle change predicted value and a distance change predicted value of the vehicle; then, correcting the observation value to obtain a corrected angle change observation value and a corrected distance change observation value; and finally, generating a panoramic image of the chassis area at the current moment based on the corrected observation value. According to the method, the predicted values of the angle change and the distance change can be determined according to the motion state parameters, the angle change observation value and the distance change observation value obtained by the image are corrected, and the accuracy of the generated chassis area image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of assisted driving image processing, and in particular to a method, device, equipment and vehicle for generating an image of a chassis area of ​​a vehicle. Background Art

[0002] With the continuous advancement of transparent chassis technology, real-time acquisition and display of vehicle chassis area images has become an important factor in improving driving safety and operating convenience. This chassis area image can effectively eliminate the blind spots generated during vehicle movement, allowing drivers to clearly understand the road conditions under the vehicle chassis, thereby enhancing their perception of the surrounding environment.

[0003] In the prior art, the image feature point detection and matching method is used to predict the image of the vehicle chassis at the next position from the image at the previous moment. When the vehicle actually arrives at the position, the chassis image predicted in advance is spliced ​​with the image at the current moment to generate a chassis area image. However, this method that relies on image feature point detection and matching needs to be performed when the vehicle speed is low and the ambient light changes little. If the vehicle speed is too fast or the ambient light changes greatly, the effect of feature point matching will be significantly reduced, resulting in a reduced accuracy of the generated chassis area image. Summary of the invention

[0004] The present application provides a method, device, equipment and vehicle for generating a chassis area image of a vehicle, which are used to improve the accuracy of the generated chassis area image.

[0005] In a first aspect, the present application provides a method for generating a chassis area image of a vehicle, comprising:

[0006] Determine an angle change observation value and a distance change observation value of the vehicle according to a panoramic image of a chassis area of ​​the vehicle at a previous moment and a bird's-eye view image at a current moment;

[0007] Determining an angle change prediction value and a distance change prediction value of the vehicle according to a first motion state parameter of the vehicle at a previous moment and a second motion state parameter of the vehicle at a current moment;

[0008] Correcting the angle change observation value and the distance change observation value according to the angle change prediction value and the distance change prediction value to obtain a corrected angle change observation value and a corrected distance change observation value;

[0009] A panoramic image of the chassis area at the current moment is generated according to the corrected angle change observation value and the corrected distance change observation value.

[0010] In a possible implementation, the correcting the angle change observed value and the distance change observed value according to the angle change predicted value and the distance change predicted value to obtain the corrected angle change observed value and the corrected distance change observed value includes:

[0011] According to the prediction vector of the vehicle at the current moment, a preset correction calculation formula is used to correct the observation vector to obtain a corrected observation vector;

[0012] Among them, the prediction vector includes the angle change prediction value and the distance change prediction value; the observation vector includes the angle change observation value and the distance change observation value; the corrected observation vector includes the corrected angle change observation value and the corrected distance change observation value.

[0013] In a possible implementation, determining the angle change prediction value and the distance change prediction value of the vehicle according to the first motion state parameter of the vehicle at a previous moment and the second motion state parameter of the vehicle at a current moment includes:

[0014] Performing linearization processing on the first motion state parameter and the second motion state parameter to obtain a state transfer matrix;

[0015] Determine a state prediction vector and an initial covariance matrix of the vehicle at a current moment according to the state transfer matrix and the first motion state parameter, wherein the state prediction vector includes a current angle change prediction value and a current distance change prediction value, and the initial covariance matrix is ​​used to represent a deviation between the state prediction vector and the second motion state parameter;

[0016] A prediction vector of the vehicle at a current moment is determined according to the state prediction vector and the initial covariance matrix, wherein the prediction vector includes the angle change prediction value and the distance change prediction value.

[0017] In a possible implementation manner, determining the prediction vector of the vehicle at the current moment according to the state prediction vector and the initial covariance matrix includes:

[0018] According to the initial covariance matrix, a prediction gain matrix is ​​calculated using a preset prediction gain calculation formula;

[0019] According to the preset update prediction calculation formula and the prediction gain matrix, the state prediction vector is updated and calculated to obtain the prediction vector of the vehicle at the current moment.

[0020] In a possible implementation manner, generating a panoramic image of the chassis area at the current moment according to the corrected angle change observation value and the corrected distance change observation value includes:

[0021] Determine a pixel transformation matrix according to the corrected angle change observation value, the corrected distance change observation value, and the projection ratio;

[0022] According to the pixel transformation matrix, extracting a predicted chassis area image from the chassis area panoramic image of the vehicle at a previous moment;

[0023] The predicted chassis area image and the bird's-eye view image at the current moment are fused to generate a panoramic image of the chassis area at the current moment.

[0024] In a possible implementation, determining the angle change observation value and the distance change observation value of the vehicle according to the chassis area panoramic image of the vehicle at the previous moment and the bird's-eye view image at the current moment includes:

[0025] Extracting feature points from the panoramic image of the chassis area of ​​the vehicle at a previous moment to obtain a first feature point set;

[0026] Extracting feature points from the bird's-eye view image at the current moment to obtain a second feature point set;

[0027] Obtaining a target homography matrix through the first feature point set and the second feature point set, wherein the target homography matrix is ​​used to transform the coordinates of the feature points in the panoramic image of the chassis area of ​​the vehicle at a previous moment into the bird's-eye view image at the current moment, so as to obtain the coordinates of each feature point in the bird's-eye view image at the current moment;

[0028] The angle change observation value and the distance change observation value of the vehicle are determined according to the target homography matrix.

[0029] In a possible implementation, extracting feature points from the panoramic image of the chassis area of ​​the vehicle at a previous moment to obtain a first feature point set includes:

[0030] Taking any pixel point in the panoramic image of the chassis area as the center, determine the pixel points on the circumference corresponding to the center by a preset radius;

[0031] For each center, if there are a plurality of continuous pixels on the circumference corresponding to the center, the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the plurality of pixels is greater than or equal to a preset number, then the center is determined as a candidate feature point in the chassis area panoramic image;

[0032] All candidate feature points in the panoramic image of the chassis area of ​​the vehicle at a previous moment are screened by a non-maximum suppression algorithm to obtain the first feature point set.

[0033] In a possible implementation, extracting feature points from the bird's-eye view image at the current moment to obtain a second feature point set includes:

[0034] Taking any pixel point in the bird's-eye view image as the center, determine the pixel points on the circumference corresponding to the center by a preset radius;

[0035] For each center, if there are a plurality of continuous pixels on the circumference corresponding to the center, the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the plurality of pixels is greater than or equal to a preset number, then the center is determined as a candidate feature point in the bird's-eye view image;

[0036] All candidate feature points in the bird's-eye view image at the current moment are screened by a non-maximum suppression algorithm to obtain the second feature point set.

[0037] In a possible implementation, acquiring a target homography matrix through the first feature point set and the second feature point set includes:

[0038] Using an optical flow method to obtain an offset vector of a feature point in a panoramic image of a chassis area of ​​the vehicle at a previous moment;

[0039] Pairing the feature points in the first feature point set with the feature points in the second feature point set according to the offset vector to obtain a set of matching feature point pairs;

[0040] Selecting a preset number of matching feature point pairs from the set of matching feature point pairs to determine a homography matrix;

[0041] For each feature point in the first feature point set, use the homography matrix to perform position transformation to obtain a first coordinate, where the first coordinate represents the position coordinate of the feature point in the bird's-eye view image at the current moment;

[0042] Determine the number of correctly matched point pairs according to a preset distance threshold, a first coordinate and a second coordinate of each feature point, wherein the second coordinate of each feature point is the actual position coordinate of the feature point in the bird's-eye view image at the current moment;

[0043] The homography matrix with the largest number of correctly matched point pairs is determined as the target homography matrix.

[0044] In a second aspect, the present application provides a vehicle chassis area image generation device, comprising:

[0045] A first processing module, for determining an angle change observation value and a distance change observation value of the vehicle according to a panoramic image of a chassis area of ​​the vehicle at a previous moment and a bird's-eye view image at a current moment;

[0046] A second processing module, used for determining an angle change prediction value and a distance change prediction value of the vehicle according to the first motion state parameter of the vehicle at a previous moment and the second motion state parameter of the vehicle at a current moment;

[0047] A correction module, used for correcting the angle change observation value and the distance change observation value according to the angle change prediction value and the distance change prediction value, so as to obtain a corrected angle change observation value and a corrected distance change observation value;

[0048] The third processing module is used to generate a panoramic image of the chassis area at the current moment according to the corrected angle change observation value and the corrected distance change observation value.

[0049] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and a communication interface;

[0050] The memory stores computer-executable instructions;

[0051] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method for generating a chassis area image of a vehicle as described in any one of the first aspects.

[0052] In a fourth aspect, the present application provides a vehicle, comprising: a vehicle body and the electronic device described in the third aspect.

[0053] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method for generating a chassis area image of a vehicle as described in any one of the first aspects.

[0054] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for generating a chassis area image of a vehicle as described in any one of the first aspects.

[0055] The present application provides a method, device, equipment and vehicle for generating a chassis area image of a vehicle. The electronic device can determine the angle change observation value and the distance change observation value of the vehicle based on the chassis area panoramic image of the vehicle at the last moment and the bird's-eye view image at the current moment; determine the angle change prediction value and the distance change prediction value of the vehicle based on the first motion state parameter of the vehicle at the last moment and the second motion state parameter of the current moment; then correct the angle change observation value and the distance change observation value to obtain the corrected observation value; finally, generate the chassis area panoramic image at the current moment based on the corrected angle change observation value and the corrected distance change observation value. Since the observation value determined according to the image can be corrected using the predicted value determined by the motion state parameter, the accuracy of the generated chassis area image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application;

[0058] Figure 2 A schematic diagram of a process flow of a first embodiment of a method for generating a chassis area image of a vehicle provided in the present application;

[0059] Figure 3 A schematic diagram of the flow chart of Embodiment 2 of the method for generating a chassis area image of a vehicle provided in the present application;

[0060] Figure 4 A schematic diagram of the flow chart of Embodiment 3 of the method for generating a chassis area image of a vehicle provided in the present application;

[0061] Figure 5 A schematic diagram of pixel distribution provided in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of a flow chart of a fourth embodiment of a method for generating a chassis area image of a vehicle provided in the present application;

[0063] Figure 7 A schematic diagram of a flow chart of a fifth embodiment of a method for generating a chassis area image of a vehicle provided in the present application;

[0064] Figure 8 A schematic flow chart of an example of a method for generating a chassis area image of a vehicle provided in this application;

[0065] Fig. 9 A schematic diagram of the structure of a vehicle chassis area image generation device provided in an embodiment of the present application;

[0066] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0067] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0068] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0069] As a cutting-edge innovation, transparent chassis technology is gradually changing the driving experience as automotive technology continues to advance. With the development of this technology, real-time acquisition and display of vehicle chassis area images has become an important means to improve driving safety and operating convenience. Through the provided chassis area images, drivers can better understand the road conditions under the vehicle, effectively eliminate blind spots during driving, and thus enhance their perception of the surrounding environment.

[0070] In the related art, the method of image feature point detection and matching is used to realize the generation of chassis area image. Specifically, the position and chassis image of the vehicle at the next moment can be predicted by analyzing the image at the previous moment. When the vehicle arrives at the new position, the predicted chassis image can be spliced ​​with the bird's-eye view image at the current moment to generate a complete chassis area image.

[0071] However, this method, which relies on image feature point detection and matching, needs to be performed when the vehicle speed is low and the ambient light changes little. If the vehicle speed is too fast or the ambient light changes greatly, the effect of feature point matching will be significantly reduced, resulting in a lower accuracy of the generated chassis area image.

[0072] In view of the above problems, the inventors found in the process of studying the generation of chassis area images of vehicles that the image motion obtained by visual features can be corrected based on the vehicle's motion state to eliminate the problem of decreased image feature point matching performance caused by excessive vehicle motion speed or large changes in ambient brightness, thereby improving the accuracy of the generated chassis area image. Specifically, the vehicle's angle change prediction value and distance change prediction value can be determined according to the vehicle's motion state parameters, and the angle change observation value and distance change observation value obtained by correcting the chassis area panoramic image at the previous moment and the bird's-eye view image at the current moment can be used to generate a chassis area panoramic image at the current moment. Since the angle change observation value and the distance change observation value can be corrected, the matching error of the feature points in the image can be eliminated, thereby improving the accuracy of the generated chassis area image. Based on this, the present application proposes a method for generating a chassis area image of a vehicle to further improve the accuracy of generating a chassis area panoramic image at the current moment.

[0073] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 During the driving process of the vehicle, the electronic device can obtain a panoramic image of the chassis area of ​​the vehicle at the previous moment and a bird's-eye view image at the current moment. By analyzing and processing the image information at these two moments, a panoramic image of the chassis area at the current moment can be generated, providing the driver with a detailed chassis view, so that the driver can more clearly grasp the real-time status of the vehicle and the environmental conditions around the bottom of the vehicle.

[0074] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0075] Figure 2 This is a flow chart of the first embodiment of the method for generating a chassis area image of a vehicle provided in this application. Figure 2 , the method may include:

[0076] S201. Determine an angle change observation value and a distance change observation value of the vehicle based on a panoramic image of a chassis area of ​​the vehicle at a previous moment and a bird's-eye view image at a current moment.

[0077] The execution subject of the embodiment of the present application may be an electronic device, or a chassis area image generating device of a vehicle set in the electronic device. The chassis area image generating device of the vehicle may be implemented by software, or by a combination of software and hardware. The chassis area image generating device of the vehicle may be a processor in the electronic device. For ease of understanding, the following description is made by taking the execution subject as an electronic device as an example.

[0078] In this step, during the driving process of the vehicle, the electronic device can obtain a panoramic image of the chassis area of ​​the vehicle at the previous moment and a bird's-eye view image at the current moment. According to the obtained panoramic image of the chassis area and the bird's-eye view image, the angle change observation value and the distance change observation value of the vehicle can be determined.

[0079] Among them, the angle change observation value is used to indicate the steering angle of the vehicle between the previous moment and the current moment; the distance change observation value is used to indicate the offset of the vehicle between the two moments, reflecting the vehicle's travel distance. The distance change observation value may include the vehicle's offset in the X-axis direction and the vehicle's offset in the Y-axis direction.

[0080] Optionally, a vehicle coordinate system can be established by using the center point of the vehicle at the initial moment as the coordinate origin, with the X-axis and Y-axis of the vehicle coordinate system respectively located in the plane where the ground is located, the Z-axis coordinate perpendicular to the ground is zero, the X-axis of the vehicle coordinate system is consistent with the forward direction of the vehicle, and the Y-axis of the vehicle is perpendicular to the X-axis, pointing to the right direction of the vehicle, wherein the initial moment can be the previous moment or the moment before the previous moment, and the moment before the previous moment can be the moment when the vehicle is started.

[0081] The bird's-eye view image can be obtained through the vehicle's panoramic imaging system. In specific implementation, the system can stitch images collected in real time by four cameras installed in the front, rear, left and right sides of the vehicle to generate a bird's-eye view image.

[0082] Optionally, the panoramic image of the chassis area at the previous moment can be generated by continuous frame bird's-eye view images at multiple moments before the current moment.

[0083] In a specific implementation, feature point extraction can be performed on the panoramic image of the chassis area of ​​the vehicle at the previous moment to obtain a first feature point set, and feature point extraction can be performed on the bird's-eye view image at the current moment to obtain a second feature point set. The target homography matrix is ​​obtained through the first feature point set and the second feature point set, and the angle change observation value and the distance change observation value of the vehicle are determined from the target homography matrix. The target homography matrix is ​​used to convert the coordinates of the feature points in the panoramic image of the chassis area of ​​the vehicle at the previous moment to the bird's-eye view image at the current moment, and obtain the coordinates of each feature point in the bird's-eye view image at the current moment.

[0084] For example, feature points can be extracted from the panoramic image of the chassis area of ​​the vehicle at the previous moment to obtain the first feature point set A, and feature points can be extracted from the bird's-eye view image at the current moment to obtain the second feature point set B. The target homography matrix M can be determined from the first feature point set A and the second feature point set B, and the angle change observation value θ of the vehicle can be determined from the target homography matrix M. it and the distance change observation (x it ,y it ).

[0085] S202. Determine a predicted value of an angle change and a predicted value of a distance change of the vehicle according to a first motion state parameter of the vehicle at a previous moment and a second motion state parameter of the vehicle at a current moment.

[0086] In this step, the electronic device can calculate the vehicle's angle change prediction value and distance change prediction value by analyzing the vehicle's first motion state parameter at the previous moment and the vehicle's second motion state parameter at the current moment.

[0087] Optionally, the first motion state parameters include the vehicle's position, rotation angle, X-axis speed, Y-axis speed, and angular velocity at the previous moment; the second motion state parameters include the vehicle's position, rotation angle, X-axis speed, Y-axis speed, and angular velocity at the current moment.

[0088] In an optional embodiment, the first motion state parameter and the second motion state parameter can be linearized to obtain a state transfer matrix. Based on the state transfer matrix and the first motion state parameter, the state prediction vector and the initial covariance matrix of the vehicle at the current moment are determined. Based on the state prediction vector and the initial covariance matrix, the prediction vector of the vehicle at the current moment is determined, and the prediction vector includes an angle change prediction value and a distance change prediction value.

[0089] In a specific implementation, the state prediction vector at the current moment can be calculated according to the state transfer matrix and the first motion state parameter through the Kalman prediction formula, and the initial covariance matrix can be obtained through the covariance matrix calculation formula. The state prediction vector includes the current angle change prediction value and the current distance change prediction value, and the initial covariance matrix is ​​used to represent the deviation between the state prediction vector and the second motion state parameter.

[0090] Among them, the Kalman prediction formula is: In the Kalman prediction formula, represents the state prediction vector at the current moment, F represents the state transfer matrix, The state vector at the previous moment. The subscript t represents the current moment, and the subscript t-1 represents the previous moment.

[0091] Optional, the state vector at the previous moment It can be determined based on the first motion state parameter. For example, the state vector at the last moment It can be determined by the first motion state parameters including the position, rotation angle, X-axis speed, Y-axis speed, and angular velocity of the vehicle at the last moment.

[0092] The formula for calculating the covariance matrix is: t =FP t-1 F T +Q t ; In the covariance matrix calculation formula, P t represents the initial covariance matrix, Q t Represents sensor noise.

[0093] Where, the sensor noise Q t This may include noise from the Inertial Measurement Unit (IMU) and wheel speedometer.

[0094] For example, Q t It can be represented by a 6*6 matrix, in which the diagonal positions from the upper left corner to the lower right corner of the matrix can be the X-axis acceleration noise of the IMU, the Y-axis acceleration noise of the IMU, the angular velocity noise of the IMU, the X-axis acceleration noise of the wheel speed meter, the Y-axis acceleration noise of the wheel speed meter, and the angular velocity noise of the wheel speed meter, and the rest of the positions in the matrix are 0.

[0095] In a specific implementation, the noise of the IMU and the wheel speed meter may be determined by sensor factory data provided by the sensor manufacturer.

[0096] Furthermore, the prediction gain matrix can be calculated based on the initial covariance matrix using a preset prediction gain calculation formula. The preset prediction gain calculation formula can be:

[0097]

[0098] In formula (1), K′ t represents the prediction gain matrix, R t represents the sensor noise of the observed quantity, H t Represents the first transformation matrix that transforms the predicted value from the state domain to the observation domain.

[0099] Optionally, the observed sensor noise R t It refers to the random error introduced by the limitations of the sensor itself and external environmental factors when acquiring observation data. The sensor noise R can be determined by analyzing historical experimental data. t The characteristics of the observed sensor noise R t .

[0100] Optionally, the observed sensor noise R t and sensor noise Q t can have the same form of expression, but the eigenvalues ​​may not be exactly the same. The same form of expression can refer to the sensor noise R of the observed quantity. t and sensor noise Q t Both can be represented by matrices of the same structure, and the eigenvalues ​​may not be completely the same, which may mean that the number of features in the two matrices and / or the eigenvalue of each feature may not be completely the same.

[0101] The first transformation matrix H t Describes the linear relationship between system state data and observation data, which can be determined by state transformation experiments on historical system state data and observation data. t System state data can be converted into observation data.

[0102] Finally, the state prediction vector can be updated and calculated according to the preset update prediction calculation formula and prediction gain matrix to obtain the prediction vector of the vehicle at the current moment. The preset update prediction calculation formula can be:

[0103]

[0104] In formula (2), represents the prediction vector of the vehicle at the current moment, z t Represents the complete observations of the IMU and wheel speed meter.

[0105] For example, according to formula (2) and the prediction gain matrix K′ t , for the state prediction vector Update to get the prediction vector of the vehicle at the current moment Prediction Vector The vehicle's angle change prediction value θ may be included mt and the predicted value of distance change (x mt ,y mt ).

[0106] S203. Correct the angle change observation value and the distance change observation value according to the angle change prediction value and the distance change prediction value to obtain a corrected angle change observation value and a corrected distance change observation value.

[0107] In this step, the angle change observation value and distance change observation value of the vehicle determined from the target homography matrix can be corrected according to the angle change prediction value and distance change prediction value obtained by the prediction vector to obtain the corrected angle change observation value and the corrected distance change observation value.

[0108] In a specific implementation, the observation vector can be corrected using a preset correction calculation formula according to the prediction vector of the vehicle at the current moment to obtain a corrected observation vector;

[0109] Among them, the prediction vector includes the angle change prediction value and the distance change prediction value; the observation vector includes the angle change observation value and the distance change observation value; the corrected observation vector includes the corrected angle change observation value and the corrected distance change observation value.

[0110] Optionally, the preset correction calculation formula may be:

[0111]

[0112] in, represents the corrected observation vector, K′ it represents the observation gain matrix, z it represents the basic observation value of IMU and wheel speed meter, H it The second transformation matrix represents the transformation of the predicted value from the state domain to the observation domain.

[0113] For the observation gain matrix K′ it , can be obtained by predicting the gain matrix K′ t For the initial covariance matrix P t , update and get the updated covariance matrix P′ t , by updating the covariance matrix P′ t and z it The covariance matrix R it Determine the observation gain matrix K′ it . Among them, the covariance matrix R it Calibration can be performed manually.

[0114] Similarly, the basic observation value z of IMU and wheel speed meter is it Can be combined with the complete observation value z of IMU and wheel speed meter t Can have the same representation but different number of features.

[0115] For example, the base observation z it It can include the vehicle's position and rotation angle at the current moment; the complete observation value z t It may include the vehicle's current position, rotation angle, X-axis speed, Y-axis speed, and angular velocity.

[0116] Similarly, the second transformation matrix H it It can be determined by conducting state transformation experiments on historical system state data and historical observation data, and the first transformation matrix H tcan have the same representation, but the number of features and / or the eigenvalues ​​of each feature can be different from the first transformation matrix H t Not the same.

[0117] For example, the observation vector can be corrected using a preset correction calculation formula based on the predicted vector of the vehicle at the current moment. Perform correction processing to obtain the corrected observation vector Among them, the prediction vector includes the angle change prediction value θ mt And the predicted value of distance change x mt and mt ; The observation vector includes the angle change observation value θ it and the distance change observation (x it ,y it ), the corrected observation vector Including the corrected angle change observation θ m And the corrected distance change observation (x m ,y m ).

[0118] S204: Generate a panoramic image of the chassis area at the current moment according to the corrected angle change observation value and the corrected distance change observation value.

[0119] In this step, the predicted chassis area image can be cut out from the panoramic image of the chassis area of ​​the vehicle at the previous moment based on the corrected angle change observation value and the corrected distance change observation value included in the corrected observation vector, and the predicted chassis area image and the bird's-eye view image at the current moment are fused to generate the chassis area image of the vehicle.

[0120] In a specific implementation, the pixel transformation matrix can be determined according to the corrected angle change observation value, the corrected distance change observation value, and the projection ratio. The predicted chassis area image can be cut out from the panoramic image of the chassis area of ​​the vehicle at the previous moment according to the pixel transformation matrix, and the predicted chassis area image and the bird's-eye view image at the current moment are fused to generate the chassis area image of the vehicle.

[0121] In an embodiment of the present application, the electronic device can determine the vehicle angle change and distance change observation values ​​through the chassis area panoramic image of the vehicle at the previous moment and the bird's-eye view image at the current moment, and determine the vehicle angle change and distance change prediction values ​​according to the motion state parameters at the previous moment and the current moment. Then, the vehicle angle change and distance change observation values ​​can be corrected by the vehicle angle change and distance change prediction values ​​to obtain the corrected vehicle angle change and distance change observation values, and finally generate the chassis area panoramic image at the current moment. In the above process, since the predicted values ​​determined by the motion state parameters can be used to correct the chassis area panoramic image of the vehicle at the previous moment and the observation values ​​determined by the bird's-eye view image at the current moment, the accuracy of the generated chassis area image can be improved.

[0122] exist Figure 2 Based on the embodiment shown below, combined with Figure 3 , the method for generating the chassis area image of the above-mentioned vehicle is further described in detail.

[0123] Figure 3 This is a flow chart of the second embodiment of the method for generating a chassis area image of a vehicle provided in this application. Figure 3 , the method may include:

[0124] S301 : extracting feature points from a panoramic image of a chassis area of ​​a vehicle at a previous moment to obtain a first feature point set.

[0125] In this step, the feature points in the panoramic image of the chassis area of ​​the vehicle at the last moment can be extracted by using an image feature point extraction algorithm.

[0126] Optionally, the feature point in the panoramic image of the chassis area of ​​the vehicle at the previous moment may be a corner point in the image. A corner point is usually located at the edge of the image or at the intersection of objects, and is characterized by a large brightness change around the focus.

[0127] The image feature extraction algorithm may include any one of a fast corner detection (FAST) algorithm, a scale-invariant feature transform (SIFT) algorithm, and a speeded up robust feature (SUFR) algorithm.

[0128] For example, the FAST algorithm may be used to extract feature points from a panoramic image of the chassis area of ​​the vehicle at the last moment to obtain a first feature point set.

[0129] S302: Extract feature points from the bird's-eye view image at the current moment to obtain a second feature point set.

[0130] For example, the FAST algorithm may be used to extract feature points from the bird's-eye view image at the current moment to obtain a second feature point set.

[0131] S303 . Obtain a target homography matrix through the first feature point set and the second feature point set.

[0132] In an optional embodiment, the offset vector of the feature point in the panoramic image of the vehicle chassis area can be obtained by the optical flow method, and the first feature point set and the second feature point set are paired to obtain a set of matching feature point pairs. Then, a preset number of matching point pairs are selected, the homography matrix is ​​determined, and the matrix is ​​used to transform the position of the feature points in the first feature point set to obtain their coordinates in the bird's-eye view image at the current moment. According to the preset distance threshold and the first coordinate and second coordinate of each feature point, the number of correctly matched point pairs is calculated, and finally the homography matrix with the largest number of point pairs is determined as the target homography matrix. Among them, the first coordinate represents the position coordinate of the feature point in the bird's-eye view image at the current moment; the second coordinate is the real position coordinate of the feature point in the bird's-eye view image at the current moment.

[0133] For example, through the first feature point set and the second feature point set, the target homography matrix M obtained can be:

[0134] Among them, m 11 is the first scaling factor, which indicates the horizontal scaling ratio of the image at the previous moment to the image at the current moment; m 12 is the first shear factor, which indicates the influence of the image at the previous moment (Y coordinate) on the image at the current moment (X coordinate); m 13 is the first translation factor, which represents the translation of the (X) coordinate in the image at the previous moment, that is, the horizontal translation of the image; m 21 is the second shear factor, which indicates the influence of the image at the previous moment (X coordinate) on the image at the current moment (Y coordinate); m 22 is the second scaling factor, which indicates the vertical scaling ratio of the image at the previous moment to the image at the current moment; m 23 is the second translation factor, which represents the translation of the (Y) coordinate in the image at the previous moment, that is, the translation of the image in the vertical direction.

[0135] S304. Determine the angle change observation value and the distance change observation value of the vehicle according to the target homography matrix.

[0136] Specifically, the vehicle’s angle change observation value θ can be determined based on the target homography matrix M: it and the distance change observation (x it ,y it ):

[0137] θ it =arctan(-m 12 / m 11 )

[0138] x it =m 13

[0139] y it =m 23

[0140] S305 , linearize the first motion state parameter and the second motion state parameter to obtain a state transfer matrix.

[0141] In this step, the first motion state parameter of the vehicle at the previous moment and the second motion state parameter at the current moment may be linearized to obtain a state transfer matrix.

[0142] Specifically, the vehicle's current position information (x t ,y t ), and the rotation angle θ t , get the current state vector

[0143]

[0144] Among them, (x t ,y t ) represents the vehicle’s position at the current moment, θ t Indicates the rotation angle of the vehicle at the current moment, vx t Indicates the longitudinal (X-axis) speed of the vehicle at the current moment, vy t represents the lateral (Y-axis) speed of the vehicle at the current moment, ω t Represents the angular velocity of the vehicle at the current moment.

[0145] Optionally, within the time interval Δt between the previous moment and the current moment, the acceleration and angular velocity of the vehicle can be considered to be unchanged, and the translation distance is Δs. The calculation formula between the state vector at the current moment and the state vector at the previous moment can be obtained as follows:

[0146]

[0147] In formula (4), ax t-1 Indicates the vehicle's acceleration in the X-axis direction at the last moment, ay t-1 Indicates the acceleration of the vehicle in the Y-axis direction at the previous moment.

[0148] Due to the angular velocity in formula (4), there is a nonlinear relationship between the state vector at the current moment and the state vector at the previous moment. Therefore, the vehicle motion state parameters can be smoothed by the extended Kalman filter method, and formula (4) can be linearized by the Jacobian formula to utilize more vehicle information (such as angular velocity, acceleration) to improve the accuracy of generating the panoramic image of the chassis area at the current moment.

[0149] In a specific implementation, the formula (4) is linearized by the Jacobian formula, and the obtained state transfer matrix is:

[0150]

[0151] S306. Determine a state prediction vector and an initial covariance matrix of the vehicle at the current moment according to the state transfer matrix and the first motion state parameter.

[0152] In this step, the state prediction vector and the initial covariance matrix of the vehicle at the current moment can be determined based on the acquired state transfer matrix and the first motion state parameter of the vehicle at the previous moment.

[0153] Specifically, according to the Kalman prediction formula Determine the vehicle's current state prediction vector The covariance matrix can be calculated according to the formula P t =FP t-1 F T +Q t , determine the initial covariance matrix P of the vehicle at the current moment t .

[0154] S307 . Determine a prediction vector of the vehicle at the current moment according to the state prediction vector and the initial covariance matrix, wherein the prediction vector includes an angle change prediction value and a distance change prediction value.

[0155] In this step, the prediction gain matrix can be calculated based on the determined initial covariance matrix using the prediction gain calculation formula preset in formula (1). Then, the state prediction vector can be updated based on the update prediction calculation formula and the prediction gain matrix preset in formula (2) to obtain the prediction vector of the vehicle at the current moment.

[0156] For example, the initial covariance matrix P can be determined based on t , the prediction gain calculation formula preset by formula (1) Calculate the prediction gain matrix K′ t Then, the update prediction calculation formula can be preset according to formula (2) And the prediction gain matrix K′ t, for the state prediction vector Perform update calculations to obtain the vehicle's prediction vector at the current moment Prediction Vector The angle change prediction value θ is included mt and the predicted value of distance change (x mt ,y mt ).

[0157] In an optional implementation, after the prediction gain matrix is ​​calculated, the initial covariance matrix may be updated to obtain an updated covariance matrix. In a specific implementation, the first covariance matrix may be updated using the preset formula: t =P t -K′ t H t P t , through the prediction gain matrix K′ t , for the initial covariance matrix P t Update and get the updated covariance matrix P′ t .

[0158] S308. According to the prediction vector of the vehicle at the current moment, the observation vector is corrected using a preset correction calculation formula to obtain a corrected observation vector; wherein the prediction vector includes an angle change prediction value and a distance change prediction value; the observation vector includes an angle change observation value and a distance change observation value; the corrected observation vector includes a corrected angle change observation value and a corrected distance change observation value.

[0159] For example, the correction calculation formula preset as formula (3) can be used: According to the prediction vector of the vehicle at the current moment, the observation vector is corrected to obtain the corrected observation vector Corrected observation vector Including the corrected angle change observation θ m And the corrected distance change observation (x m ,y m ).

[0160] In an optional implementation, the observation gain matrix K′ in the preset correction calculation formula can be determined by the following steps: it .

[0161] Step 1: Determine the vehicle's angle change observation value θ from the target homography matrix M it and the distance change observation (x it ,y it ), determine the Kalman filter observation vector z it :

[0162]

[0163] Among them, the Kalman filter observation vector is also the basic observation value of the IMU and wheel speed meter.

[0164] Step ②: Based on the Kalman filter observation vector z it The calibrated covariance matrix R it , and update the covariance matrix P′ t , through the preset observation gain calculation formula The observation gain matrix K′ can be calculated it .

[0165] In another optional implementation, after the observation gain matrix is ​​determined, the update covariance matrix may be updated to obtain a second updated covariance matrix. In a specific implementation, the second covariance matrix update formula may be preset: it =P′ t -K′ it H it P′ t , according to the observation gain matrix K′ it , to update the covariance matrix P′ t Update again to get the covariance matrix P′ after the second update it .

[0166] S309, determining a pixel transformation matrix according to the corrected angle change observation value, the corrected distance change observation value, and the projection ratio.

[0167] In this step, the pixel transformation matrix from the panoramic image of the chassis area of ​​the vehicle at the previous moment to the bird's-eye view image at the current moment can be determined based on the corrected angle change observation values ​​and distance change observation values, as well as the projection ratio determined by the world coordinate system and the image coordinate system of the bird's-eye view image.

[0168] For example, according to the corrected angle change observation θ m and the distance change observation (x m ,y m ), and the projection ratio k determined by the world coordinate system and the image coordinate system of the bird's-eye view image, determine the pixel transformation matrix from the chassis area panoramic image of the vehicle at the previous moment to the bird's-eye view image at the current moment:

[0169]

[0170] S310 . According to the pixel transformation matrix, extract a predicted chassis area image from the chassis area panoramic image of the vehicle at the previous moment.

[0171] For example, the predicted chassis area image may be extracted from the chassis area panoramic image of the vehicle at the last moment according to the pixel transformation matrix N.

[0172] S311, fusing the predicted chassis area image with the bird's-eye view image at the current moment to generate a panoramic image of the chassis area at the current moment.

[0173] In this step, the predicted chassis area image and the bird's-eye view image at the current moment may be fused. During the fusion process, a bilinear interpolation method may be used to fill in missing pixels to generate a panoramic image of the chassis area at the current moment.

[0174] In an embodiment of the present application, feature points can be extracted from the panoramic image of the vehicle chassis area and the current bird's-eye view image to obtain the target homography matrix, and the angle change observation value and distance change observation value of the vehicle can be determined. The motion state parameters are linearized to obtain a state transfer matrix, and the predicted vector of the vehicle at the current moment is determined based on the matrix. The observation vector is processed using a correction formula to obtain a corrected observation vector. The pixel transformation matrix is ​​determined based on the corrected observation value and the projection ratio, so as to intercept the predicted chassis area image, and fuse it with the current bird's-eye view image to generate a panoramic image of the chassis area at the current moment. In the above process, the visual feature point matching can be supplemented by combining the vehicle's dynamic information, effectively coping with the degradation of image feature point matching performance caused by high-speed vehicle movement or changes in ambient brightness, and improving the accuracy of generating a panoramic image of the chassis area at the current moment.

[0175] Furthermore, the vehicle chassis area image generation method provided in the embodiment of the present application adopts the extended Kalman filter method, which can effectively smooth the motion state parameters and make full use of various vehicle data such as angular velocity and acceleration, thereby improving the accuracy and reliability of the prediction. It further improves the accuracy of the chassis area panoramic image at the current moment and avoids problems such as image distortion and dislocation.

[0176] Figure 4 This is a flow chart of the third embodiment of the method for generating a chassis area image of a vehicle provided in this application. Figure 4 On the basis of the above-mentioned embodiment 2, in the specific implementation of the vehicle chassis area image generation method, in step S301, feature points are extracted from the chassis area panoramic image of the vehicle at the previous moment to obtain a first feature point set, and the specific implementation further includes the following steps:

[0177] S401, taking any pixel point in the chassis area panoramic image as the center, and determining the pixel points on the circumference corresponding to the center by a preset radius.

[0178] In this step, any pixel point in the panoramic image of the chassis area may be taken as the center, and the pixel points on the circumference corresponding to the center may be determined using a preset radius.

[0179] Figure 5 This is a schematic diagram of pixel distribution provided in the embodiment of the present application. Figure 5 , pixel point A in the panoramic image of the chassis area can be selected as the center. If the preset radius is 3, the preset radius 3 can be used to determine that the number of pixel points on the circumference corresponding to the center is 16, namely pixel points 1, 2, ..., 16.

[0180] Optionally, a preset radius may be preset. For example, if the preset radius is 1, the number of pixel points on the circumference corresponding to the center may be determined to be 8; if the preset radius is 2, the number of pixel points on the circumference corresponding to the center may be determined to be 12.

[0181] S402. For each center, if there are multiple continuous pixel points on the circumference corresponding to the center, and the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the multiple pixel points is greater than or equal to a preset number, the center is determined as a candidate feature point in the panoramic image of the chassis area.

[0182] In this step, for each center point, if there are multiple continuous pixel points on the circumference corresponding to the center, the brightness difference between them and the center point is greater than the preset brightness threshold, and the number of these pixel points is greater than or equal to the preset number, then the center point can be determined as a candidate feature point in the panoramic image of the chassis area.

[0183] The brightness of a pixel refers to the brightness of a pixel in an image, which can be measured by the RGB value (red, green, blue) of the color. In order to simplify processing or analysis, a color image can be converted to a grayscale image. Each pixel of a grayscale image contains only brightness information, not color information. A common method for converting RGB values ​​to grayscale values ​​is the weighted average method, which is to obtain the grayscale value by calculating the weighted sum of each color component. The specific formula is: brightness (grayscale value) = 0.299*R+0.587*G+0.114*B.

[0184] Optionally, the brightness difference between the pixel points on the circumference corresponding to the center and the center point is greater than a preset brightness threshold, which means that the number of pixels with larger brightness values ​​minus the number of pixels with smaller brightness values ​​between the pixel points on the circumference and the center point is greater than the preset brightness threshold.

[0185] For example, according to the preset radius of 3, the pixel points on the circumference corresponding to the center A are determined to be pixel points 1, 2, ..., 16. If the brightness threshold is 10, and the preset number is set to 12. If the brightness difference between pixel points 1 to 12 and the center A is greater than the preset brightness threshold 10, and the number of these pixel points 12 is equal to the preset number 12, the center point A can be determined as a candidate feature point in the panoramic image of the chassis area.

[0186] For another example, according to the preset radius of 3, the pixel points on the circumference corresponding to the center A are determined to be pixel points 1, 2, ..., 16. If the brightness threshold is 10, and the preset number is set to 12. The brightness difference between pixel points 1 to 10 and the center A is greater than the preset brightness threshold 10, and the brightness difference between pixel point 11 and the center A is 8 less than the preset brightness threshold 10. Since there are 10 consecutive pixel points on the circumference corresponding to the center A, the brightness difference between them and the center point is greater than the preset brightness threshold, but the number of these pixel points 10 is less than the preset number 12, the center point A cannot be determined as a candidate feature point in the panoramic image of the chassis area.

[0187] S403 , screening all candidate feature points in the panoramic image of the chassis area of ​​the vehicle at the last moment by a non-maximum suppression algorithm to obtain a first feature point set.

[0188] In this step, the non-maximum suppression algorithm can be used to screen the candidate feature points in the panoramic image of the vehicle chassis area. By comparing the local neighborhood of each candidate feature point, only the feature point with the highest brightness value in the neighborhood is retained, and other points with lower brightness are suppressed. Ensure that only the most representative feature points are screened out, thereby forming the first feature point set.

[0189] In an embodiment of the present application, the electronic device can set a radius to determine the pixel points on the circumference with any pixel point in the panoramic image of the chassis area as the center. If the brightness difference between the center and multiple consecutive pixel points on the circumference exceeds a preset threshold and the number reaches a preset value, the center can be determined as a candidate feature point. Then, all candidate feature points can be screened by a non-maximum suppression algorithm to form a first feature point set. In the above process, by setting the brightness threshold and quantity requirements, it can be ensured that the selected feature points have a high degree of distinction in the image, enhance the effect of the overall image analysis, and lay the foundation for improving the generation of a panoramic image of the chassis area at the current moment.

[0190] Figure 6 This is a flow chart of the fourth embodiment of the vehicle chassis area image generation method provided in this application. Figure 6On the basis of the above-mentioned second embodiment, in the specific implementation of the vehicle chassis area image generation method, in step S302, feature points are extracted from the bird's-eye view image at the current moment to obtain a second feature point set, and the specific implementation further includes the following steps:

[0191] S601: Taking any pixel point in the bird's-eye view image as the center, determine the pixel points on the circumference corresponding to the center by using a preset radius.

[0192] S602: For each center, if there are multiple continuous pixel points on the circumference corresponding to the center, and the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the multiple pixel points is greater than or equal to a preset number, the center is determined as a candidate feature point in the bird's-eye view image.

[0193] S603: Screen all candidate feature points in the bird's-eye view image at the current moment by using a non-maximum suppression algorithm to obtain a second feature point set.

[0194] It should be noted that Figure 6 The processing steps (S601-S603) shown in the embodiment can be referred to Figure 4 The processing steps (S401-S403) shown in the embodiment have similar implementation principles and beneficial effects, which will not be described in detail here.

[0195] Figure 7 This is a flow chart of the fifth embodiment of the method for generating a chassis area image of a vehicle provided in this application. Figure 7 Based on the above-mentioned embodiment 2, in the specific implementation of the vehicle chassis area image generation method, the target homography matrix is ​​obtained through the first feature point set and the second feature point set in step S303, and the specific implementation also includes the following steps:

[0196] S701, using an optical flow method to obtain an offset vector of a feature point in a panoramic image of a chassis area of ​​the vehicle at a previous moment.

[0197] In an optional implementation, the calculation formula for the offset vector of the feature point in the panoramic image of the chassis area of ​​the vehicle at the last moment obtained by the optical flow method is:

[0198] v=(A T A) -1 A T (-b) Formula (5)

[0199] In formula (5), A is the partial derivative matrix of the pixel point in the sliding window in the X-axis direction and the Y-axis direction in the image coordinate system, and b is the derivative vector of the pixel point in the sliding window with respect to time.

[0200] Among them, the partial derivative matrix A is:

[0201]

[0202] In the partial derivative matrix A, I xi represents the partial derivative of the image brightness in the X direction at the i-th pixel in the sliding window); I yi represents the partial derivative of the image brightness in the Y direction at the i-th pixel in the sliding window); i = 1, 2, …, n.

[0203] The derivative vector b is:

[0204]

[0205] In the derivative vector b, I ti is the derivative of the image brightness at the i-th pixel in the sliding window with respect to time, i = 1, 2, …, n.

[0206] S702 . Pair feature points in the first feature point set with feature points in the second feature point set according to the offset vector to obtain a set of matching feature point pairs.

[0207] In this step, each feature point in the first feature point set can be paired with the corresponding feature point in the second feature point set based on the calculated offset vector. Through the offset vector, the position of each feature point in the first feature point set in the second feature point set can be predicted, thereby forming a set of matching feature point pairs.

[0208] For example, the first feature point set includes feature points A1, A2, ..., A100; the second feature point set includes feature points B1, B2, ..., B50. Then, each feature point in the first feature point set can be paired with the corresponding feature point in the second feature point set according to the offset vector V to form a matching feature point pair set, which can include 30 successfully matched feature point pairs.

[0209] S703: Select a preset number of matching feature point pairs from the matching feature point pair set to determine a homography matrix.

[0210] In this step, a random sampling consensus algorithm may be adopted to randomly select a preset number of matching feature point pairs from the matching feature point pair set, and each time a preset number of matching feature point pairs are randomly selected, a homography matrix may be determined.

[0211] For example, if the preset number is 4, 4 matching feature point pairs can be randomly selected from the matching feature point pair set for the first time to determine the homography matrix M1, and 4 matching feature point pairs can be randomly selected from the matching feature point pair set for the second time to determine the homography matrix M2. Similarly, 4 matching feature point pairs can be randomly selected from the matching feature point pair set multiple times to determine the homography matrix M1 in turn. i. Wherein, i = 1, 2, ..., n (n ≥ 1).

[0212] S704 . For each feature point in the first feature point set, a homography matrix is ​​used to perform position transformation to obtain a first coordinate, where the first coordinate represents the position coordinate of the feature point in the bird's-eye view image at the current moment.

[0213] S705. Determine the number of correctly matched point pairs according to a preset distance threshold, a first coordinate and a second coordinate of each feature point, where the second coordinate of each feature point is the actual position coordinate of the feature point in the bird's-eye view image at the current moment.

[0214] In this step, the first coordinate of each feature point and the corresponding second coordinate can be compared to calculate the distance between the two coordinates. If the distance between the two coordinates is less than a preset distance threshold, it is determined that the pair of feature points is correctly matched. By traversing all feature point pairs, the number of matching point pairs that meet the conditions can be counted.

[0215] For example, the set of matched feature point pairs may include a successfully matched feature point pair [A1, B1], the preset distance threshold is 2, the feature point A1 is transformed in position using the homography matrix M1, and the first coordinate (2, 2) is obtained. The first coordinate (2, 2) represents the position coordinate of the feature point A1 in the bird's-eye view image at the current moment.

[0216] Furthermore, based on the preset distance threshold 2, the first coordinate (2, 2) and the second coordinate (2, 3) of the feature point A1, it can be determined that the feature point A1 and the feature point B1 are correctly matched, and the second coordinate of the feature point A1 is the actual position coordinate (2, 3) of the feature point B1 in the bird's-eye view image at the current moment.

[0217] S706: Determine the homography matrix with the largest number of correctly matched point pairs as the target homography matrix.

[0218] In this step, according to the number of correctly matched point pairs, the homography matrix with the largest number of correctly matched point pairs from the determined homography matrices can be determined as the target homography matrix.

[0219] For example, a preset number of matching feature point pairs can be selected from the matching feature point pair set to determine three homography matrices, namely homography matrix M1, homography matrix M2, and homography matrix M3. Through homography matrix M1, it is determined that the number of correctly matched point pairs is 10; through homography matrix M2, it is determined that the number of correctly matched point pairs is 15; through homography matrix M3, it is determined that the number of correctly matched point pairs is 12. Therefore, based on the fact that homography matrix M2 has the largest number of correctly matched point pairs (15), homography matrix M2 can be determined as the target homography matrix M.

[0220] In an embodiment of the present application, an optical flow method can be used to obtain an offset vector of a feature point in a panoramic image of a vehicle chassis area, and a first feature point set and a second feature point set are paired according to the vector to obtain a set of matching feature point pairs. Then, a preset number of matching feature point pairs are selected therefrom to determine a homography matrix. The feature points in the first feature point set are transformed in position by the matrix to obtain a first coordinate representing their position in the bird's-eye view image at the current moment. According to a preset distance threshold and the first coordinate and the second coordinate of each feature point, the number of correctly matched point pairs is determined, and the homography matrix with the most correctly matched point pairs is determined as the target homography matrix. In the above process, a preset number of matching feature point pairs can be selected to calculate the homography matrix, which can effectively reduce the complexity of the calculation. This method not only improves the calculation efficiency, but also ensures the accuracy of feature point matching, and provides a reliable data basis for the subsequent accurate generation of a panoramic image of the chassis area at the current moment.

[0221] Figure 8 This is a flow chart of an example of a method for generating a chassis area image of a vehicle provided in this application. Figure 8 , the process may include:

[0222] S801: Extract feature points.

[0223] In this step, the image stitching algorithm can be used to obtain a bird's-eye view of continuous frames, thereby obtaining a panoramic image of the chassis area of ​​the vehicle at the previous moment and a bird's-eye view image at the current moment. The FAST algorithm can be used to extract feature points from the panoramic image of the chassis area of ​​the vehicle at the previous moment to obtain a first feature point set; feature points can be extracted from the bird's-eye view image at the current moment to obtain a second feature point set.

[0224] S802: Calculate the pixel offset vector using the optical flow method.

[0225] S803: Determine the target homography matrix.

[0226] In this step, according to the number of correctly matched point pairs in the first feature point set and the second feature point set, a homography matrix with the largest number of matched correct point pairs can be selected from multiple homography matrices and determined as the target homography matrix.

[0227] S804. Extracting the vehicle's angle change observation value and distance change observation value from the target homography matrix.

[0228] S805. Determine the vehicle's angle change prediction value and distance change prediction value based on the vehicle's motion state parameters.

[0229] S806. Obtain the corrected angle change observation value and the corrected distance change observation value.

[0230] S807: construct a pixel transformation matrix in the image coordinate system to generate a panoramic image of the chassis area at the current moment.

[0231] In this step, the constructed pixel transformation matrix can be applied to the panoramic image of the chassis area of ​​the vehicle at the previous moment, the predicted chassis area image can be cut out, and the predicted chassis area image and the bird's-eye view image at the current moment can be fused to generate the panoramic image of the chassis area at the current moment.

[0232] The execution process of the vehicle chassis area image generation method example provided in the embodiment of the present application can refer to the process shown in the above method embodiment. The implementation principles and beneficial effects are similar and will not be repeated here.

[0233] Fig. 9 This is a schematic diagram of the structure of the vehicle chassis area image generation device provided in the embodiment of the present application. Fig. 9 The vehicle chassis area image generating device 10 may include:

[0234] The first processing module 11 is used to determine the angle change observation value and the distance change observation value of the vehicle according to the chassis area panoramic image of the vehicle at the previous moment and the bird's-eye view image at the current moment;

[0235] A second processing module 12 is used to determine the angle change prediction value and the distance change prediction value of the vehicle according to the first motion state parameter of the vehicle at a previous moment and the second motion state parameter of the vehicle at a current moment;

[0236] A correction module 13, used for correcting the angle change observation value and the distance change observation value according to the angle change prediction value and the distance change prediction value, to obtain a corrected angle change observation value and a corrected distance change observation value;

[0237] The third processing module 14 is used to generate a panoramic image of the chassis area at the current moment according to the corrected angle change observation value and the corrected distance change observation value.

[0238] The vehicle chassis area image generation device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0239] In a possible implementation manner, the correction module 13 is specifically configured to:

[0240] According to the prediction vector of the vehicle at the current moment, a preset correction calculation formula is used to correct the observation vector to obtain a corrected observation vector;

[0241] Among them, the prediction vector includes the angle change prediction value and the distance change prediction value; the observation vector includes the angle change observation value and the distance change observation value; the corrected observation vector includes the corrected angle change observation value and the corrected distance change observation value.

[0242] In a possible implementation manner, the second processing module 12 is specifically configured to:

[0243] Performing linearization processing on the first motion state parameter and the second motion state parameter to obtain a state transfer matrix;

[0244] Determine a state prediction vector and an initial covariance matrix of the vehicle at a current moment according to the state transfer matrix and the first motion state parameter, wherein the state prediction vector includes a current angle change prediction value and a current distance change prediction value, and the initial covariance matrix is ​​used to represent a deviation between the state prediction vector and the second motion state parameter;

[0245] A prediction vector of the vehicle at a current moment is determined according to the state prediction vector and the initial covariance matrix, wherein the prediction vector includes the angle change prediction value and the distance change prediction value.

[0246] In a possible implementation manner, the second processing module 12 is specifically configured to:

[0247] According to the initial covariance matrix, a prediction gain matrix is ​​calculated using a preset prediction gain calculation formula;

[0248] According to the preset update prediction calculation formula and the prediction gain matrix, the state prediction vector is updated and calculated to obtain the prediction vector of the vehicle at the current moment.

[0249] In a possible implementation manner, the third processing module 14 is specifically configured to:

[0250] Determine a pixel transformation matrix according to the corrected angle change observation value, the corrected distance change observation value, and the projection ratio;

[0251] According to the pixel transformation matrix, extracting a predicted chassis area image from the chassis area panoramic image of the vehicle at a previous moment;

[0252] The predicted chassis area image and the bird's-eye view image at the current moment are fused to generate a panoramic image of the chassis area at the current moment.

[0253] In a possible implementation manner, the first processing module 11 is specifically configured to:

[0254] Extracting feature points from the panoramic image of the chassis area of ​​the vehicle at a previous moment to obtain a first feature point set;

[0255] Extracting feature points from the bird's-eye view image at the current moment to obtain a second feature point set;

[0256] Obtaining a target homography matrix through the first feature point set and the second feature point set, wherein the target homography matrix is ​​used to transform the coordinates of the feature points in the panoramic image of the chassis area of ​​the vehicle at a previous moment into the bird's-eye view image at the current moment, so as to obtain the coordinates of each feature point in the bird's-eye view image at the current moment;

[0257] The angle change observation value and the distance change observation value of the vehicle are determined according to the target homography matrix.

[0258] In a possible implementation manner, the first processing module 11 is specifically configured to:

[0259] Taking any pixel point in the panoramic image of the chassis area as the center, determine the pixel points on the circumference corresponding to the center by a preset radius;

[0260] For each center, if there are a plurality of continuous pixels on the circumference corresponding to the center, the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the plurality of pixels is greater than or equal to a preset number, then the center is determined as a candidate feature point in the chassis area panoramic image;

[0261] All candidate feature points in the panoramic image of the chassis area of ​​the vehicle at a previous moment are screened by a non-maximum suppression algorithm to obtain the first feature point set.

[0262] In a possible implementation manner, the first processing module 11 is specifically configured to:

[0263] Taking any pixel point in the bird's-eye view image as the center, determine the pixel points on the circumference corresponding to the center by a preset radius;

[0264] For each center, if there are a plurality of continuous pixels on the circumference corresponding to the center, the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the plurality of pixels is greater than or equal to a preset number, then the center is determined as a candidate feature point in the bird's-eye view image;

[0265] All candidate feature points in the bird's-eye view image at the current moment are screened by a non-maximum suppression algorithm to obtain the second feature point set.

[0266] In a possible implementation manner, the first processing module 11 is specifically configured to:

[0267] Using an optical flow method to obtain an offset vector of a feature point in a panoramic image of a chassis area of ​​the vehicle at a previous moment;

[0268] Pairing the feature points in the first feature point set with the feature points in the second feature point set according to the offset vector to obtain a set of matching feature point pairs;

[0269] Selecting a preset number of matching feature point pairs from the set of matching feature point pairs to determine a homography matrix;

[0270] For each feature point in the first feature point set, use the homography matrix to perform position transformation to obtain a first coordinate, where the first coordinate represents the position coordinate of the feature point in the bird's-eye view image at the current moment;

[0271] Determine the number of correctly matched point pairs according to a preset distance threshold, a first coordinate and a second coordinate of each feature point, wherein the second coordinate of each feature point is the actual position coordinate of the feature point in the bird's-eye view image at the current moment;

[0272] The homography matrix with the largest number of correctly matched point pairs is determined as the target homography matrix.

[0273] The vehicle chassis area image generation device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0274] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 The electronic device 20 may include a processor 21, a memory 22 and a communication interface 24. Exemplarily, the processor 21, the memory 22 and the communication interface 24 are interconnected via a bus 23.

[0275] The memory 22 stores computer-executable instructions;

[0276] The processor 21 executes the computer-executable instructions stored in the memory 22 , so that the processor 21 executes the vehicle chassis area image generation method provided by the above method embodiment.

[0277] Optionally, the electronic device may be a domain controller in a vehicle, for example, any one of a chassis domain controller, a cockpit domain controller, and a driving domain controller.

[0278] The present application also provides a vehicle, including a vehicle body and Fig.10The electronic device shown is used to execute the vehicle chassis area image generation method in the above method embodiment.

[0279] Accordingly, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the vehicle chassis area image generation method provided in any of the above method embodiments.

[0280] Accordingly, an embodiment of the present application may also provide a computer program product, including a computer program. When the computer program is executed by a processor, the method for generating a chassis area image of a vehicle provided in any of the above method embodiments can be implemented.

[0281] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0282] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating a chassis area image of a vehicle, characterized in that: include: Determine an angle change observation value and a distance change observation value of the vehicle according to a panoramic image of a chassis area of ​​the vehicle at a previous moment and a bird's-eye view image at a current moment; Determining an angle change prediction value and a distance change prediction value of the vehicle according to a first motion state parameter of the vehicle at a previous moment and a second motion state parameter of the vehicle at a current moment; Correcting the angle change observation value and the distance change observation value according to the angle change prediction value and the distance change prediction value to obtain a corrected angle change observation value and a corrected distance change observation value; A panoramic image of the chassis area at the current moment is generated according to the corrected angle change observation value and the corrected distance change observation value.

2. The method according to claim 1, characterized in that The method of correcting the angle change observed value and the distance change observed value according to the angle change predicted value and the distance change predicted value to obtain a corrected angle change observed value and a corrected distance change observed value comprises: According to the prediction vector of the vehicle at the current moment, a preset correction calculation formula is used to correct the observation vector to obtain a corrected observation vector; Among them, the prediction vector includes the angle change prediction value and the distance change prediction value; the observation vector includes the angle change observation value and the distance change observation value; the corrected observation vector includes the corrected angle change observation value and the corrected distance change observation value.

3. The method according to claim 2, characterized in that The step of determining the angle change prediction value and the distance change prediction value of the vehicle according to the first motion state parameter of the vehicle at a previous moment and the second motion state parameter of the vehicle at a current moment includes: Performing linearization processing on the first motion state parameter and the second motion state parameter to obtain a state transfer matrix; Determine a state prediction vector and an initial covariance matrix of the vehicle at a current moment according to the state transfer matrix and the first motion state parameter, wherein the state prediction vector includes a current angle change prediction value and a current distance change prediction value, and the initial covariance matrix is ​​used to represent a deviation between the state prediction vector and the second motion state parameter; A prediction vector of the vehicle at a current moment is determined according to the state prediction vector and the initial covariance matrix, wherein the prediction vector includes the angle change prediction value and the distance change prediction value.

4. The method according to claim 3, characterized in that The step of determining the prediction vector of the vehicle at the current moment according to the state prediction vector and the initial covariance matrix includes: According to the initial covariance matrix, a prediction gain matrix is ​​calculated using a preset prediction gain calculation formula; According to the preset update prediction calculation formula and the prediction gain matrix, the state prediction vector is updated and calculated to obtain the prediction vector of the vehicle at the current moment.

5. The method according to any one of claims 1 to 4, characterized in that: The step of generating a panoramic image of the chassis area at the current moment according to the corrected angle change observation value and the corrected distance change observation value comprises: Determine a pixel transformation matrix according to the corrected angle change observation value, the corrected distance change observation value, and the projection ratio; According to the pixel transformation matrix, extracting a predicted chassis area image from the chassis area panoramic image of the vehicle at a previous moment; The predicted chassis area image and the bird's-eye view image at the current moment are fused to generate a panoramic image of the chassis area at the current moment.

6. The method according to any one of claims 1 to 4, characterized in that: The method of determining the angle change observation value and the distance change observation value of the vehicle according to the chassis area panoramic image of the vehicle at the previous moment and the bird's-eye view image at the current moment includes: Extracting feature points from the panoramic image of the chassis area of ​​the vehicle at a previous moment to obtain a first feature point set; Extracting feature points from the bird's-eye view image at the current moment to obtain a second feature point set; Obtaining a target homography matrix through the first feature point set and the second feature point set, wherein the target homography matrix is ​​used to transform the coordinates of the feature points in the panoramic image of the chassis area of ​​the vehicle at a previous moment into the bird's-eye view image at the current moment, so as to obtain the coordinates of each feature point in the bird's-eye view image at the current moment; The angle change observation value and the distance change observation value of the vehicle are determined according to the target homography matrix.

7. The method according to claim 6, characterized in that The extracting feature points from the panoramic image of the chassis area of ​​the vehicle at the last moment to obtain a first feature point set includes: Taking any pixel point in the panoramic image of the chassis area as the center, determine the pixel points on the circumference corresponding to the center by a preset radius; For each center, if there are a plurality of continuous pixels on the circumference corresponding to the center, the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the plurality of pixels is greater than or equal to a preset number, then the center is determined as a candidate feature point in the chassis area panoramic image; All candidate feature points in the panoramic image of the chassis area of ​​the vehicle at a previous moment are screened by a non-maximum suppression algorithm to obtain the first feature point set.

8. The method according to claim 6, characterized in that The extracting feature points from the bird's-eye view image at the current moment to obtain a second feature point set includes: Taking any pixel point in the bird's-eye view image as the center, determine the pixel points on the circumference corresponding to the center by a preset radius; For each center, if there are a plurality of continuous pixels on the circumference corresponding to the center, the brightness difference between the center and the center is greater than a preset brightness threshold, and the number of the plurality of pixels is greater than or equal to a preset number, then the center is determined as a candidate feature point in the bird's-eye view image; All candidate feature points in the bird's-eye view image at the current moment are screened by a non-maximum suppression algorithm to obtain the second feature point set.

9. The method according to claim 7 or 8, characterized in that: The step of obtaining a target homography matrix by using the first feature point set and the second feature point set includes: Using an optical flow method to obtain an offset vector of a feature point in a panoramic image of a chassis area of ​​the vehicle at a previous moment; Pairing the feature points in the first feature point set with the feature points in the second feature point set according to the offset vector to obtain a set of matching feature point pairs; Selecting a preset number of matching feature point pairs from the set of matching feature point pairs to determine a homography matrix; For each feature point in the first feature point set, use the homography matrix to perform position transformation to obtain a first coordinate, where the first coordinate represents the position coordinate of the feature point in the bird's-eye view image at the current moment; Determine the number of correctly matched point pairs according to a preset distance threshold, a first coordinate and a second coordinate of each feature point, wherein the second coordinate of each feature point is the actual position coordinate of the feature point in the bird's-eye view image at the current moment; The homography matrix with the largest number of correctly matched point pairs is determined as the target homography matrix.

10. A vehicle chassis area image generation device, characterized in that: include: A first processing module, for determining an angle change observation value and a distance change observation value of the vehicle according to a panoramic image of a chassis area of ​​the vehicle at a previous moment and a bird's-eye view image at a current moment; A second processing module, used for determining an angle change prediction value and a distance change prediction value of the vehicle according to the first motion state parameter of the vehicle at a previous moment and the second motion state parameter of the vehicle at a current moment; A correction module, used for correcting the angle change observation value and the distance change observation value according to the angle change prediction value and the distance change prediction value, so as to obtain a corrected angle change observation value and a corrected distance change observation value; The third processing module is used to generate a panoramic image of the chassis area at the current moment according to the corrected angle change observation value and the corrected distance change observation value.

11. An electronic device, characterized in that: include: Processor, memory and communication interface; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the vehicle chassis area image generation method according to any one of claims 1 to 9.

12. A vehicle, characterized in that: include: A vehicle body and the electronic device according to claim 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method for generating a chassis area image of a vehicle as described in any one of claims 1 to 9.

Images