Reverse assist method, device, electronic device and storage medium

Abstract

An embodiment of the present application discloses a reverse assist method, device, electronic device, and storage medium. The method includes: during the forward movement of the vehicle, determining the driving trajectory of the vehicle according to the first sensor information of the vehicle; determining the mapping position of the vehicle on the driving trajectory, and obtaining the visual mapping information collected by the camera device of the vehicle when the vehicle is at the mapping position; when it is detected that the reverse assist module of the vehicle is turned on, obtaining the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle; collecting the current image information corresponding to the trajectory positioning information through the camera device, and determining the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information; determining the true positioning information of the vehicle at the current position according to the trajectory positioning information and the visual positioning information. Implementing the embodiments of the present application can solve the problem of angular error in dead reckoning, thereby improving the accuracy of reverse assist.

Description

Technical Field

[0001] This application relates to the field of automotive electronics, and particularly to a reverse assistance method, device, electronic device, and storage medium. Background Art

[0002] Currently, during driving, drivers often encounter some narrow and long roads which are generally not suitable for passage. The driver needs to use the rearview mirror or reverse camera to carefully return along the original route. In response to this situation, the existing reverse assistance technology extracts vehicle information such as vehicle speed and steering wheel angle from the vehicle through the Controller Area Network (CAN) bus, and uses this vehicle information to perform dead reckoning on the vehicle to obtain the driving trajectory, so that the vehicle can automatically return to a certain distance on the original path position according to this driving trajectory. However, this traditional reverse assistance technology is difficult to solve the angle error caused when the vehicle turns, resulting in a large deviation when the vehicle returns to the starting point. Summary of the Invention

[0003] Embodiments of this application disclose a reverse assistance method, device, electronic device, and storage medium, which can solve the angle error problem in dead reckoning, thereby improving the accuracy of reverse assistance.

[0004] Embodiments of this application disclose a reverse assistance method, characterized in that the method includes:

[0005] During the forward movement of the vehicle, determine the driving trajectory of the vehicle according to the first sensor information of the vehicle; the first sensor information includes the first inertial sensor data and the first odometer sensor data collected by the vehicle during the forward movement;

[0006] Determine the mapping position of the vehicle on the driving trajectory, and obtain the visual mapping information collected by the camera device of the vehicle when the vehicle is at the mapping position, so as to obtain complete mapping information;

[0007] When it is detected that the reverse assistance module of the vehicle is turned on, obtain the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle; the second sensor information includes the second inertial sensor data and the second odometer sensor data collected by the vehicle during the reverse movement;

[0008] Collect the current image information corresponding to the trajectory positioning information through the camera device, and determine the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information;

[0009] Determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the visual positioning information.

[0010] As an alternative implementation, determining the mapping position of the vehicle on the driving trajectory includes:

[0011] Detect the driving state of the vehicle;

[0012] Determine a corresponding preset mapping interval according to the driving state of the vehicle;

[0013] Select the mapping position of the vehicle on the driving trajectory from the driving trajectory according to the preset mapping interval.

[0014] As an alternative implementation, the visual mapping information in the complete mapping information includes image feature points; determining the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information includes:

[0015] Extract first feature points from the current image information;

[0016] Match the first feature points with the image feature points included in the complete mapping information to determine second feature points that match the first feature points from the image feature points;

[0017] According to the complete mapping information corresponding to the second feature points, determine the relative mapping position of the vehicle in the vehicle motion coordinate system corresponding to the second feature points from the mapping positions included in the complete mapping information; the coordinate origin of the vehicle motion coordinate system is at a preset path length from the reverse starting position corresponding to the activation of the reverse assist module of the vehicle;

[0018] Obtain the transformation matrix between the first feature points and the second feature points;

[0019] Determine the visual positioning information of the vehicle in the vehicle motion coordinate system through the relative mapping position of the vehicle and the transformation matrix.

[0020] As an alternative implementation, determining the visual positioning information of the vehicle in the vehicle motion coordinate system through the relative mapping position of the vehicle and the transformation matrix includes:

[0021] Obtain the relative pose of the camera when collecting the current image information with respect to when collecting the visual mapping information according to the transformation matrix;

[0022] Obtain the external parameters of the camera device relative to the vehicle;

[0023] Obtain the relative pose of the vehicle at the current position relative to the vehicle at the mapping position according to the relative pose of the camera and the extrinsic parameters of the imaging device relative to the vehicle;

[0024] Determine the visual positioning information of the vehicle in the vehicle motion coordinate system based on the relative position of the vehicle for mapping and the relative pose of the vehicle.

[0025] As an alternative implementation, the obtaining the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle includes:

[0026] Integrate the second inertial sensor data and the second odometer sensor data included in the second sensor information of the vehicle respectively, and determine the pose increment according to the integration results of the second inertial sensor data and the integration results of the second odometer sensor data;

[0027] Accumulate the pose increment to the previous fusion result to obtain the trajectory positioning information of the vehicle on the driving trajectory.

[0028] As an alternative implementation, after determining the visual positioning information of the vehicle in the vehicle motion coordinate system by the relative position of the vehicle for mapping and the transformation matrix, the method further includes:

[0029] Perform coordinate transformation on the visual positioning information in the vehicle motion coordinate system to obtain the visual positioning information in the world coordinate system; the origin of the world coordinate system is the starting point of the driving trajectory.

[0030] As an alternative implementation, before detecting that the reverse assist module of the vehicle is turned on, the method further includes:

[0031] When detecting a touch operation instruction, a voice instruction or an enabling instruction sent from a terminal device, control the vehicle to turn on the reverse assist module.

[0032] An embodiment of the present application discloses a reverse assist device, the device includes:

[0033] A trajectory mapping module, configured to determine the driving trajectory of the vehicle according to the first sensor information of the vehicle during the forward movement of the vehicle; the first sensor information includes the first inertial sensor data and the first odometer sensor data collected by the vehicle during the forward movement;

[0034] A visual mapping module, configured to determine the mapping position of the vehicle on the driving trajectory, and obtain the visual mapping information collected by the imaging device of the vehicle when the vehicle is at the mapping position, so as to obtain complete mapping information;

[0035] A trajectory positioning module, configured to obtain trajectory positioning information of the vehicle on the driving trajectory according to second sensor information of the vehicle when detecting that the reverse assist module of the vehicle is turned on; the second sensor information includes second inertial sensor data and second odometer sensor data collected during the reverse driving of the vehicle.

[0036] A vision positioning module, configured to collect current image information corresponding to the trajectory positioning information through the imaging device, and determine the vision positioning information of the vehicle according to the feature matching relationship between the current image information and the vision mapping information.

[0037] A fusion positioning module, configured to determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the vision positioning information.

[0038] An embodiment of the present application discloses an electronic device, including a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor implements any one of the reverse assist methods disclosed in the embodiments of the present application.

[0039] An embodiment of the present application discloses a computer-readable storage medium, which stores a computer program, wherein the computer program enables a computer to execute any one of the reverse assist methods disclosed in the embodiments of the present application.

[0040] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0041] Collect first sensor information including first inertial sensor data and first odometer sensor data during the forward driving of the vehicle, and determine the driving trajectory of the vehicle according to the first sensor information; determine a mapping position on the driving trajectory of the vehicle, and collect vision mapping information through the imaging device of the vehicle when the vehicle is at the mapping position, and construct complete mapping information by using the driving trajectory, the mapping position on the driving trajectory, and the corresponding vision mapping information; when detecting that the reverse assist module of the vehicle is turned on, collect second sensor information including second inertial sensor data and second odometer sensor data during the reverse driving of the vehicle, and determine the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information; collect current image information corresponding to the trajectory positioning information through the imaging device of the vehicle, and determine the vision positioning information of the vehicle according to the feature matching relationship between the current image information and the vision mapping information; determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the vision positioning information.

[0042] In the embodiments of the present application, complete mapping information is generated based on the first sensor information and the visual mapping information during the forward movement of the vehicle, and the trajectory positioning information and the visual positioning information are fused during the reverse movement of the vehicle to obtain the true positioning information of the vehicle at the current position, which can solve the angle error problem existing in dead reckoning, thereby improving the accuracy of reverse assistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1 is a schematic flowchart of a reverse assistance method disclosed in an embodiment of the present application;

[0045] Figure 2 is a schematic flowchart of another reverse assistance method disclosed in an embodiment of the present application;

[0046] Figure 3 is a hardware architecture diagram of a reverse assistance system disclosed in an embodiment of the present application;

[0047] Figure 4 is a schematic flowchart of another reverse assistance method disclosed in an embodiment of the present application;

[0048] Figure 5 is a process diagram of visual mapping and positioning disclosed in an embodiment of the present application;

[0049] Figure 6 is a schematic structural diagram of a reverse assistance device disclosed in an embodiment of the present application;

[0050] Figure 7 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0052] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0053] The embodiments of the present application disclose a reverse assistance method, device, electronic device, and storage medium, which can solve the problem of angular error in dead reckoning, thereby improving the accuracy of reverse assistance. The following will be described in detail separately.

[0054] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a reverse assistance method disclosed in the embodiments of the present application. Among them, Figure 1 the described reverse assistance method is applicable to electronic devices such as in-vehicle devices or vehicle controllers, etc., and the embodiments of the present application do not make any limitations. As Figure 1 shown, the reverse assistance method may include the following steps:

[0055] 101. During the forward movement of the vehicle, determine the driving trajectory of the vehicle according to the first sensor information of the vehicle.

[0056] The first sensor information includes the first inertial sensor data and the first odometer sensor data collected during the forward movement of the vehicle. The first inertial sensor data and the first odometer sensor data are the vehicle information measured by the inertial sensor and the odometer sensor of the vehicle during the forward movement of the vehicle, and can be used to determine the driving trajectory of the vehicle.

[0057] An inertial sensor (Inertial Measurement Unit, IMU) is a sensor used to detect and measure acceleration and rotational motion. Using the principle of inertia law, it may include an accelerometer and an angular velocity meter (such as a gyroscope, etc.).

[0058] The odometer sensor can detect the distance that the wheel moves relative to the ground, the change amount of the direction angle, and the speed of the wheel, so as to calculate the pose (position and attitude) of the vehicle.

[0059] During the forward movement of the vehicle during normal operation, the vehicle obtains the first sensor information through the Controller Area Network (CAN) bus, and uses the first sensor information to calculate the vehicle's trajectory through the vehicle kinematic model. This process can be called dead reckoning, and dead reckoning is continuously running during the forward and backward stages of vehicle path tracking. During the forward movement of the vehicle, that is, when the vehicle's reverse assist function is not triggered, dead reckoning is performed on the vehicle based on the first sensor information, and the calculated pose is used to create a driving trajectory and provide a visual map origin. Among them, the visual map can include the driving trajectory and the coordinates of each position on the driving trajectory in the world coordinate system.

[0060] The driving trajectory includes the pose information of each position of the vehicle in the world coordinate system during the forward movement. The coordinate origin of the world coordinate system is the starting point of the driving trajectory.

[0061] 102. Determine the mapping position of the vehicle on the driving trajectory, and obtain the visual mapping information collected by the vehicle's camera device when the vehicle is at the mapping position to obtain complete mapping information.

[0062] Among them, classified by the number of camera lenses, the camera device can be a monocular camera, a binocular camera, a trinocular camera, etc.; classified by the camera's viewing angle, the camera device can be a wide-angle camera, a telephoto camera, etc.; classified by the camera's installation position, the camera device can be a front camera, a rear camera, a side camera, a surround camera, etc.

[0063] In some embodiments, multiple mapping positions are selected from the driving trajectory at a preset mapping interval, and the visual mapping information collected by the vehicle's camera device when the vehicle is at each mapping position is obtained. The visual mapping information can include the effective feature points extracted from the images captured by the camera device and the corresponding pose of the camera device. Exemplarily, the effective feature points during the vehicle's forward movement can be extracted by the front camera in the vehicle's panoramic surround system, where the effective feature points can be extracted from environmental feature information such as road markers and road markings during the vehicle's forward movement. At the same time, a fish-eye camera is used to increase the field of view, capture richer environmental feature information, and use the environmental feature information for mapping. The effective feature points and descriptors in the environmental feature information are saved in the server memory to obtain visual mapping information.

[0064] Optionally, the image feature points can be extracted from the video input by the camera device in real time at a preset sampling interval, and the corresponding camera pose information is saved in the server memory as visual mapping information. Among them, if image feature points are continuously extracted during the vehicle's driving process, it will cause excessive occupation of the server memory, resulting in information redundancy and a decrease in the running speed. Therefore, the input video can be extracted for image feature points by specifying an appropriate preset sampling interval.

[0065] The complete mapping information may include the driving trajectory, the mapping positions determined on the driving trajectory, and the visual mapping information corresponding to the mapping positions.

[0066] 103. When it is detected that the reverse assist module of the vehicle is turned on, obtain the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle.

[0067] The reverse assist module may be a functional module that controls the vehicle to automatically return to the original path position at a certain distance after the vehicle has traveled a certain distance. Based on different scenario types, this section of the driving path may include straight, S-shaped, L-shaped, U-shaped, etc.

[0068] When it is detected that the reverse assist module of the vehicle is turned on, the second sensor information of the vehicle can be obtained through the CAN bus to determine the trajectory positioning information of the vehicle on the driving trajectory.

[0069] The second sensor information includes the second inertial sensor data and the second odometer sensor data collected during the reverse driving of the vehicle. The second inertial sensor data and the second odometer sensor data are the vehicle information measured by the inertial sensor and the odometer sensor during the reverse driving of the vehicle, and can be used to obtain the trajectory positioning information of the vehicle on the driving trajectory.

[0070] The trajectory positioning information may be the current pose of the vehicle when it returns to a certain position on the driving trajectory during the reverse driving process deduced from the second sensor information.

[0071] In some embodiments, when it is detected that the reverse assist module of the vehicle is turned on, the second inertial sensor data and the second odometer sensor data at the same moment are fused through the extended Kalman filter algorithm to determine the trajectory positioning information of the vehicle. In the extended Kalman filter algorithm, more trust is placed in the translation amount in the second odometer sensor data and the rotation amount in the second inertial sensor data.

[0072] 104. Collect the current image information corresponding to the trajectory positioning information through the imaging device, and determine the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information.

[0073] The current image information includes the current image feature points extracted from the image captured by the imaging device when the vehicle returns to the current position during the reverse driving process. The current image feature points may be extracted from environmental feature information such as road markers and road markings during the reverse driving process of the vehicle.

[0074] The imaging device for collecting the current image information may be the front camera in the panoramic surround view system, or a fish-eye camera, and the specific type is not limited.

[0075] The feature matching relationship can be determined by feature matching algorithms such as the ORB algorithm, SURF algorithm, SIFT algorithm, etc., and is not specifically limited. Exemplarily, when the vehicle is at the current position, the current image is captured by a camera device, and effective feature points are extracted from the current image to generate current image information; the effective feature points included in the current image information are feature-matched with the effective feature points included in the visual mapping information, and after successful matching, the visual positioning information of the vehicle can be obtained.

[0076] The visual positioning information can be the current pose of the vehicle returning to a certain position in the driving trajectory during the reverse process, deduced from the visual mapping information and the current image information obtained by the camera device of the vehicle. Based on the known trajectory positioning information, the visual positioning information is used as the basis for relocalization.

[0077] 105. Determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the visual positioning information.

[0078] The true positioning information includes the pose of the vehicle at the current position relative to the world coordinate system. Among them, the coordinate origin of the world coordinate system is the origin of the driving trajectory.

[0079] Fusing the trajectory positioning information and the visual positioning information can integrate the advantages of each sensor and obtain a higher-precision pose of the vehicle. The fusion method can be an algorithm based on filter estimation, such as the Extended Kalman Filter (EKF), Kalman Filter (KF), etc.; it can also be an algorithm based on Bayesian inference, such as the Markov Localization (MKV), Monte Carlo Localization (MCL), etc., and is not specifically limited.

[0080] Exemplarily, the current predicted pose of the vehicle is obtained by integrating the data of the second inertial sensor; the data of the second odometer sensor is fused through the EFK algorithm to correct the translation amount in the current predicted pose; and then the visual positioning information is fused to correct the angular cumulative error caused by the rotation of the vehicle. By integrating the advantages of each sensor, a higher-precision pose can be obtained.

[0081] In the embodiment of the present application, by generating complete mapping information according to the first sensor information and the visual mapping information during the forward movement of the vehicle, and fusing the trajectory positioning information and the visual positioning information during the reverse movement of the vehicle to obtain the true positioning information of the vehicle at the current position, the problem of angular error existing in dead reckoning can be solved, thereby improving the accuracy of reverse assistance.

[0082] Please refer to Figure 2 ,Figure 2 is a schematic flowchart of another reverse assist method disclosed in an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0083] 201. During the forward movement of the vehicle, determine the driving trajectory of the vehicle according to the first sensor information of the vehicle.

[0084] Please refer to Figure 3 , Figure 3 is a hardware architecture diagram of a reverse assist system disclosed in an embodiment of the present application. As Figure 3 shown, the hardware architecture of the reverse assist system may include the following structures:

[0085] The in-vehicle multimedia host HUT in the vehicle can be used for the interaction between the vehicle and the driver. The interface processor IP can be used for vehicle communication and control. The controller area network CAN bus can be used for communication between the system and other electronic and electrical systems of the vehicle. The first sensor information and the second sensor information measured by the odometer sensor and the inertial sensor can be transmitted to the interface processor IP through the CAN bus, and then transmitted to the controller of the reverse assist system through the interface processor IP. The visual image information determined by the imaging device can also be transmitted to the controller. The controller can be used to control the vehicle to return to the original path position at a certain distance along the original path according to the real positioning information determined by the trajectory positioning information and the visual positioning information. Once the controller is powered on, the reverse assist system changes from the off state to the standby state, and runs dead reckoning in the background to determine the driving trajectory using the first sensor information and determine the visual mapping information through the imaging device to obtain the complete mapping information.

[0086] 202. Detect the driving state of the vehicle.

[0087] The driving state of the vehicle may include a straight driving state and a turning state. Exemplarily, by detecting vehicle information such as the steering wheel angle and the tire angle of the vehicle, it can be determined whether the driving state of the vehicle belongs to the turning state or the straight driving state.

[0088] 203. Determine the corresponding preset mapping interval according to the driving state of the vehicle.

[0089] When the vehicle is in different driving states, the requirements for mapping are also different. Exemplarily, when the vehicle is in the turning state, the environmental feature information that the vehicle needs to obtain within a short distance is more complex. Therefore, the preset mapping interval corresponding to the vehicle in the turning state can be smaller than the preset mapping interval corresponding to the vehicle in the straight driving state.

[0090] 204. Select the mapping positions of the vehicle on the driving trajectory at the preset mapping interval from the driving trajectory.

[0091] The driving trajectory can be determined by continuously dead reckoning the first sensor information, and the visual mapping information can be mapped not on the entire driving trajectory but on the mapping positions selected from the driving trajectory because continuous mapping will occupy too much server memory. Therefore, it is necessary to select mapping positions from the driving trajectory according to a preset mapping interval.

[0092] 205. Obtain the visual mapping information collected by the vehicle's camera device when the vehicle is at the mapping position to obtain complete mapping information.

[0093] 206. When a touch operation instruction, a voice instruction, or an enabling instruction sent from a terminal device is detected, control the vehicle to activate the reverse assist module.

[0094] Among them, the touch operation instruction can be generated by the driver clicking the soft key in the HUT. Optionally, when the controller detects the touch operation instruction or the driver's voice instruction detected by the voice recognition module in the HUT, it controls the vehicle to activate the reverse assist module.

[0095] Among them, the enabling instruction sent from the terminal device can include the enabling instruction sent by the driver through modules such as mobile phone Bluetooth or Ethernet. Optionally, after the controller detects the enabling instruction sent by the driver through modules such as mobile phone Bluetooth or Ethernet, it sends the enabling instruction to the wireless communication system, such as the Bluetooth system and the Ethernet system, etc., to control the vehicle to activate the reverse assist module.

[0096] 207. When it is detected that the reverse assist module of the vehicle is activated, integrate the second inertial sensor data and the second odometer sensor data included in the second sensor information of the vehicle respectively, and determine the pose increment according to the integration result of the second inertial sensor data and the integration result of the second odometer sensor data.

[0097] In the real working condition, there will be a large bias in the inertial sensor. The bias is generally related to temperature, factory accuracy, and actual motion dynamics. Therefore, in pose estimation, these biases will have a cumulative effect, resulting in cumulative errors. Optionally, the bias of the inertial sensor can be dynamically estimated to obtain more accurate integration of the inertial sensor data.

[0098] Exemplarily, by integrating the second inertial sensor data and the second odometer sensor data respectively, adding the integration result of the second inertial sensor data and the integration result of the second odometer sensor data, and determining the added result as the pose increment of the two sensors in the same time period.

[0099] As an optional implementation manner, the estimated state quantity of the vehicle can be:

[0100]

[0101] Among them, p represents the position in the world coordinate system, v represents the velocity in the world coordinate system, θ represents the heading angle in the world coordinate system, and the xy subscript represents the xy axis direction. b accx represents the acceleration bias in the x-axis direction, b accy represents the acceleration bias in the y-axis direction, b gyrz represents the angular velocity bias around the z-axis.

[0102] (1) Integrate the second inertial sensor data of the vehicle, and the process is as follows:

[0103] Position update:

[0104] p t = p t-1 + v t-1 * δt + 0.5 * a G * δt 2 ; Formula (1);

[0105] Among them, t represents time, δt represents the time increment, and a G represents the acceleration in the world coordinate system;

[0106] Velocity update:

[0107] v t = v t-1 + a G * δt; Formula (2);

[0108] Attitude update:

[0109] θ t = θ t-1 + δθ; Formula (3);

[0110] δθ represents the angle increment;

[0111] Bias update:

[0112] b acc(t) = b acc(t-1) ; Formula (4);

[0113] b acc represents the acceleration bias.

[0114] b gyr(t) = b gyr(t-1) ; Formula (5);

[0115] b gyr represents the angular velocity bias;

[0116] Among them:

[0117] a G = R * (aL +b acc ); Equation (6);

[0118] a L represents the acceleration in the IMU coordinate system; R represents the rotation matrix;

[0119] δθ = (V gyr +b gyr ) * δt; Equation (7);

[0120] V gyr represents the angular velocity in the IMU coordinate system;

[0121] F matrix:

[0122]

[0123] wherein, the F matrix can represent the fundamental matrix;

[0124] (2) Update the estimated state variables using the data of the second odometer sensor;

[0125] The data of the second odometer sensor provides position constraints;

[0126]

[0127] wherein, the H matrix can represent the homography matrix;

[0128] (3) Update the estimated state variables using the visual positioning information;

[0129] The visual positioning information provides angular constraints;

[0130] H = [0 0 0 0 1 0 0 0]; Equation (10).

[0131] 208. Accumulate the pose increment to the previous fusion result to obtain the trajectory positioning information of the vehicle on the driving trajectory.

[0132] The previous fusion result can be the result of integrating and fusing the second inertial sensor data and the second odometer sensor data respectively last time, that is, the trajectory positioning information of the vehicle on the driving trajectory last time.

[0133] 209. Collect the current image information corresponding to the trajectory positioning information through the camera device, and determine the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information.

[0134] 210. Determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the visual positioning information.

[0135] As an implementation manner, the pose increments corresponding to two sensors in the same time period are respectively accumulated onto their respective previous fusion results to obtain the poses to be fused of the second inertial sensor data and the second odometer sensor data. The extended Kalman filter is used to fuse the poses to be fused of the second inertial sensor data and the second odometer sensor data, so as to obtain the trajectory positioning information of the vehicle on the driving trajectory. The second inertial sensor data mainly provides the rotation amount, and the second odometer sensor data mainly provides the translation amount and the position constraint. Finally, the fused visual positioning information can correct the cumulative error caused by the second inertial sensor data and the second odometer sensor data, and at the same time, the second inertial sensor data and the second odometer sensor data are used to smooth and filter the incorrect poses provided by the visual positioning information of the camera device. Among them, the visual positioning information mainly provides the angle constraint and is the current pose of the vehicle without cumulative error.

[0136] Since dead reckoning is performed by directly obtaining vehicle information using the vehicle's CAN bus, there will be large errors in rotation or angle. Introducing an inertial sensor can solve the errors in rotation and angle. At the same time, determining the visual positioning information through the camera device, that is, performing visual relocalization on the vehicle, also solves the errors in angle to a certain extent, greatly improving the accuracy of reverse assistance and the driver experience.

[0137] Please refer to Figure 4 , Figure 4 which is a schematic flowchart of another reverse assistance method disclosed in the embodiments of the present application.

[0138] 401. During the forward movement of the vehicle, determine the driving trajectory of the vehicle according to the first sensor information of the vehicle.

[0139] 402. Determine the mapping position of the vehicle on the driving trajectory, and obtain the visual mapping information collected by the camera device of the vehicle when the vehicle is at the mapping position, so as to obtain the complete mapping information.

[0140] 403. When it is detected that the reverse assistance module of the vehicle is turned on, obtain the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle.

[0141] 404. Collect the current image information corresponding to the trajectory positioning information through the camera device, and extract the first feature points from the current image information.

[0142] When the vehicle returns to the current position again, extract the first feature points from the current image information collected by the camera device. The first feature points are the feature points extracted from the current image information.

[0143] 405. Match the first feature point with the image feature points included in the complete mapping information to determine, from the image feature points, a second feature point that matches the first feature point.

[0144] The visual mapping information in the complete mapping information includes image feature points. A second feature point that matches the first feature point is determined through feature matching from the image feature points included in the complete mapping information.

[0145] 406. According to the complete mapping information corresponding to the second feature point, determine the relative vehicle mapping position in the vehicle motion coordinate system corresponding to the second feature point from the mapping positions included in the complete mapping information.

[0146] The origin of the vehicle motion coordinate system is at a preset path length from the reverse starting position corresponding to when the vehicle's reverse assist module is turned on. Exemplarily, the user can preset the preset path length. For example, 50 meters. Whenever the vehicle's reverse assist module is turned on, the vehicle will automatically return to the original path position 50 meters from the reverse starting position. This original path position at 50 meters is the reverse target point during the reverse assist process. Each reverse uses the reverse target point as the origin of the vehicle motion coordinate system. Therefore, the origin of the vehicle motion coordinate system changes with the vehicle position corresponding to when the reverse assist module is turned on. Using the reverse target point as the origin of the vehicle motion coordinate system is beneficial for using the relative pose information between the camera device corresponding to the current position and the camera device corresponding to the mapping position to perform coordinate transformation, obtaining the pose of the vehicle at the current position relative to the vehicle motion coordinate system, and improving the efficiency of determining the visual positioning information.

[0147] 407. Obtain the transformation matrix between the first feature point and the second feature point.

[0148] Optionally, after determining the first feature point and the second feature point that matches the first feature point, calculate the fundamental matrix and the essential matrix. The essential matrix is the fundamental matrix in normalized image coordinates and can be used to calculate the transformation matrix. The transformation matrix includes a rotation matrix and a translation matrix.

[0149] 408. According to the transformation matrix, obtain the relative camera pose of the camera device when collecting the current image information with respect to when collecting the visual mapping information.

[0150] The transformation matrix in the two-dimensional coordinate system does not include depth information, and the relative camera pose can be the rotation amount of the camera device when collecting the current image information with respect to when collecting the visual mapping information.

[0151] Please refer to Figure 5 , Figure 5 which is a process diagram of visual mapping and positioning disclosed in an embodiment of the present application. As shown in Figure 5In the U-shaped path described in [reference], the vehicle advances from position A to position B, and then when the controller detects that the reverse assist module is turned on, the vehicle automatically returns from position B to point A along the original path.

[0152] During the vehicle's forward movement from position A to position B, the first sensor information is continuously obtained in real time through dead reckoning to generate a driving trajectory, and multiple mapping positions are selected on the driving trajectory, such as Figure 5 the mapping position P2 in [reference]. When the vehicle is at the mapping position P2, the camera C2 on the vehicle transmits two-dimensional image information containing image feature points Point, and feature points and descriptors are extracted from the two-dimensional image information. Thousands of pixel points similar to the image feature point Point are extracted as features and stored in the server memory. These processes all run in the background without the need for driver-triggered operations. When the driver advances to the end position Pend, the reverse assist module is activated, and the vehicle reverses 100 meters back. The position 100 meters away from the end position Pend is determined as the starting position P0. A vehicle motion coordinate system Oxy is established with the starting position P0 as the origin. Assume that when the vehicle retreats from the end position Pend to the current position P1, the rotation amount of camera C1 relative to camera C2 is determined through the feature matching relationship between the current image information obtained at the current position P1 and the visual mapping information determined at the mapping position P2. The rotation amount of camera C1 relative to camera C2 is determined as the relative pose of the camera.

[0153] 409. Obtain the extrinsic parameters of the imaging device relative to the vehicle.

[0154] The extrinsic parameters of the imaging device relative to the vehicle are technical parameters provided when the imaging device is installed. The extrinsic parameters of the imaging device relative to the vehicle when the vehicle is at the mapping position are the same as those when the vehicle is at the current position.

[0155] 410. Obtain the relative pose of the vehicle at the current position relative to the vehicle at the mapping position based on the relative pose of the cameras and the extrinsic parameters of the imaging device relative to the vehicle.

[0156] The relative pose of the vehicle can be the rotation amount of the vehicle at the current position P1 relative to the vehicle at the mapping position P2.

[0157] 411. Determine the visual positioning information of the vehicle in the vehicle motion coordinate system through the relative position of vehicle mapping and the relative pose of the vehicle.

[0158] The relative position of vehicle mapping includes the relative position of the vehicle at the mapping position P2 relative to the vehicle motion coordinate system, that is, the rotation amount of the vehicle at the mapping position P2 relative to the starting position P0.

[0159] The visual positioning information of the vehicle in the vehicle motion coordinate system includes the rotation amount of the vehicle at the current position P1 relative to the vehicle at the starting position P0.

[0160] 412. Perform coordinate transformation on the visual positioning information in the vehicle motion coordinate system to obtain the visual positioning information in the world coordinate system.

[0161] The origin of the world coordinate system is the starting point of the driving trajectory. Since the driving trajectory is determined in the world coordinate system, the trajectory positioning information determined from the driving trajectory is also represented in the world coordinate system. To better fuse the visual positioning information with the trajectory positioning information, the visual positioning information in the vehicle motion coordinate system can be first converted to the visual positioning information in the world coordinate system to obtain the true positioning information of the vehicle in the world coordinate system.

[0162] 413. Determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the visual positioning information.

[0163] After introducing the inertial sensor data and the odometer sensor data, dead reckoning is performed on the vehicle. As time and distance increase, the cumulative error will slowly increase. By fusing the visual positioning information with the trajectory positioning information, repositioning of the vehicle is achieved, the cumulative error caused by dead reckoning can be eliminated, and the accuracy of reverse assistance can be improved.

[0164] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a reverse assistance device disclosed in an embodiment of the present application. The device can be applied to electronic devices such as in-vehicle devices or vehicle controllers, etc., and is not specifically limited. As Figure 6 shown, the reverse assistance device 600 may include: a trajectory mapping module 610, a visual mapping module 620, a trajectory positioning module 630, a visual positioning module 640, and a fusion positioning module 650.

[0165] The trajectory mapping module 610 is used to determine the driving trajectory of the vehicle according to the first sensor information of the vehicle during the forward movement of the vehicle; the first sensor information includes the first inertial sensor data and the first odometer sensor data collected by the vehicle during the forward movement.

[0166] The visual mapping module 620 is used to determine the mapping position of the vehicle on the driving trajectory, and obtain the visual mapping information collected by the camera device of the vehicle when the vehicle is at the mapping position to obtain the complete mapping information.

[0167] A trajectory positioning module 630, configured to obtain trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle when it is detected that the reverse assist module of the vehicle is turned on; the second sensor information includes second inertial sensor data and second odometer sensor data collected during the reverse driving process of the vehicle;

[0168] A vision positioning module 640, configured to collect current image information corresponding to the trajectory positioning information through a camera device, and determine the vision positioning information of the vehicle according to the feature matching relationship between the current image information and the vision mapping information;

[0169] A fusion positioning module 650, configured to determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the vision positioning information.

[0170] In one embodiment, the vision mapping module 620 may include: a detection unit, a determination unit, and a selection unit;

[0171] The detection unit is configured to detect the driving state of the vehicle;

[0172] The determination unit is configured to determine a corresponding preset mapping interval according to the driving state of the vehicle;

[0173] The selection unit is configured to select the mapping position of the vehicle on the driving trajectory from the driving trajectory at the preset mapping interval.

[0174] In one embodiment, the vision positioning module 640 is further configured to extract first feature points from the current image information; match the first feature points with the image feature points included in the complete mapping information to determine second feature points that match the first feature points from the image feature points; determine the relative vehicle mapping position in the vehicle motion coordinate system corresponding to the second feature points from the mapping positions included in the complete mapping information according to the complete mapping information corresponding to the second feature points; the coordinate origin of the vehicle motion coordinate system is at a preset path length from the reverse starting position corresponding to when the reverse assist module of the vehicle is turned on; obtain the transformation matrix between the first feature points and the second feature points; determine the vision positioning information of the vehicle in the vehicle motion coordinate system through the relative vehicle mapping position and the transformation matrix.

[0175] In one embodiment, the vision positioning module 640 is further configured to obtain the relative camera pose of the camera device when collecting the current image information relative to when collecting the vision mapping information according to the transformation matrix; obtain the external parameters of the camera device relative to the vehicle; obtain the relative vehicle pose of the vehicle at the current position relative to the mapping position according to the relative camera pose and the external parameters of the camera device relative to the vehicle; determine the vision positioning information of the vehicle in the vehicle motion coordinate system through the relative vehicle mapping position and the relative vehicle pose.

[0176] In one embodiment, the trajectory positioning module 630 is further configured to integrate the second inertial sensor data and the second odometer sensor data included in the second sensor information of the vehicle respectively, and determine a pose increment according to the integration results of the second inertial sensor data and the integration results of the second odometer sensor data; and accumulate the pose increment to the previous fusion result to obtain trajectory positioning information of the vehicle on the driving trajectory.

[0177] In one embodiment, the vision positioning module 640 further includes a coordinate conversion unit;

[0178] The coordinate conversion unit is configured to perform coordinate conversion on the vision positioning information in the vehicle motion coordinate system to obtain vision positioning information in the world coordinate system; the coordinate origin of the world coordinate system is the starting point of the driving trajectory.

[0179] In one embodiment, the reverse parking assistance device 600 further includes a control unit;

[0180] The control unit is configured to control the vehicle to turn on the reverse parking assistance module when a touch operation instruction, a voice instruction, or an activation instruction sent from a terminal device is detected.

[0181] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application.

[0182] As Figure 7 shown, the electronic device 700 may include:

[0183] A memory 710 storing executable program code;

[0184] A processor 720 coupled to the memory 710;

[0185] Wherein, the processor 720 calls the executable program code stored in the memory 710 and executes any one of the reverse parking assistance methods disclosed in the embodiments of the present application.

[0186] An embodiment of the present application discloses a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by the processor, the processor implements any one of the reverse parking assistance methods disclosed in the embodiments of the present application.

[0187] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the "in one embodiment" or "in an embodiment" that appears throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0188] In various embodiments of the present application, it should be understood that the magnitude of the serial numbers of the above processes does not necessarily mean the inevitable sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0189] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0191] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server or a network device, etc., specifically, the processor in the computer device) to execute some or all of the steps of the above methods in the various embodiments of the present application.

[0192] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disc memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0193] The above has introduced in detail a reverse assist method, device, electronic device and storage medium disclosed in the embodiments of the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A reverse assist method, characterized in that, The method includes: During the forward movement of the vehicle, determining the driving trajectory of the vehicle according to the first sensor information of the vehicle; the first sensor information includes the first inertial sensor data and the first odometer sensor data collected by the vehicle during the forward movement; Determining the mapping position of the vehicle on the driving trajectory, and obtaining the visual mapping information collected by the camera device of the vehicle when the vehicle is at the mapping position, so as to obtain the complete mapping information; When it is detected that the reverse assist module of the vehicle is turned on, obtaining the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle; the second sensor information includes the second inertial sensor data and the second odometer sensor data collected by the vehicle during the reverse movement; Collecting the current image information corresponding to the trajectory positioning information through the camera device, and determining the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information; Determining the true positioning information of the vehicle at the current position according to the trajectory positioning information and the visual positioning information; wherein, The visual mapping information in the complete mapping information includes image feature points; the determining the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information includes: extracting first feature points from the current image information; matching the first feature points with the image feature points included in the complete mapping information to determine second feature points that match the first feature points from the image feature points; determining the relative mapping position of the vehicle corresponding to the second feature points from the mapping positions included in the complete mapping information according to the complete mapping information corresponding to the second feature points; the relative mapping position of the vehicle is the positioning position in the vehicle motion coordinate system, and the coordinate origin of the vehicle motion coordinate system is at a preset path length from the reverse start position corresponding to when the reverse assist module of the vehicle is turned on; obtaining the transformation matrix between the first feature points and the second feature points; determining the visual positioning information of the vehicle in the vehicle motion coordinate system through the relative mapping position of the vehicle and the transformation matrix.

2. The method according to claim 1, wherein The determining the mapping position of the vehicle on the driving trajectory includes: Detecting the driving state of the vehicle; Determining the corresponding preset mapping interval according to the driving state of the vehicle; Selecting the mapping position of the vehicle on the driving trajectory from the driving trajectory according to the preset mapping interval.

3. The method according to claim 1, characterized in that, The determining the visual positioning information of the vehicle in the vehicle motion coordinate system through the relative mapping position of the vehicle and the transformation matrix includes: Obtaining the relative pose of the camera when collecting the current image information with respect to when collecting the visual mapping information according to the transformation matrix; Obtaining the external parameters of the camera device relative to the vehicle; Obtaining the relative pose of the vehicle at the current position with respect to the vehicle at the mapping position according to the relative pose of the camera and the external parameters of the camera device relative to the vehicle; Determine the visual positioning information of the vehicle in the vehicle motion coordinate system based on the relative position of vehicle map building and the relative pose of the vehicle.

4. The method according to claim 1, wherein Obtaining the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle includes: Integrate the second inertial sensor data and the second odometer sensor data included in the second sensor information of the vehicle respectively, and determine the pose increment according to the integration results of the second inertial sensor data and the integration results of the second odometer sensor data; Accumulate the pose increment to the previous fusion result to obtain the trajectory positioning information of the vehicle on the driving trajectory.

5. According to the method described in claim 1, after determining the visual positioning information of the vehicle in the vehicle motion coordinate system by the relative position of vehicle map building and the transformation matrix, the method further includes: Perform coordinate transformation on the visual positioning information in the vehicle motion coordinate system to obtain the visual positioning information in the world coordinate system; The coordinate origin of the world coordinate system is the starting point of the driving trajectory.

6. The method according to claim 1, characterized in that, Before detecting that the reverse assist module of the vehicle is turned on, the method further includes: When detecting a touch operation instruction, a voice instruction or an activation instruction sent by a terminal device, control the vehicle to turn on the reverse assist module.

7. A reverse assist device, characterized in that, Including: A trajectory mapping module, configured to determine the driving trajectory of the vehicle according to the first sensor information of the vehicle during the forward movement of the vehicle; The first sensor information includes the first inertial sensor data and the first odometer sensor data collected by the vehicle during the forward movement; A visual mapping module, configured to determine the mapping position of the vehicle on the driving trajectory, and obtain the visual mapping information collected by the camera device of the vehicle when the vehicle is at the mapping position, so as to obtain the complete mapping information; A trajectory positioning module, configured to obtain the trajectory positioning information of the vehicle on the driving trajectory according to the second sensor information of the vehicle when detecting that the reverse assist module of the vehicle is turned on; the second sensor information includes the second inertial sensor data and the second odometer sensor data collected by the vehicle during the reverse movement; A visual positioning module, configured to collect the current image information corresponding to the trajectory positioning information through the camera device, and determine the visual positioning information of the vehicle according to the feature matching relationship between the current image information and the visual mapping information; A fusion positioning module, configured to determine the true positioning information of the vehicle at the current position according to the trajectory positioning information and the visual positioning information; the visual mapping information in the complete mapping information includes image feature points, and the fusion positioning module is further configured to extract first feature points from the current image information; match the first feature points with the image feature points included in the complete mapping information to determine second feature points that match the first feature points from the image feature points; determine the relative vehicle mapping position corresponding to the second feature points from the mapping positions included in the complete mapping information according to the complete mapping information corresponding to the second feature points; the relative vehicle mapping position is the positioning position in the vehicle motion coordinate system, and the coordinate origin of the vehicle motion coordinate system is at a preset path length from the reverse starting position corresponding to when the vehicle's reverse assist module is turned on; obtain the transformation matrix between the first feature points and the second feature points; determine the visual positioning information of the vehicle in the vehicle motion coordinate system through the relative vehicle mapping position and the transformation matrix.

8. An electronic device, characterized in that, Comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is caused to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented.

Images