Automatic refueling vehicle guiding method

Abstract

The invention discloses an automatic refueling vehicle guiding method, and aims to solve the technical problem that an existing vehicle guiding system is poor in reliability and low in efficiency. According to the method, a single depth camera is used for identifying the type of the vehicle, accurate vehicle guiding and fuel tank cover plate pose calculation are achieved through real-time detection of the depth camera, the cost of vehicle guiding equipment is effectively reduced, meanwhile, the problems that traditional multi-camera deployment is tedious, reliability is poor, and repeated calculation exists are solved, and the vehicle guiding efficiency is improved. And the positioning time of the oil tank cover plate is shortened, and the refueling efficiency of the refueling robot can be improved. And wheel point clouds are detected and obtained in real time, so that the technical problem that point clouds cannot be generated by a traditional method due to vehicle paint reflection in an outdoor high-illumination environment is solved, and pose calculation of the fuel tank cover plate in the high-illumination environment is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of refueling robots, and more particularly to a vehicle guidance method for automatic refueling. Background Art

[0002] The vehicle guidance system is an important part of the refueling robot and one of the intelligent systems equipped in the smart gas station. It has functions such as intelligent detection, intelligent data collection, intelligent judgment, and intelligent indication and guidance. With the continuous development of refueling robot technology, the vehicle guidance system can effectively improve the safety and intelligence of the refueling robot and improve the refueling efficiency.

[0003] The vehicle guidance system currently generally uses visual sensors to achieve guidance and recognition. By identifying and spatially locating the wheels and fuel tank cover of the vehicle, the vehicle guidance and the rough positioning of the fuel tank cover are completed. In addition, the key technologies of vehicle guidance include intelligent detection, intelligent judgment, intelligent indication guidance, and intelligent positioning. Among them, one of the technical focuses of vehicle guidance is the judgment and posture calculation of the refueling vehicle.

[0004] At present, the vehicle guidance of gas stations mainly uses multiple camera lenses and multiple projection display devices at different shooting positions to perform real-time shooting and splicing to form a holographic image display. The display screen displays the relative position dynamic graphics between the vehicle and the parking frame in real time based on the vehicle position information obtained by the control system to guide the vehicle into the refueling parking area. In addition, the control system recognizes the vehicle license plate through the third visual recognition device and obtains the vehicle model corresponding to the vehicle license plate through the vehicle information system.

[0005] However, in the process of implementing the technical solutions in the embodiments of the present application, the inventors of the present application discovered that the above-mentioned technology has at least the following technical problems: the use of multiple camera lenses in different shooting positions and multiple display devices greatly increases the implementation and subsequent maintenance costs, and since multiple devices need to work together, the failure of one of the devices will affect the final imaging effect and calculation accuracy, and the failure cannot be automatically detected; in addition, after guiding the vehicle into the refueling area, it is necessary to rescan and calculate the position information of the fuel tank cover, which is relatively inefficient.

[0006] The information disclosed in this background technology section is only used to deepen the understanding of the background technology of the present disclosure, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art known to those skilled in the art. Summary of the invention

[0007] Through research, the inventors discovered that the reason why existing vehicle guidance systems use multiple cameras and display devices is that they are affected by outdoor lighting. Under high outdoor lighting, the reflection of the car paint will cause the existing vehicle guidance system to be unable to generate a point cloud, which will lead to errors in the recognition and judgment of the fuel tank cover and poor reliability. Based on this, an alternative approach is to set up multiple cameras to scan and identify from multiple angles to reduce the adverse effects of the reflection of the car paint.

[0008] In view of at least one of the above technical problems, the present disclosure provides a vehicle guidance method for automatic refueling, aiming to solve the technical problems of poor reliability and low efficiency of existing vehicle guidance systems.

[0009] According to one aspect of the present disclosure, a method for guiding a refueling robot vehicle in a high-light environment is provided, which comprises the following steps: (1) The depth camera installed at the corresponding position of the refueling island identifies the vehicle entering its field of view, obtains the vehicle model information and inspects the wheels and fuel tank cover; (2) According to the vehicle model information, the prior spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover corresponding to the vehicle model is obtained from the prior database; (3) setting a ROI area corresponding to the working area of ​​the refueling robot within the field of view of the depth camera, generating a corresponding indication signal based on the comparison between the wheel and fuel tank cover detection information and the ROI area and transmitting it to the indication screen to guide the vehicle into the working area of ​​the refueling robot; after the vehicle's wheels and fuel tank cover enter the ROI area at the same time and park, mapping the fuel tank cover and the rear wheel target on the side of the fuel tank cover to a three-dimensional point cloud, and calculating the spatial pose of the center of the rear wheel on the side of the fuel tank cover; (4) Based on the prior spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover side and the spatial posture of the center of the rear wheel on the fuel tank cover side, the spatial posture of the fuel tank cover is calculated.

[0010] In some embodiments of the present disclosure, in step (1), the depth camera is calibrated before operation, and hand-eye calibration is completed with the refueling arm of the refueling robot.

[0011] In some embodiments of the present disclosure, in step (1), a vehicle that appears within the field of view of the depth camera is detected by HRNet , YOLO , CenterNet Any of the network models can be used to identify the vehicle model and obtain vehicle model information carType .

[0012] In some embodiments of the present disclosure, in step (1), a vehicle that appears within the field of view of the depth camera is detected by MobileNet , SSD ,YOLO Any of the network models is used to detect wheels and fuel tank covers, and obtain the detection area corresponding to the wheels wheelRect And the detection area corresponding to the fuel tank cover capRect .

[0013] In some embodiments of the present disclosure, in step (2), the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover of each vehicle model are marked respectively, and the center of the rear wheel of the corresponding vehicle model is measured. X , Y , Z Axial distance to the center of the fuel tank cover X_Prior , Y_Prior , Z_Prior , and correspondingly establish the prior database.

[0014] In some embodiments of the present disclosure, step (3) includes the following sub-steps: (31) Calculate the wheel point cloud corresponding to the wheel area in the depth map based on the three-dimensional information obtained by the depth camera scanning PC_Wheel ; (32) Wheel point cloud PC_Wheel Perform plane fitting; (33) Calculating wheel coordinate values ​​including wheel direction vector and wheel center coordinates; (34) Compare the wheel coordinate values ​​with the set ROI area to determine and guide the vehicle into the working area of ​​the refueling robot.

[0015] In some embodiments of the present disclosure, step (4) includes the following sub-steps: (41) Get the center coordinates of the rear wheel on the fuel tank cover side ( X wheel , Y wheel , Z wheel ) and direction vector ; (42) Get the spatial distance from the center of the fuel tank cover to the center of the rear wheel on the fuel tank cover side X_Prior , Y_Prior , Z_ Prior ; (43) Calculate the position and posture of the fuel tank cover P cap : P cap_x = X wheel +X_Prior; P cap_y= Y wheel +Y_Prior; P cap_z = Z wheel +Z_Prior; P cap_normal = .

[0016] One or more technical solutions provided in the embodiments of the present application have at least any of the following technical effects or advantages: 1. A single depth camera is used to identify the vehicle model, and the real-time detection of the depth camera is used to accurately guide the vehicle and calculate the position of the fuel tank cover. While effectively reducing the cost of vehicle guidance equipment, it solves the problems of cumbersome deployment, poor reliability and repeated calculation of traditional multi-cameras, shortens the positioning time of the fuel tank cover, and helps to improve the refueling efficiency of the refueling robot.

[0017] 2. Real-time detection to obtain wheel point clouds solves the technical problem that traditional methods cannot generate point clouds due to paint reflections in outdoor high-light environments, and realizes the position calculation of the fuel tank cover in high-light environments.

[0018] 3. Automatic vehicle type recognition is introduced, and deep learning is used to identify and locate the vehicle type of the refueling vehicle, which has good robustness and intelligence; and while guiding the vehicle, the fuel tank cover can be roughly positioned, which helps to improve refueling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of depth camera guided recognition in one embodiment of the present application.

[0020] Figure 2 This is a schematic diagram of the vehicle guidance process in one embodiment of the present application.

[0021] Figure 3 This is a schematic diagram of the result of wheel detection performed by a depth camera in one embodiment of the present application.

[0022] Figure 4 This is the calculation result of the depth map corresponding to the point cloud in one embodiment of the present application.

[0023] Figure 5 This is the plane fitting result of the wheel point cloud in one embodiment of the present application.

[0024] Figure 6 Schematic diagram of the position of the wheel point cloud relative to the origin in one embodiment of the present application.

[0025] In the above figures, 1 is the refueling island, 2 is the refueling robot, 3 is the depth camera, and 4 is the vehicle to be refueled. α is the field of view of the depth camera. DETAILED DESCRIPTION

[0026] The programs involved or relied upon in the following embodiments are all conventional programs or simple programs in the technical field, and those skilled in the art can make conventional selections or adaptive adjustments according to specific application scenarios. In order to better understand the technical solution of this application, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0027] To solve the technical problems of the numerous, poor reliability and relatively low efficiency of traditional vehicle guidance equipment, see Figure 2 This example discloses a vehicle guidance method for automatic refueling, which specifically includes the following steps: (1) The depth camera installed at the corresponding position of the refueling island identifies the vehicle entering its field of view, obtains the vehicle model information, and inspects the wheels and fuel tank cover.

[0028] Since traditional guidance methods require the use of multiple cameras and display guidance devices, the equipment deployment is cumbersome and the failure rate is relatively high. If any of the devices fails, the vehicle cannot be reliably guided. In addition, in order to avoid the adverse effects of paint reflection on recognition in high-light outdoor environments, multiple devices are required to perform multi-angle scanning to compensate for the lack of information in the reflective area, but this greatly increases the amount of calculation.

[0029] In this embodiment, in order to solve the technical problems of poor reliability and low efficiency of multiple devices, a single depth camera is used to obtain the corresponding information of the vehicle. Figure 1 The depth camera is fixedly installed at the refueling island and located on one side of the refueling robot to ensure that the depth camera's field of view is unobstructed. In order to ensure the reliability of the information obtained by the depth camera and to be used for subsequent refueling posture calculations, the camera is calibrated before the depth camera works, and the hand-eye calibration between the depth camera and the refueling robot's refueling arm is completed. The depth map obtained by the depth camera is aligned with the environment map according to the following steps: (11) Set the point cloud coordinates ( X , Y , Z ), the point cloud coordinates of the RGB camera coordinate system are ( X RGB , Y RGB , Z RGB ), convert the point cloud in the left camera coordinate system to the point cloud in the RGB camera coordinate system: ; in, R RGB is the rotation matrix from the left camera to the RGB camera, T RGB is the translation matrix from the left camera to the RGB camera.

[0030] (12) Normalize the three-dimensional space: ; (13) Perform de-distortion processing: X distorted =x ( 1+k 1 r 2 +k 2 r 4 +k 3 r 6 ) +2p 1 xy + p 2 ( r 2 +2x 2 ); Y distorted =y ( 1+k 1 r 2 +k 2 r 4 +k 3 r 6 ) +2p 1 xy + p 2 ( r 2 +2y 2 ); in, r 2 =x 2 +y 2 , k 1, k 2, k 3 represents the radial distortion of the RGB camera.

[0031] (14) The position of the point on the image is obtained by projecting the intrinsic parameter matrix onto the pixel plane ( u, v ), so that the depth map and the environment map, i.e. the RGB map, are aligned: u = f x Xdistorted +c x ; v = f Y Y distorted +c y ; Among them, the optical center and focal length of the camera are ( c x , c y )and( f x , f y ).

[0032] During operation, the depth camera transmits images in real time. When a vehicle enters its field of view, its recognition mechanism is activated to identify the vehicle model and detect the wheels and fuel tank cover. In this embodiment, the pre-trained HRNet The network model identifies the car model that enters the field of view of the depth camera and obtains the vehicle model information carType In some other embodiments, a trained YOLO , CenterNet In addition, in this embodiment, the trained MobileNet The network model detects the rear wheels and fuel tank cover of the vehicle that enters the depth camera's field of view. Figure 3 , get the detection area corresponding to the wheel wheelRect And the detection area corresponding to the fuel tank cover capRect In some other embodiments, a trained SSD , YOLO Any of the network models can be used for wheel and fuel tank cover detection.

[0033] (2) According to the vehicle model information, the prior spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover corresponding to the vehicle model is obtained from the prior database.

[0034] The priori database stores the spatial distance data from the center of the fuel tank cover to the center of the rear wheel on the fuel tank cover corresponding to various vehicle models. This data assists the calculation of the fuel tank cover posture and improves the calculation efficiency. The priori database needs to be established in advance. Specifically, when establishing the database, vehicles of different categories are selected for actual measurement, and the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover of each vehicle are marked respectively, and then the rear wheel center is measured at X , Y , Z Axial distance to the center of the fuel tank coverX_Prior , Y_Prior , Z_Prior , and establish a priori database based on this data.

[0035] After the vehicle model entering the field of view of the depth camera is recognized, the prior spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover side that is suitable for the vehicle model is searched and extracted from the prior database.

[0036] (3) A ROI area corresponding to the working area of ​​the refueling robot is set within the field of view of the depth camera. Based on the comparison between the wheel and fuel tank cover detection information and the ROI area, the vehicle is guided into the working area of ​​the refueling robot through the large screen indication signal. After the vehicle's wheels and fuel tank cover enter the ROI area at the same time and park, the fuel tank cover and the rear wheel target on the fuel tank cover side are mapped to the three-dimensional point cloud, and the spatial pose of the center of the rear wheel on the fuel tank cover side is calculated.

[0037] After setting the ROI area corresponding to the working area of ​​the refueling robot, the depth camera scans and detects the vehicle entering its field of view, guides the vehicle into the designated area through monitoring feedback, and calculates the spatial posture of the rear wheel center on the fuel tank cover side at the same time. The specific sub-steps include the following: (31) Calculate the wheel point cloud corresponding to the wheel area in the depth map based on the three-dimensional information obtained by the depth camera scanning PC_Wheel.

[0038] Specifically, in step (1), the detection area corresponding to the wheel is obtained by the depth camera wheelRect, See also Figure 4 , and then according to wheelRect, Calculate the corresponding wheel point cloud in the depth map PC_Wheel. In this embodiment, it is assumed that the depth map ( u, v ), the depth value is depth_value , then the corresponding wheel point cloud is calculated as follows: PC_Wheel_x = ( u - c x )× depth_value ÷ 1000f ÷ f x ; PC_Wheel_y = ( v - c y )× depth_value ÷ 1000f ÷ f y ; PC_Wheel_z = depth_value ÷ 1000f .

[0039] (32) SeeFigure 5 , for the obtained wheel point cloud PC_Wheel In this embodiment, the plane fitting is performed. RANSAC Plane fitting method.

[0040] First, in the point cloud PC_Wheel Choose any three points in and calculate the corresponding plane equations: Ax + By + Cz = 0; Then calculate PC_Wheel The algebraic distance from all points (xi, yi, zi) to the plane equation is: d i = |Ax i + By i + Cz i | ; And select the threshold d th ,like d i <d th , then record the point as a sample point inside the model, otherwise record the point as a sample point outside the model; record the total number of internal points under the current parameters. Then repeat the above steps, calculate the plane equation and the algebraic distance from the point to the plane equation, select the best fitting parameters, that is, the model parameters corresponding to the plane with the largest number of internal points, as the final estimated wheel plane parameters, and the set of internal points under this parameter is the wheel plane point cloud. PC_Wheel , and the normal is .

[0041] (33) See Figure 6 , calculate the wheel coordinate values ​​including the wheel direction vector and the wheel center coordinates.

[0042] Let the wheel direction vector be ,definition Equal to the y-axis of the camera coordinate system, then calculate: ; ; Get mutually orthogonal wheel direction vectors ; And the wheel center area is p wheel , and calculate the center region coordinates: ; The center coordinates of the wheel are obtained from this ( x wheel , ywheel , z wheel ).

[0043] (34) Compare the wheel coordinate values ​​with the ROI area corresponding to the set refueling robot action area, and determine and guide the vehicle into the refueling robot working area.

[0044] (4) Based on the prior spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover side and the spatial posture of the center of the rear wheel on the fuel tank cover side, the spatial posture of the fuel tank cover is calculated.

[0045] In the step (1), the priori spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover side that is suitable for the vehicle model has been obtained from the priori database. The spatial posture of the fuel tank cover is roughly positioned by calculating the priori spatial distance and based on the spatial posture of the center of the rear wheel on the fuel tank cover side. Specifically, the following sub-steps are included: (41) Obtain the center coordinates of the rear wheel on the fuel tank cover side in step (3) ( X wheel , Y wheel , Z wheel ) and direction vector ; (42) Obtain the spatial distance from the center of the fuel tank cover to the center of the rear wheel on the fuel tank cover side in step (2) X_Prior , Y_ Prior , Z_Prior ; (43) Calculate the position and posture of the fuel tank cover P cap : P cap_x = X wheel +X_Prior; P cap_y = Y wheel +Y_Prior; P cap_z = Z wheel +Z_Prior; P cap_normal = .

[0046] In this way, the position information of the fuel tank cover of the refueling vehicle can be obtained while guiding the vehicle, and the rough positioning of the fuel tank cover can be achieved, so as to prepare for the subsequent fine positioning of the fuel tank cover, which helps to improve the refueling efficiency.

[0047] Although some preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0048] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the inventive concept. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A vehicle guidance method for automatic refueling, characterized in that: The steps include: (1) The depth camera located at the corresponding position of the refueling island identifies the vehicle entering its field of view, obtains the vehicle model information, and inspects the wheels and fuel tank cover; (2) According to the vehicle model information, the prior spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover corresponding to the vehicle model is obtained from the prior database; (3) setting a ROI area corresponding to the working area of ​​the refueling robot within the field of view of the depth camera, generating a corresponding indication signal based on the comparison between the wheel and fuel tank cover detection information and the ROI area and transmitting it to the indication screen to guide the vehicle into the working area of ​​the refueling robot; after the vehicle's wheels and fuel tank cover enter the ROI area at the same time and park, mapping the fuel tank cover and the rear wheel target on the side of the fuel tank cover to a three-dimensional point cloud, and calculating the spatial pose of the center of the rear wheel on the side of the fuel tank cover; (4) Based on the prior spatial distance between the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover side and the spatial posture of the center of the rear wheel on the fuel tank cover side, the spatial posture of the fuel tank cover is calculated.

2. The vehicle guiding method according to claim 1, characterized in that: In step (1), the depth camera is calibrated before operation, and the hand-eye calibration with the refueling arm of the refueling robot is completed.

3. The vehicle guiding method according to claim 1, characterized in that: In step (1), a vehicle that appears within the field of view of the depth camera is detected by HRNet , YOLO , CenterNet Any of the network models can be used to identify the vehicle model to obtain vehicle model information carType .

4. The vehicle guiding method according to claim 1, characterized in that: In step (1), a vehicle that appears within the field of view of the depth camera is detected by MobileNet , SSD , YOLO Any of the network models is used to detect wheels and fuel tank covers to obtain the detection area corresponding to the wheels. wheelRect And the detection area corresponding to the fuel tank cover capRect .

5. The vehicle guiding method according to claim 1, characterized in that: In step (2), the center of the fuel tank cover and the center of the rear wheel on the fuel tank cover of each vehicle model are marked respectively, and the center of the rear wheel of the corresponding vehicle model is measured at X , Y , Z Axial distance to the center of the fuel tank cover X_Prior , Y_Prior , Z_Prior , and correspondingly establish the prior database.

6. The vehicle guiding method according to claim 1, characterized in that: In the step (3), the following sub-steps are included: (31) Calculate the wheel point cloud corresponding to the wheel area in the depth map based on the three-dimensional information obtained by the depth camera scanning PC_Wheel ; (32) Wheel point cloud PC_Wheel Perform plane fitting; (33) Calculating wheel coordinate values ​​including wheel direction vector and wheel center coordinates; (34) Compare the wheel coordinate values ​​with the set ROI area to determine and guide the vehicle into the working area of ​​the refueling robot.

7. The vehicle guiding method according to claim 1, characterized in that: In the step (4), the following sub-steps are included: (41) Get the center coordinates of the rear wheel on the fuel tank cover side ( X wheel , Y wheel , Z wheel ) and direction vector ; (42) Get the spatial distance from the center of the fuel tank cover to the center of the rear wheel on the fuel tank cover side X_Prior , Y_Prior , Z_ Prior ; (43) The position and posture of the fuel tank cover are calculated based on the following formula: P cap : P cap_x = X wheel +X_Prior; P cap_y = Y wheel +Y_Prior; P cap_z = Z wheel +Z_Prior; P cap_normal = 。

Images