Training sample generation method and device and vehicle
By generating color maps and converting the coordinates of parking space corner points, and combining the coordinate transformation matrix of multiple cameras, the generation efficiency of training samples is improved, and the problem of low generation efficiency of training samples in the existing technology is solved, and the accuracy of parking space corner points detection is improved. and model generalization.
Patent Information
- Application Number
- CN202411189283.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2025-01-07
AI Technical Summary
The existing training sample generation method is less efficient in the autonomous parking domain, which affects the accuracy and efficiency of parking space detection.
By generating a color map based on the laser point cloud data, camera images, IMU data and RTK data in the specified scene, determine the coordinates of the parking space corner point, and convert them into the coordinates of the parking space corner point under the vehicle coordinate system, and then use multiple cameras The coordinate transformation matrix generates multiple training samples.
It improves the generation efficiency of training samples, enhances the accuracy of parking spot corner detection and model generalization, and reduces the cost of image data labeling.
Smart Images
Figure CN119274158A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle technology, and more specifically, to a training sample generation method, device and vehicle. Background Art
[0002] In the field of autonomous driving parking, parking space detection is a core functional module. In the current mainstream parking space detection methods, the mainstream method is to first detect the parking space corner points and then infer the parking space information, so parking space corner point detection plays an important role in parking space detection.
[0003] In a related method, parking space corner point detection can be achieved through AI (Artificial Intelligence) algorithm, but the AI algorithm requires a large number of training samples, and the existing training sample generation method still has the problem of low efficiency. Summary of the invention
[0004] In view of the above problems, the present application proposes a training sample generation method, device and vehicle to improve the above problems.
[0005] In a first aspect, the present application provides a method for generating training samples, the method comprising: generating a color map based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, the color map being a map showing ground layer information in the specified scene; determining a first parking space corner point coordinate, the first parking space corner point coordinate being the parking space corner point coordinate of the parking space in the color map; converting the first parking space corner point coordinate into a second parking space corner point coordinate, the second parking space corner point coordinate being the parking space corner point coordinate in a vehicle coordinate system; obtaining a plurality of training samples based on the second parking space corner point coordinate and respective coordinate transformation matrices of a plurality of cameras in the vehicle, each of the training samples comprising a third parking space corner point coordinate and an image from a corresponding camera perspective, the third parking space corner point coordinate being the parking space corner point coordinate in the image from a corresponding camera perspective.
[0006] In a second aspect, the present application provides a training sample generating device, the device comprising: a color map generating unit, for generating a color map based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, the color map being a map showing ground layer information in the specified scene; a first parking space corner point coordinate determining unit, for determining the first parking space corner point coordinates, the first parking space corner point coordinates being the parking space corner point coordinates of the parking space in the color map; a second parking space corner point coordinate acquiring unit, for converting the first parking space corner point coordinates into second parking space corner point coordinates, the second parking space corner point coordinates being the parking space corner point coordinates in the vehicle coordinate system; a training sample generating unit, for obtaining multiple training samples based on the second parking space corner point coordinates and respective coordinate transformation matrices of multiple cameras in the vehicle, each of the training samples comprising a third parking space corner point coordinates and an image under the corresponding camera viewing angle, the third parking space corner point coordinates being the parking space corner point coordinates in the image under the corresponding camera viewing angle.
[0007] In a third aspect, the present application provides a vehicle comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.
[0008] In a fourth aspect, the present application provides a computer-readable storage medium, in which program code is stored, wherein the above method is executed when the program code is run.
[0009] The present application provides a training sample generation method, device, vehicle and storage medium. After a color map is generated based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, the coordinates of the first parking corner point of the parking space in the color map are determined, and the first parking corner point coordinates are converted into the second parking corner point coordinates in the vehicle coordinate system. Based on the second parking corner point coordinates and the coordinate transformation matrix of each of the multiple cameras in the vehicle, multiple training samples are obtained. Through the above method, after the first parking corner point coordinates are determined from the generated color map, the first parking corner point coordinates are first converted into the second parking corner point coordinates, and then based on the second parking corner point coordinates and the coordinate transformation matrix of each of the multiple cameras in the vehicle, the parking corner point coordinates under the perspective of multiple cameras are obtained. That is, multiple third parking corner point coordinates can be generated through the same first parking corner point coordinates, thereby obtaining multiple training samples, thereby improving the generation efficiency of training samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0011] Figure 1 A flowchart of a training sample generation method proposed in an embodiment of the present application;
[0012] Figure 2 A schematic diagram of a vehicle proposed in the present application is shown;
[0013] Figure 3 A schematic diagram of a ground layer proposed in the present application is shown;
[0014] Figure 4 A schematic diagram of a color map proposed in the present application is shown;
[0015] Figure 5 A schematic diagram of a vehicle body coordinate system proposed in the present application is shown;
[0016] Figure 6 A schematic diagram of marking the coordinates of the corner points of a first parking space proposed in the present application is shown;
[0017] Figure 7 A flowchart of a training sample generation method proposed in another embodiment of the present application is shown;
[0018] Figure 8 A schematic diagram of a third parking space corner point coordinate marking proposed in the present application is shown;
[0019] Fig. 9 A schematic diagram of a parking space corner point detection proposed in the present application is shown;
[0020] Fig.10 A structural block diagram of a training sample generating device proposed in an embodiment of the present application is shown;
[0021] Fig.11 A structural block diagram of a vehicle proposed in this application is shown. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0023] In an embodiment of the present application, the inventor proposes a training sample generation method, device and vehicle. After a color map is generated based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, the coordinates of the first parking space corner point of the parking space in the color map are determined, and the first parking space corner point coordinates are converted into the second parking space corner point coordinates in the vehicle coordinate system. Based on the second parking space corner point coordinates and the coordinate transformation matrix of each of the multiple cameras in the vehicle, multiple training samples are obtained. Through the above method, after the first parking space corner point coordinates are determined from the generated color map, the first parking space corner point coordinates are first converted into the second parking space corner point coordinates, and then based on the second parking space corner point coordinates and the coordinate transformation matrix of each of the multiple cameras in the vehicle, the parking space corner point coordinates under the perspective of multiple cameras are obtained. That is to say, multiple third parking space corner point coordinates can be generated through the same first parking space corner point coordinates, thereby obtaining multiple training samples, thereby improving the generation efficiency of training samples.
[0024] See also Figure 1 , an embodiment of the present application provides a method for generating training samples, the method comprising:
[0025] S110: Generate a color map based on the laser point cloud data, camera images, IMU data and RTK data in the specified scene, where the color map is a map that displays ground layer information in the specified scene.
[0026] The designated scene may be a scene marked with parking spaces, such as an underground parking lot, a ground parking lot, and the like.
[0027] In the embodiments of the present application, Figure 2 As shown, the vehicle can be equipped with data acquisition devices, such as multiple radars (such as laser radar, millimeter wave radar, etc.), multiple cameras (such as fisheye cameras, depth cameras, etc.), IMU (Inertial Measurement Unit), RTK (Real Time Kinematic, real-time differential positioning) modules, etc., so that the vehicle can collect laser point cloud data, camera images, IMU data and RTK data in a specified scene in different poses based on the data acquisition devices carried. Among them, the camera image can include images collected by each camera when the vehicle is in different poses. The posture of the vehicle may include the posture of the vehicle body and the position of the vehicle. The posture of the vehicle is known information after the point cloud map is generated.
[0028] It should be noted that in order to implement the training sample generation method proposed in this application, the number of data acquisition devices carried by the vehicle needs to reach a specified number, which is the number required to meet the requirements of SLAM (Simultaneous Localization and Mapping) and coloring the point cloud map.
[0029] As a method, a point cloud map can be generated based on laser point cloud data, IMU data and RTK data through SLAM technology; a color map can be generated based on camera images and point cloud maps.
[0030] Optionally, the point cloud map may be highly intercepted to obtain a ground layer of the point cloud map; and a color map may be generated based on the camera image and the ground layer.
[0031] Optionally, the point cloud map has height information (Z-axis coordinate), which can be intercepted according to the preset height range to obtain Figure 3 The ground layer shown. The preset height range may be a height range corresponding to the ground layer obtained through multiple tests.
[0032] In an embodiment of the present application, the difficulty of coloring the ground layer can be reduced by highly intercepting the point cloud map.
[0033] Optionally, the pixel points in the camera image can be projected onto the ground layer by presetting the projection relationship, so as to obtain Figure 4 Color map shown.
[0034] The formula corresponding to the preset projection relationship may be:
[0035] X m =resolution*X p +offset_x
[0036] Y m =resolution*Y p +offset_y
[0037] Among them, resolution can represent the resolution of the color map, that is, the scale of a pixel in the color map in the world coordinate system; (offset_x, offset_y) can represent the offset of the color map, that is, the position offset of the origin of the color map from the origin of the world coordinate system when it is projected to the world coordinate system; resolution and (offset_x, offset_y) are known quantities after SLAM mapping. (X p , Y p) can represent the coordinates of the pixel points under the color map; (X m , Y m ) can represent the coordinates of the pixel points in the camera image in the world coordinate system.
[0038] Optionally, before executing step S110, all data acquisition devices in the vehicle may be calibrated, and the calibration may include data time synchronization, internal parameter calibration, and external parameter calibration. When calibrating the external parameter, the external parameters of all data acquisition devices may be calibrated to Figure 5 In the vehicle body coordinate system shown (the vehicle body coordinate system can take the center point of the vehicle's rear axle as the coordinate origin), the coordinates of the corresponding data acquisition device data in the world coordinate system are obtained through the internal parameters and external parameters corresponding to the data acquisition device, and the conversion relationship between the vehicle body coordinate system and the world coordinate system.
[0039] Optionally, in order to improve the accuracy of the projection, before projecting the pixels in the camera image to the ground layer, the camera image may be dedistorted, and then the pixels in the processed camera image may be projected to the ground layer.
[0040] S120: Determine the coordinates of a first parking space corner point, where the first parking space corner point coordinates are the coordinates of a parking space corner point in the color map.
[0041] Among them, Figure 6 As shown, there may be multiple coordinates of the first parking position corner point. In the SLAM mapping process, there may be multiple vehicle postures, which may include vehicle body posture and vehicle position. The vehicle posture is known information after the point cloud map is generated.
[0042] As a method, the color map can be uploaded to the cloud, and the staff in the cloud can mark the parking corner points in the color map to obtain the coordinates of the first parking corner point, and then the cloud can send the coordinates of the first parking corner point to the vehicle.
[0043] S130: Convert the first parking space corner point coordinates into second parking space corner point coordinates, where the second parking space corner point coordinates are the parking space corner point coordinates in a vehicle coordinate system.
[0044] As a method, in each posture, multiple first parking space corner point coordinates can be converted into multiple reference parking space corner point coordinates to obtain multiple sets of reference parking space corner point coordinates, each set of reference parking space corner point coordinates can include the parking space corner point coordinates in the world coordinate system corresponding to each of the multiple first parking space corner point coordinates when the vehicle is in the corresponding posture; the multiple sets of reference parking space corner point coordinates are converted into multiple sets of second parking space corner point coordinates, each set of second parking space corner point coordinates can include the parking space corner point coordinates in the vehicle coordinate system corresponding to each of the multiple first parking space corner point coordinates when the vehicle is in the corresponding posture.
[0045] Optionally, the offset and resolution of the color map can be obtained, where the offset can be the position offset of the coordinate origin of the color map from the coordinate origin of the world coordinate system when the coordinate origin of the color map is projected into the world coordinate system, and the resolution can be the scale of a pixel in the color map in the world coordinate system; in each posture, based on the offset and the resolution, the coordinates of multiple first parking space corner points are converted into the coordinates of multiple reference parking space corner points to obtain multiple sets of reference parking space corner point coordinates.
[0046] Among them, in each posture, the formula for converting the coordinates of each first parking space corner point into the coordinates of the corresponding reference parking space corner point can be:
[0047]
[0048] Among them, resolution can represent the resolution of the color map; (offset_x, offset_y) can represent the offset of the color map; resolution and (offset_x, offset_y) are known quantities after SLAM mapping. It can represent the coordinates of the corner point of the i-th parking space under the color map; It can represent the reference parking space corner point coordinates corresponding to the i-th first parking space corner point coordinates.
[0049] Optionally, multiple sets of reference parking space corner point coordinates may be converted into multiple sets of second parking space corner point coordinates through a coordinate conversion matrix between the vehicle coordinate system and the world coordinate system.
[0050] In an embodiment of the present application, the same first parking space corner point coordinate can be used to generate multiple second parking space corner point coordinates through multiple vehicle postures. Since there are multiple first parking space corner point coordinates in the color map, multiple second parking space corner point coordinates can be generated under each vehicle posture (the first parking space corner point coordinates correspond to the second parking space corner point coordinates one by one), so that multiple second parking space corner point coordinates under one vehicle posture can form a group of second parking space corner point coordinates, thereby expanding the number of training samples from the perspective of vehicle posture, and improving the generalization and detection accuracy of the model trained by the training samples under vehicle posture.
[0051] S140: Based on the coordinates of the second parking space corner point and the coordinate transformation matrices of each of the multiple cameras in the vehicle, multiple training samples are obtained, each of the training samples includes the coordinates of the third parking space corner point and an image under the corresponding camera viewing angle, and the coordinates of the third parking space corner point are the coordinates of the parking space corner point in the image under the corresponding camera viewing angle.
[0052] The coordinate transformation matrix may refer to a transformation matrix between the vehicle coordinate system and the camera coordinate system corresponding to each camera.
[0053] As one approach, multiple training samples may be obtained based on multiple sets of second parking space corner point coordinates and respective coordinate transformation matrices of multiple cameras.
[0054] The present embodiment provides a training sample generation method, after generating a color map based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, the first parking space corner point coordinates of the parking space in the color map are determined, the first parking space corner point coordinates are converted into the second parking space corner point coordinates in the vehicle coordinate system, and multiple training samples are obtained based on the second parking space corner point coordinates and the respective coordinate transformation matrices of multiple cameras in the vehicle. Through the above method, after determining the first parking space corner point coordinates from the generated color map, the first parking space corner point coordinates are first converted into the second parking space corner point coordinates, and then based on the second parking space corner point coordinates and the respective coordinate transformation matrices of multiple cameras in the vehicle, the parking space corner point coordinates under the perspectives of multiple cameras are obtained, that is, multiple third parking space corner point coordinates can be generated through the same first parking space corner point coordinates, thereby obtaining multiple training samples, thereby improving the generation efficiency of training samples.
[0055] See also Figure 7 , an embodiment of the present application provides a method for generating training samples, the method comprising:
[0056] S210: Generate a color map based on the laser point cloud data, camera images, IMU data and RTK data in the specified scene, where the color map is a map that displays ground layer information in the specified scene.
[0057] S220: Determine the coordinates of a first parking space corner point, where the coordinates of the first parking space corner point are the coordinates of a parking space corner point in the color map.
[0058] S230: Convert the first parking space corner point coordinates into second parking space corner point coordinates, where the second parking space corner point coordinates are the parking space corner point coordinates in the vehicle coordinate system.
[0059] S240: Based on the coordinate transformation matrix of each camera and the multiple groups of second parking space corner point coordinates, obtain multiple groups of third parking space corner point coordinates corresponding to each camera, each group of third parking space corner point coordinates includes the parking space corner point coordinates in the camera coordinate system corresponding to each of the multiple first parking space corner point coordinates when the vehicle is in the corresponding posture under the same camera.
[0060] As a method, multiple sets of second parking space corner point coordinates can be converted into multiple sets of projected parking space corner point coordinates based on the coordinate transformation matrix of each camera; the multiple sets of projected parking space corner point coordinates are corrected, and the corrected multiple sets of projected parking space corner point coordinates are used as multiple sets of third parking space corner point coordinates.
[0061] Wherein, under the coordinate transformation matrix of the same camera, the formula for converting each second parking space corner point coordinate in each set of second parking space corner point coordinates into the corresponding third parking space corner point coordinate can be:
[0062]
[0063] Among them, Tp_v m It can represent the coordinate transformation matrix of the mth camera; It can represent the jth second parking space corner point coordinate in the i-th group of second parking space corner point coordinates; It can represent the j-th third parking space corner point coordinates in the i-th group of third parking space corner point coordinates under the m-th camera viewing angle.
[0064] Optional, such as Figure 8 As shown in the figure, under the same camera perspective, due to external parameter calibration, vehicle positioning deviation and other reasons, if the coordinate system of the first parking space corner point coordinates is directly converted and projected into the image of the corresponding camera perspective, there will be corner point annotation deviation; and due to camera distortion, the first parking space corner point coordinates are converted into the second parking space corner point coordinates and then into the projected parking space corner point coordinates, so that the projected parking space corner point coordinates may not be at the position of the parking space corner point in the image. At this time, the projected parking space corner point coordinates can be corrected through post-processing to obtain the third parking space corner point coordinates. Among them, post-processing can use basic image processing technology plus logical constraints, including straight line extraction, straight line intersection detection, etc., and at the same time limit the pixel range to prevent the wrong intersection that is too far away from being considered as the parking space corner point.
[0065] S250: Based on the camera image and multiple groups of third parking space corner point coordinates corresponding to each camera, the multiple training samples are obtained, wherein the camera image includes images captured by each camera when the vehicle is in different postures.
[0066] Among them, there are multiple camera images, and each camera of the vehicle can collect images in multiple postures, so as to obtain images corresponding to different postures under the perspective of each camera.
[0067] As a method, a camera image under the same camera perspective and the same vehicle posture and a set of third parking space corner point coordinates can be used as a training sample, so that multiple training samples can be obtained.
[0068] As a method, after obtaining multiple training samples, the deformable_detr framework can be trained based on the multiple training samples to obtain a trained deformable_detr framework. The trained deformable_detr framework can be used for parking corner point detection. The reason for using the deformable_detr framework instead of the detr framework is that the parking corner points are smaller targets and may be deformed due to camera distortion, and the deformable_detr framework can effectively solve this problem. In addition, adapting the training samples of parking corner point detection to the deformable_detr algorithm framework can achieve true end-to-end detection.
[0069] For example, the image captured by the camera can be Fig. 9 As shown in the figure above, by inputting this image into Fig. 9 The trained deformable_detr framework shown in the middle figure is obtained Fig. 9 The figure below shows the parking space corner point detection results.
[0070] The training sample generation method provided in this embodiment can be such that after determining the first parking space corner point coordinates from the generated color map, the first parking space corner point coordinates are first converted into the second parking space corner point coordinates, and then based on the second parking space corner point coordinates and the coordinate transformation matrix of each of the multiple cameras in the vehicle, the parking space corner point coordinates under the perspectives of multiple cameras are obtained, that is, multiple third parking space corner point coordinates can be generated by the same first parking space corner point coordinates, thereby obtaining multiple training samples, thereby improving the generation efficiency of training samples. In addition, in this embodiment, by performing global annotation on a color map, and then projecting the vehicle into multiple local images (each local image corresponds to a camera perspective) when the vehicle is in different postures, more training samples can be generated by performing global annotation on only one color map (for example, the vehicle can have n postures, m cameras, and then m*n training samples can be generated.), thereby reducing the image data annotation cost.
[0071] See also Fig.10 The present application provides a training sample generating device 400, the device 400 comprising:
[0072] A color map generating unit 410 is used to generate a color map based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, wherein the color map is a map showing ground layer information in the specified scene;
[0073] A first parking space corner point coordinate determining unit 420, configured to determine first parking space corner point coordinates, wherein the first parking space corner point coordinates are the parking space corner point coordinates of the parking space in the color map;
[0074] A second parking space corner point coordinates acquisition unit 430 is used to convert the first parking space corner point coordinates into second parking space corner point coordinates, where the second parking space corner point coordinates are the parking space corner point coordinates in the vehicle coordinate system;
[0075] The training sample generation unit 440 is used to obtain multiple training samples based on the coordinates of the second parking space corner point and the coordinate transformation matrices of each of the multiple cameras in the vehicle, each of the training samples includes the coordinates of the third parking space corner point and the image under the corresponding camera viewing angle, and the coordinates of the third parking space corner point are the coordinates of the parking space corner point in the image under the corresponding camera viewing angle.
[0076] As a method, the color map generation unit 410 is specifically used to generate a point cloud map based on the laser point cloud data, IMU data and RTK data; and to generate the color map based on the camera image and the point cloud map.
[0077] Optionally, the color map generation unit 410 is specifically used to perform a height interception on the point cloud map to obtain a ground layer of the point cloud map; and generate the color map based on the camera image and the ground layer.
[0078] As a method, there are multiple first parking space corner point coordinates, and the vehicle in the color map has multiple postures. The second parking space corner point coordinate acquisition unit 430 is specifically used to convert the multiple first parking space corner point coordinates into multiple reference parking space corner point coordinates in each posture to obtain multiple groups of reference parking space corner point coordinates, each group of the reference parking space corner point coordinates includes the parking space corner point coordinates in the world coordinate system corresponding to the multiple first parking space corner point coordinates when the vehicle is in the corresponding posture; convert the multiple groups of reference parking space corner point coordinates into multiple groups of second parking space corner point coordinates, each group of the second parking space corner point coordinates includes the parking space corner point coordinates in the vehicle coordinate system corresponding to the multiple first parking space corner point coordinates when the vehicle is in the corresponding posture.
[0079] Optionally, the second parking space corner point coordinate acquisition unit 430 is specifically used to obtain the offset and resolution of the color map, the offset being the position offset of the coordinate origin of the color map from the coordinate origin of the world coordinate system when the coordinate origin of the color map is projected onto the world coordinate system, and the resolution being the scale of a pixel in the color map in the world coordinate system; in each posture, based on the offset and the resolution, the multiple first parking space corner point coordinates are converted into the multiple reference parking space corner point coordinates to obtain the multiple sets of reference parking space corner point coordinates.
[0080] As a method, the training sample generation unit 440 is specifically used to obtain multiple groups of third parking space corner point coordinates corresponding to each camera based on the coordinate transformation matrix of each camera and the multiple groups of second parking space corner point coordinates, each group of third parking space corner point coordinates includes the parking space corner point coordinates in the camera coordinate system corresponding to the multiple first parking space corner point coordinates when the vehicle is in the corresponding posture under the same camera; based on the camera image and the multiple groups of third parking space corner point coordinates corresponding to each camera, the multiple training samples are obtained, and the camera image includes the images captured by each camera when the vehicle is in different postures.
[0081] Optionally, the training sample generation unit 440 is specifically used to convert the multiple sets of second parking space corner point coordinates into multiple sets of projected parking space corner point coordinates based on the coordinate transformation matrix of each of the cameras; correct the multiple sets of projected parking space corner point coordinates, and use the corrected multiple sets of projected parking space corner point coordinates as the multiple sets of third parking space corner point coordinates.
[0082] The following will be combined Fig.11 A vehicle provided in the present application is described.
[0083] See also Fig.11 Based on the above training sample generation method and device, the embodiment of the present application also provides another vehicle 100 that can execute the above training sample generation method. The vehicle 100 includes a processor 102, a memory 104, and a data acquisition module 106. The memory 104 stores a program that can execute the content of the above embodiment, and the backup processor 102 can execute the program stored in the memory 104.
[0084] Among them, the processor 102 may include one or more processing cores. The processor 102 uses various interfaces and lines to connect various parts in the entire vehicle 100, and executes various functions and processes data of the vehicle 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 104, and calling data stored in the memory 104. Optionally, the processor 102 can be implemented in at least one hardware form of a network processor (Neural network Processing Unit, NPU), a digital signal processing (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 102 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a network processor (Neural network Processing Unit, NPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; the NPU is responsible for processing multimedia data such as video and images; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 102, but may be implemented separately through a communication chip.
[0085] The memory 104 may include a random access memory (RAM), and may also include a read-only memory (RAM) and a double data rate synchronous dynamic random access memory (DDR). The memory 104 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the vehicle 100 during use (such as a phone book, audio and video data, chat record data), etc.
[0086] The data acquisition module 106 may include but is not limited to: a camera, a laser radar, a millimeter wave radar, an inertial measurement unit (IMU), a level, a light sensor, a motion sensor, a pressure sensor, an infrared thermal sensor, a distance sensor, and other sensors. Among them, the camera can collect images of the environment around the vehicle 100, and the laser radar can collect laser point cloud data around the vehicle 100. The inertial measurement unit can be used to measure the three-axis attitude angle (or angular rate) and acceleration of the vehicle 100. The RTK data of the vehicle 100 can also be obtained through the above sensors.
[0087] Among them, the pressure sensor may be a sensor that detects pressure generated by pressing on the vehicle 100. That is, the pressure sensor detects pressure generated by contact or pressing between the user and the vehicle 100, such as pressure generated by contact or pressing between the user's hand and the vehicle 100. Therefore, the pressure sensor can be used to determine whether contact or pressing occurs between the user and the vehicle 100, and the magnitude of the pressure.
[0088] In addition, the vehicle 100 may also be equipped with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, etc., which will not be described in detail here.
[0089] An embodiment of the present application provides a computer-readable storage medium in which a program code is stored, and the program code can be called by a processor to execute the method described in the above method embodiment.
[0090] The computer readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium includes a non-transitory computer-readable storage medium. The computer readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.
[0091] In summary, the present application provides a training sample generation method, device, and vehicle. After generating a color map based on laser point cloud data, camera images, IMU data, and RTK data in a specified scene, the coordinates of the first parking space corner point of the parking space in the color map are determined, and the first parking space corner point coordinates are converted into the second parking space corner point coordinates in the vehicle coordinate system. Based on the second parking space corner point coordinates and the respective coordinate transformation matrices of multiple cameras in the vehicle, multiple training samples are obtained. Through the above-mentioned method, after determining the first parking space corner point coordinates from the generated color map, the first parking space corner point coordinates are first converted into the second parking space corner point coordinates, and then based on the second parking space corner point coordinates and the respective coordinate transformation matrices of multiple cameras in the vehicle, the parking space corner point coordinates under the perspectives of multiple cameras are obtained. In other words, multiple third parking space corner point coordinates can be generated through the same first parking space corner point coordinates, thereby obtaining multiple training samples, thereby improving the generation efficiency of training samples.
[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A training sample generation method, characterized in that: The method comprises: Generate a color map based on the laser point cloud data, camera image, IMU data and RTK data in the specified scene, wherein the color map is a map showing the ground layer information in the specified scene; Determine the coordinates of a first parking space corner point, where the coordinates of the first parking space corner point are the coordinates of a parking space corner point in the color map; Converting the first parking space corner point coordinates into the second parking space corner point coordinates, where the second parking space corner point coordinates are the parking space corner point coordinates in the vehicle coordinate system; Based on the coordinates of the second parking space corner point and the coordinate transformation matrices of each of the multiple cameras in the vehicle, multiple training samples are obtained, each of the training samples includes the coordinates of the third parking space corner point and the image under the corresponding camera viewing angle, and the coordinates of the third parking space corner point are the coordinates of the parking space corner point in the image under the corresponding camera viewing angle.
2. The method according to claim 1, characterized in that There are multiple first parking space corner point coordinates, and the vehicle in the color map has multiple postures. The step of converting the first parking space corner point coordinates into the second parking space corner point coordinates includes: In each posture, the multiple first parking space corner point coordinates are converted into multiple reference parking space corner point coordinates to obtain multiple groups of reference parking space corner point coordinates, each group of reference parking space corner point coordinates includes the parking space corner point coordinates in the world coordinate system corresponding to each of the multiple first parking space corner point coordinates when the vehicle is in the corresponding posture; The multiple groups of reference parking space corner point coordinates are converted into multiple groups of second parking space corner point coordinates, each group of the second parking space corner point coordinates includes parking space corner point coordinates in the vehicle coordinate system corresponding to each of the multiple first parking space corner point coordinates when the vehicle is in a corresponding posture.
3. The method according to claim 2, characterized in that The method of converting the plurality of first parking space corner point coordinates into a plurality of reference parking space corner point coordinates in each position and posture to obtain a plurality of sets of reference parking space corner point coordinates includes: Obtaining the offset and resolution of the color map, wherein the offset is the position offset of the coordinate origin of the color map from the coordinate origin of the world coordinate system when the coordinate origin of the color map is projected into the world coordinate system, and the resolution is the scale of a pixel in the color map in the world coordinate system; In each posture, based on the offset and the resolution, the multiple first parking space corner point coordinates are converted into the multiple reference parking space corner point coordinates to obtain the multiple groups of reference parking space corner point coordinates.
4. The method according to claim 2, characterized in that: The obtaining of multiple training samples based on the coordinates of the second parking space corner point and the respective coordinate transformation matrices of multiple cameras in the vehicle includes: Based on the coordinate transformation matrix of each camera and the multiple groups of second parking space corner point coordinates, multiple groups of third parking space corner point coordinates corresponding to each camera are obtained, each group of the multiple groups of third parking space corner point coordinates includes the parking space corner point coordinates in the camera coordinate system corresponding to each of the multiple first parking space corner point coordinates when the vehicle is in a corresponding posture under the same camera; The multiple training samples are obtained based on the camera images and multiple groups of third parking space corner point coordinates corresponding to each of the cameras, wherein the camera images include images collected by each of the cameras when the vehicle is in different postures.
5. The method according to claim 4, characterized in that The step of obtaining a plurality of sets of third parking space corner point coordinates corresponding to each camera based on the coordinate transformation matrix of each camera and the plurality of sets of second parking space corner point coordinates comprises: Based on the coordinate transformation matrix of each camera, the multiple sets of second parking space corner point coordinates are converted into multiple sets of projection parking space corner point coordinates; The multiple groups of projected parking space corner point coordinates are corrected, and the corrected multiple groups of projected parking space corner point coordinates are used as the multiple groups of third parking space corner point coordinates.
6. The method according to claim 1, characterized in that The method generates a color map based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, including: Generate a point cloud map based on the laser point cloud data, IMU data and RTK data; The color map is generated based on the camera image and the point cloud map.
7. The method according to claim 6, characterized in that The step of generating the color map based on the camera image and the point cloud map comprises: Performing height interception on the point cloud map to obtain a ground layer of the point cloud map; The color map is generated based on the camera image and the ground layer.
8. A training sample generating device, characterized in that: The device comprises: A color map generation unit, used to generate a color map based on laser point cloud data, camera images, IMU data and RTK data in a specified scene, wherein the color map is a map showing ground layer information in the specified scene; A first parking space corner point coordinate determining unit, used to determine the first parking space corner point coordinates, wherein the first parking space corner point coordinates are the parking space corner point coordinates of the parking space in the color map; A second parking space corner point coordinate acquisition unit, used for converting the first parking space corner point coordinate into a second parking space corner point coordinate, where the second parking space corner point coordinate is the parking space corner point coordinate in a vehicle coordinate system; A training sample generating unit is used to obtain a plurality of training samples based on the coordinates of the second parking space corner point and the coordinate transformation matrices of each of the plurality of cameras in the vehicle, each of the training samples comprising the coordinates of the third parking space corner point and an image under the corresponding camera viewing angle, wherein the coordinates of the third parking space corner point are the coordinates of the parking space corner point in the image under the corresponding camera viewing angle.
9. A vehicle, characterized in that: including one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is run, the method according to any one of claims 1 to 7 is executed.