Vehicle scratch identification method and apparatus, based on binocular stereo vision
Through binocular stereoscopic vision sensors to obtain data in a narrow space in real time and perform three-dimensional matching to identify and avoid the risk of scratches, the problem of vehicle scratches in a narrow turning space is solved, and driving safety is improved.
Patent Information
- Application Number
- PCT/CN2024/088437
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-04-17
- Publication Date
- 2024-12-05
AI Technical Summary
The existing active anti-collision mechanism is difficult to accurately handle in narrow turning spaces such as narrow spiral channels, resulting in easy scratches in vehicles.
The binocular stereoscopic vision sensor is used to obtain the initial image data and vehicle working condition data in real time, and the point cloud data is generated through three-dimensional stereoscopic matching, and the scratch risk information is determined based on the vehicle's forward driving trajectory data, and the vehicle is actively controlled to avoid scratches.
By accurately reconstructing three-dimensional point cloud data in narrow turning spaces, effectively identify and avoid scratch risks, and improve the safety of vehicles in narrow spaces.
Smart Images

Figure CN2024088437_05122024_PF_FP_ABST
Abstract
Description
Vehicle scratch recognition method and device based on binocular stereo vision
[0001] This application claims priority to a Chinese patent application filed with the Patent Office of China on June 1, 2023, with application number 202310645723.6 and invention name “A method and device for identifying vehicle scratches based on binocular stereo vision”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of vehicle assisted driving technology, and in particular to a vehicle scratch recognition method based on binocular stereo vision, a vehicle scratch recognition device based on binocular stereo vision, a vehicle, and a storage medium. Background Art
[0003] For example, in multi-story underground and above-ground parking garages in shopping malls and office buildings, vehicles change levels through narrow spiral tunnels. Due to the narrowness of these tunnels and the large size of many vehicles, such as large SUVs (sport utility vehicles) and full-size MPVs (multi-purpose vehicles), vehicles are prone to scratches when maneuvering through these tight turning spaces.
[0004] Existing active collision avoidance mechanisms or path planning mechanisms often rely on GPS (Global Positioning System), RTK (Real-time kinematic) carrier phase differential technology or millimeter-wave radar. However, they are unable to accurately handle narrow turning spaces such as the aforementioned narrow indoor spiral passages, resulting in vehicles still being prone to scratches in narrow turning spaces.
[0005] Summary of the Invention
[0006] In view of the above problems, embodiments of the present application are proposed to provide a vehicle scratch recognition method based on binocular stereo vision, a vehicle scratch recognition device based on binocular stereo vision, a vehicle and a storage medium that overcome the above problems or at least partially solve the above problems.
[0007] In some embodiments of the present application, a vehicle scratch recognition method based on binocular stereo vision is disclosed. A binocular stereo vision sensor is installed on a vehicle. The method includes:
[0008] When the vehicle reaches a narrow turning space, it acquires the initial image data and vehicle operating condition data collected by the binocular stereo vision sensor in real time;
[0009] Perform three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data;
[0010] Generate vehicle forward driving trajectory data based on 3D point cloud data and vehicle operating condition data;
[0011] Determine scratch risk information based on the vehicle's forward driving trajectory data and three-dimensional point cloud data.
[0012] Optionally, before performing three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data, the method further includes:
[0013] The camera coordinate system corresponding to the initial image data is mapped to the vehicle coordinate system of the vehicle.
[0014] Optionally, the three-dimensional point cloud data includes a time-continuous point cloud; and the step of generating vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data includes:
[0015] Generate a first vehicle displacement vector and a first rotation matrix based on the point cloud and vehicle operating condition data at a previous moment;
[0016] Generate a second vehicle displacement vector and a second rotation matrix based on the current point cloud and vehicle operating condition data;
[0017] Vehicle forward driving trajectory data is generated according to the first vehicle displacement vector, the first rotation matrix, the second vehicle displacement vector, and the second rotation matrix.
[0018] Optionally, the method further comprises:
[0019] Get vehicle size information;
[0020] The vehicle body size information is combined with the vehicle forward driving trajectory data to correct the vehicle forward driving trajectory data.
[0021] Optionally, the step of determining the scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data includes:
[0022] When the trajectory of the vehicle's forward driving trajectory data overlaps with the point cloud at the next moment, the scratch risk information is determined to indicate that there is a scratch risk;
[0023] When the trajectory of the vehicle's forward driving trajectory data and the point cloud at the next moment do not overlap, the scratch risk information is determined to be that there is no scratch risk.
[0024] Optionally, the method further comprises:
[0025] Read the wall and ground point clouds from the 3D point cloud data;
[0026] The road curvature is calculated based on the position of the wall ground point cloud;
[0027] The point cloud at the next moment is fitted according to the road curvature to generate a virtual point cloud. The virtual point cloud is the subsequent point cloud of the point cloud at the next moment in time. The virtual point cloud is used to increase the point cloud at the next moment.
[0028] Optionally, the method further comprises:
[0029] When the scratch risk information indicates that there is a scratch risk, the operation of the vehicle is controlled.
[0030] In some embodiments of the present application, a vehicle scratch recognition device based on binocular stereo vision is disclosed. The binocular stereo vision sensor is installed on the vehicle. The device includes:
[0031] The first acquisition module is used to acquire the initial image data and vehicle operating condition data collected by the binocular stereo vision sensor in real time when the vehicle reaches a narrow turning space;
[0032] A 3D matching module is used to perform 3D stereo matching on the initial image data to generate 3D point cloud data;
[0033] A trajectory determination module is used to generate vehicle forward driving trajectory data based on three-dimensional point cloud data and vehicle operating condition data;
[0034] The scratch recognition module is used to determine scratch risk information based on the vehicle's forward driving trajectory data and three-dimensional point cloud data.
[0035] In some embodiments of the present application, the embodiments of the present application also disclose a vehicle, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the above-mentioned vehicle scratch recognition method based on binocular stereo vision are implemented.
[0036] In some embodiments of the present application, the embodiments of the present application further disclose a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned vehicle scratch recognition method based on binocular stereo vision are implemented.
[0037] The embodiments of the present application include the following advantages:
[0038] The embodiments of the present application acquire initial image data and vehicle operating condition data collected by the binocular stereo vision sensor in real time when the vehicle reaches a narrow turning space; perform three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data; generate vehicle forward driving trajectory data based on the three-dimensional point cloud data and vehicle operating condition data; and determine scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data. The narrow turning space is accurately reconstructed using the binocular stereo vision sensor's point cloud data, and the risk of scratches is determined based on the binocular stereo vision sensor's point cloud data and vehicle operating condition data. The system proactively intervenes to prevent the vehicle from scratching, thereby improving vehicle driving safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] FIG1 is a flowchart of an embodiment of a vehicle scratch recognition method based on binocular stereo vision of the present application;
[0040] FIG2 is a flowchart of an embodiment of a vehicle scratch recognition method based on binocular stereo vision of the present application;
[0041] FIG3 is a schematic diagram of point cloud distribution of an example of a vehicle scratch recognition method based on binocular stereo vision according to the present application;
[0042] FIG4 is a second schematic diagram of point cloud distribution of an example of a vehicle scratch recognition method based on binocular stereo vision of the present application;
[0043] FIG5 is a third schematic diagram of point cloud distribution of an example of a vehicle scratch recognition method based on binocular stereo vision of the present application;
[0044] FIG6 is a structural block diagram of an embodiment of a vehicle scratch recognition device based on binocular stereo vision of the present application;
[0045] FIG7 is a structural block diagram of an electronic device provided in an embodiment of the present application;
[0046] FIG8 is a structural block diagram of a storage medium provided in an embodiment of the present application. Specific embodiments
[0047] Referring to Figure 1, a flowchart illustrating the steps of an embodiment of a vehicle scratch detection method based on binocular stereo vision is shown. In this embodiment, a binocular stereo vision sensor is installed on a vehicle. The specific installation location can be determined based on the actual vehicle conditions and is not limited in this embodiment. For example, the binocular stereo vision sensor is installed on the top inside of the vehicle's front windshield.
[0048] The vehicle scratch recognition method based on binocular stereo vision may specifically include the following steps:
[0049] Step 101 : When a vehicle reaches a narrow turning space, initial image data and vehicle operating condition data collected by a binocular stereo vision sensor are acquired in real time.
[0050] In an embodiment of the present application, the binocular stereo vision sensor continuously collects initial image data while the vehicle is traveling; when the vehicle reaches a narrow turning space, the initial image data collected by the binocular stereo vision sensor and the vehicle operating condition data of the vehicle are obtained in real time.
[0051] Among them, a narrow turning space refers to a road space scenario where the width of the vehicle's driving space is close to the width of the vehicle and is in a turning state, such as a narrow spiral channel. The embodiment of the present application does not limit the specific implementation of the narrow turning space.
[0052] Step 102: Perform three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data.
[0053] After obtaining the initial image data collected by the binocular stereo vision sensor, the initial image data can be subjected to three-dimensional stereo matching to perform three-dimensional reconstruction of the narrow turning space and obtain dense three-dimensional point cloud data. Among them, since there are many and mature three-dimensional stereo matching methods, such as semi-global matching method, deep learning method, etc. In the embodiment of the present application, there is no specific limitation on the method for three-dimensional matching of binocular images. Those skilled in the art can select an appropriate point cloud three-dimensional matching method according to actual conditions, perform three-dimensional stereo matching on the initial image data collected by the binocular stereo vision sensor, and obtain three-dimensional point cloud data. Furthermore, according to information such as the installation position of the camera in the vehicle, the point cloud data corresponding to the initial image data can be mapped to the vehicle coordinate system to obtain the three-dimensional point cloud data of the vehicle coordinate system.
[0054] Step 103 : Generate vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data.
[0055] The three-dimensional point cloud data can be used to obtain accurate reconstructed three-dimensional spatial information of the current narrow turning space. In this reconstructed three-dimensional space, combined with the vehicle operating condition data, the vehicle can be fitted to determine the vehicle's motion trajectory in the reconstructed three-dimensional space, that is, the forward driving path of the vehicle in the current narrow turning space can be determined, that is, the vehicle's forward driving trajectory data can be generated.
[0056] Step 104: Determine the scratch risk information based on the vehicle's forward driving trajectory data and the three-dimensional point cloud data.
[0057] The 3D point cloud data is used to accurately reconstruct the 3D space of the current narrow turning space. The vehicle's forward trajectory data can be used to determine the vehicle's driving path. Therefore, the forward trajectory data and the 3D point cloud data can be combined to determine the vehicle's driving path in the reconstructed 3D space information. This determines whether the vehicle's driving path overlaps with the reconstructed 3D space, indicating whether there is a scratch, and determines the scratch risk information. The scratch risk information shows whether the vehicle's current operating condition data indicates that a scratch would occur in the current narrow turning space.
[0058] The embodiment of the present application acquires initial image data and vehicle operating condition data collected by the binocular stereo vision sensor in real time when the vehicle reaches a narrow turning space; performs three-dimensional stereo matching on the initial image data and maps it to the vehicle coordinate system to generate three-dimensional point cloud data; generates vehicle forward driving trajectory data based on the three-dimensional point cloud data and vehicle operating condition data; and determines scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data. The narrow turning space is accurately reconstructed using the binocular stereo vision sensor's point cloud data, and the risk of scratches is determined based on the binocular stereo vision sensor's point cloud data and vehicle operating condition data. The system proactively intervenes to prevent the vehicle from scratching, thereby improving vehicle driving safety.
[0059] Referring to Figure 2, a flowchart illustrating another embodiment of a vehicle scratch detection method based on binocular stereo vision is shown. In this embodiment, a binocular stereo vision sensor is installed on a vehicle. The vehicle scratch detection method based on binocular stereo vision may specifically include the following steps:
[0060] Step 201 , when the vehicle reaches a narrow turning space, initial image data and vehicle operating condition data collected by a binocular stereo vision sensor are acquired in real time;
[0061] When the vehicle reaches a narrow turning space, initial image data captured by the binocular stereo vision sensor and vehicle operating condition data are acquired in real time. The vehicle operating condition data includes at least vehicle speed, steering wheel angle information, and vehicle posture information. Those skilled in the art may also add other vehicle operating condition information for auxiliary calculations based on actual circumstances, which is not limited in this embodiment of the present application. The initial image data may include image data of the walls and road surface of the narrow turning space.
[0062] Step 202 : Mapping the camera coordinate system corresponding to the initial image data into the vehicle coordinate system of the vehicle.
[0063] In an embodiment of the present application, the initial image data is collected in the camera coordinate system of the binocular stereo vision sensor. In order to be able to process it in the same coordinate system, the camera coordinate system corresponding to the initial image data can be mapped to the vehicle coordinate system of the vehicle.
[0064] Specifically, an offset matrix between the coordinate system of the binocular stereo vision sensor and the vehicle coordinate system may be calculated, and the coordinate system of the binocular stereo vision sensor may be mapped to the vehicle coordinate system based on the offset matrix.
[0065] Step 203: Perform three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data;
[0066] After mapping the camera coordinate system to the vehicle coordinate system, the initial image data collected by the binocular stereo vision sensor is subjected to three-dimensional stereo matching and mapped to the vehicle coordinate system accordingly, based on the three-dimensional point cloud data of the vehicle coordinate system.
[0067] Step 204: generating vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data;
[0068] The forward trajectory of the vehicle in a narrow turning space can be predicted and calculated based on the 3D point cloud data and the vehicle operating condition data to generate vehicle forward trajectory data. The vehicle forward trajectory data can be wheel tracks.
[0069] In an optional embodiment of the present application, the three-dimensional point cloud data includes a time-continuous point cloud; and the step of generating vehicle forward driving trajectory data based on the three-dimensional point cloud data and vehicle operating condition data includes:
[0070] Sub-step S2041 , generating a first vehicle displacement vector and a first rotation matrix based on the point cloud and vehicle operating condition data at the previous moment;
[0071] In an embodiment of the present application, a first vehicle displacement vector and a first rotation matrix of the vehicle can be calculated using point cloud data measured at a previous moment and vehicle operating condition data, such as vehicle speed, steering wheel angle information, and vehicle body posture information. The first vehicle displacement vector and the first rotation matrix of the vehicle are the vehicle displacement vector and the rotation matrix obtained by the vehicle based on the point cloud at a previous moment. The first and second values are used to distinguish the vehicle displacement vector and the rotation matrix calculated at the previous moment from the vehicle displacement vector and the rotation matrix calculated at the current moment.
[0072] Sub-step S2042 , generating a second vehicle displacement vector and a second rotation matrix based on the current point cloud and vehicle operating condition data;
[0073] Based on the same calculation, the point cloud at the current moment and the vehicle operating condition data at the current moment are used to calculate the second vehicle displacement vector and the second rotation matrix corresponding to the current moment.
[0074] Sub-step S2043 , generating vehicle forward driving trajectory data according to the first vehicle displacement vector, the first rotation matrix, the second vehicle displacement vector, and the second rotation matrix.
[0075] Then, based on the first vehicle displacement vector, the first rotation matrix, the second vehicle displacement vector and the second rotation matrix at the previous moment and the current moment, the vehicle's driving trajectory is calculated to generate vehicle forward driving trajectory data.
[0076] Step 205: Obtain vehicle body size information.
[0077] Since the forward driving trajectory data is based on the trajectory of the wheels, in order to accurately identify scratches, the vehicle body size information can be obtained and integrated to calculate the trajectory.
[0078] Step 206 : Combining the vehicle body size information with the vehicle forward driving trajectory data to correct the vehicle forward driving trajectory data.
[0079] In an embodiment of the present application, the calculated vehicle forward driving trajectory data is corrected in combination with the vehicle body size information to obtain more accurate vehicle forward driving trajectory data, and the vehicle scratch risk is predicted based on the corrected vehicle forward driving trajectory data.
[0080] Step 207: determining scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data;
[0081] After obtaining the vehicle's forward driving trajectory data and three-dimensional point cloud data, based on the relationship between the vehicle's forward driving trajectory data and the three-dimensional point cloud data in the reconstructed three-dimensional space, it is determined whether the vehicle will be at risk of being scratched, and the scratch risk information is determined.
[0082] In an optional embodiment of the present application, the step of determining the scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data includes:
[0083] Sub-step S2071: when the trajectory of the vehicle's forward driving trajectory data overlaps with the point cloud at the next moment, determining that the scratch risk information indicates that a scratch risk exists;
[0084] Specifically, if the vehicle's forward trajectory data overlaps with the point cloud at the next moment, that is, at the next moment, the vehicle, traveling with the current vehicle operating condition data, will collide with the location it will be traveling to at the next moment, i.e., a scratch will occur. In this case, the existence of a scratch risk can be determined as scratch risk information to alert the driver to the existence of the scratch risk.
[0085] Sub-step S2072: When the trajectory of the vehicle's forward driving trajectory data and the point cloud at the next moment do not overlap, it is determined that the scratch risk information is that there is no scratch risk.
[0086] When the trajectory of the vehicle's forward driving trajectory data and the point cloud at the next moment do not overlap, that is, at the next moment, the vehicle travels with the current vehicle operating condition data and will not collide with the place it will travel to at the next moment, at this time, the absence of scratch risk can be determined as scratch risk information, that is, the scratch risk information can be empty.
[0087] In order to increase the response time and predict the risk of scratches earlier, in an optional embodiment of the present application, the method further includes the following steps:
[0088] Step S1, reading the wall and ground point cloud from the three-dimensional point cloud data;
[0089] The point clouds corresponding to the wall and the ground can be read from the three-dimensional point cloud data, that is, the wall and ground point clouds.
[0090] Step S2, calculating the road curvature based on the position of the wall ground point cloud;
[0091] Since the wall and the ground do not change suddenly when turning, the curvature of the road can be calculated based on the position of the wall and ground point clouds to obtain the specific road curvature.
[0092] Step S3, fitting the point cloud at the next moment according to the road curvature to generate a virtual point cloud. The virtual point cloud is the subsequent point cloud of the point cloud at the next moment in time, and the virtual point cloud is used to increase the point cloud at the next moment.
[0093] The next point cloud is then fitted based on the road curvature, predicting a point cloud ahead of the current detection range in the direction of travel. This is called a virtual point cloud, which is the subsequent point cloud in time to the next point cloud. By adding the next point cloud to the virtual point cloud, the predicted forward range is increased.
[0094] Step 208: When the scratch risk information indicates that there is a scratch risk, control the operation of the vehicle.
[0095] When the scratch risk information indicates that there is a scratch risk, the vehicle's speed, steering angle and other operating data can be modified, and the vehicle's operation control can be intervened to assist the driver in passing through a narrow turning space.
[0096] The embodiment of the present application acquires initial image data and vehicle operating condition data collected by a binocular stereo vision sensor in real time when the vehicle reaches a narrow turning space; maps the initial image data into the vehicle coordinate system to generate three-dimensional point cloud data of the vehicle coordinate system; generates vehicle forward driving trajectory data based on the three-dimensional point cloud data and vehicle operating condition data; determines scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data; and controls vehicle operation when the scratch risk information indicates a scratch risk exists. The narrow turning space is accurately reconstructed using the point cloud data from the binocular stereo vision sensor, the scratch risk is determined based on the point cloud data from the binocular stereo vision sensor and the vehicle operating condition data, and proactively intervenes to avoid a scratch, thereby improving vehicle driving safety. Furthermore, by predicting uncollected point clouds, the scratch prediction space is extended, allowing for earlier prediction of vehicle scratch risks, increasing response time, and further improving vehicle driving safety.
[0097] In order to enable those skilled in the art to better understand the embodiments of the present application, the embodiments of the present application are described below using an example:
[0098] 3 , there is shown a first schematic diagram of point cloud distribution of an example of a vehicle scratch recognition method based on binocular stereo vision according to the present application;
[0099] 1. The binocular stereo vision sensor performs real-time stereo imaging of the front, acquiring dense point cloud data and mapping the point cloud into the vehicle body coordinate system.
[0100] 2. When the vehicle reaches a narrow turning space, it uses precise point cloud measurements of the space ahead, including walls and the ground. Combined with the vehicle's own dimensions, it plans a reasonable wheel path (vehicle forward trajectory data).
[0101] 4 , there is shown a second schematic diagram of point cloud distribution of an example of a vehicle scratch recognition method based on binocular stereo vision of the present application;
[0102] 3. When the vehicle enters a narrow turning space, the point cloud data measured at the previous moment is combined with the vehicle's speed, steering wheel angle, and body posture information to obtain the vehicle's displacement vector T and rotation matrix R. The vehicle's displacement vector T and rotation matrix R can be used to calculate the current position of the point cloud at the previous moment (such as the point cloud in the wireframe in Figure 4).
[0103] 4. Based on the current vehicle speed and cornering information, the system can estimate the measured point cloud position at the next moment. Combined with vehicle dimensions, the system can predict the vehicle's optimal driving path. If a scratch risk is detected, the system can proactively intervene to adjust the vehicle's speed and cornering direction to avoid a scratch.
[0104] 5 , there is shown a third schematic diagram of point cloud distribution of an example of a vehicle scratch recognition method based on binocular stereo vision of the present application;
[0105] 5. For the measured wall and ground point clouds, the wall curvature can be calculated and the point cloud position at the next moment (such as the point cloud in the wireframe of Figure 5) can be estimated. This increases the response time and the prediction range, allowing for earlier intervention in vehicle control when there is a scratch.
[0106] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0107] 6 , a block diagram of an embodiment of a vehicle scratch recognition device based on binocular stereo vision is shown. In this embodiment, a binocular stereo vision sensor is installed on a vehicle. The vehicle scratch recognition device based on binocular stereo vision may specifically include the following modules:
[0108] The first acquisition module 601 is used to acquire the initial image data and vehicle operating condition data collected by the binocular stereo vision sensor in real time when the vehicle reaches a narrow turning space;
[0109] 3D matching module 602 is used to perform 3D stereo matching on the initial image data to generate 3D point cloud data
[0110] The trajectory determination module 603 is used to generate vehicle forward driving trajectory data based on the three-dimensional point cloud data and vehicle operating condition data;
[0111] The scratch identification module 604 is used to determine the scratch risk information based on the vehicle's forward driving trajectory data and three-dimensional point cloud data.
[0112] In an optional embodiment of the present application, the device further includes:
[0113] The mapping module is used to map the camera coordinate system corresponding to the initial image data into the vehicle coordinate system of the vehicle.
[0114] In an optional embodiment of the present application, the three-dimensional point cloud data includes a time-continuous point cloud; the trajectory determination module 603 includes:
[0115] A first moment pose determination submodule is used to generate a first vehicle displacement vector and a first rotation matrix based on the point cloud and vehicle operating condition data at the previous moment;
[0116] A second moment pose determination submodule, configured to generate a second vehicle displacement vector and a second rotation matrix based on the current moment point cloud and vehicle operating condition data;
[0117] The trajectory determination submodule is used to generate vehicle forward driving trajectory data based on the first vehicle displacement vector, the first rotation matrix, the second vehicle displacement vector and the second rotation matrix.
[0118] In an optional embodiment of the present application, the device further includes:
[0119] The second acquisition module is used to obtain vehicle body size information;
[0120] The correction module is used to combine the vehicle body size information with the vehicle forward driving trajectory data to correct the vehicle forward driving trajectory data.
[0121] In an optional embodiment of the present application, the scratch identification module 604 includes:
[0122] The first scratch recognition submodule is used to determine that a scratch risk exists when the trajectory of the vehicle's forward driving trajectory data overlaps with the point cloud at the next moment;
[0123] The second scratch recognition submodule is used to determine that the scratch risk information is that there is no scratch risk when the trajectory of the vehicle's forward driving trajectory data and the point cloud at the next moment do not overlap.
[0124] In an optional embodiment of the present application, the device further includes:
[0125] A reading module is used to read the wall and ground point clouds from the three-dimensional point cloud data;
[0126] The road curvature calculation module is used to calculate the road curvature based on the position of the wall and ground point cloud;
[0127] The virtual point cloud module is used to fit the point cloud at the next moment according to the road curvature to generate a virtual point cloud. The virtual point cloud is the subsequent point cloud of the point cloud at the next moment in time. The virtual point cloud is used to increase the point cloud at the next moment.
[0128] In an optional embodiment of the present application, the device further includes:
[0129] The control module is used to control the operation of the vehicle when the scratch risk information indicates that there is a scratch risk.
[0130] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0131] 7 , an embodiment of the present application further provides a vehicle, including:
[0132] Processor 701 and storage medium 702, storage medium 702 stores a computer program executable by processor 701, when the vehicle is running, processor 701 executes the computer program to perform the vehicle scratch recognition method based on binocular stereo vision as any of the embodiments of the present application. The vehicle scratch recognition method based on binocular stereo vision includes:
[0133] When the vehicle reaches a narrow turning space, it acquires the initial image data and vehicle operating condition data collected by the binocular stereo vision sensor in real time;
[0134] Perform three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data;
[0135] Generate vehicle forward driving trajectory data based on 3D point cloud data and vehicle operating condition data;
[0136] Determine scratch risk information based on the vehicle's forward driving trajectory data and three-dimensional point cloud data.
[0137] Optionally, before performing three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data, the method further includes:
[0138] The camera coordinate system corresponding to the initial image data is mapped to the vehicle coordinate system of the vehicle.
[0139] Optionally, the three-dimensional point cloud data includes a time-continuous point cloud; and the step of generating vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data includes:
[0140] Generate a first vehicle displacement vector and a first rotation matrix based on the point cloud and vehicle operating condition data at a previous moment;
[0141] Generate a second vehicle displacement vector and a second rotation matrix based on the current point cloud and vehicle operating condition data;
[0142] Vehicle forward driving trajectory data is generated according to the first vehicle displacement vector, the first rotation matrix, the second vehicle displacement vector, and the second rotation matrix.
[0143] Optionally, the method further comprises:
[0144] Get vehicle size information;
[0145] The vehicle body size information is combined with the vehicle forward driving trajectory data to correct the vehicle forward driving trajectory data.
[0146] Optionally, the step of determining the scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data includes:
[0147] When the trajectory of the vehicle's forward driving trajectory data overlaps with the point cloud at the next moment, the scratch risk information is determined to indicate that there is a scratch risk;
[0148] When the trajectory of the vehicle's forward driving trajectory data and the point cloud at the next moment do not overlap, the scratch risk information is determined to be that there is no scratch risk.
[0149] Optionally, the method further comprises:
[0150] Read the wall and ground point clouds from the 3D point cloud data;
[0151] The road curvature is calculated based on the position of the wall ground point cloud;
[0152] The point cloud at the next moment is fitted according to the road curvature to generate a virtual point cloud. The virtual point cloud is the subsequent point cloud of the point cloud at the next moment in time. The virtual point cloud is used to increase the point cloud at the next moment.
[0153] Optionally, the method further comprises:
[0154] When the scratch risk information indicates that there is a scratch risk, the operation of the vehicle is controlled.
[0155] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.
[0156] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.
[0157] 8 , the present invention also provides a computer-readable storage medium 801 , which stores a computer program. When the computer program is executed by a processor, the vehicle scratch recognition method based on binocular stereo vision as described in any of the embodiments of the present invention is executed. The vehicle scratch recognition method based on binocular stereo vision includes:
[0158] When the vehicle reaches a narrow turning space, it acquires the initial image data and vehicle operating condition data collected by the binocular stereo vision sensor in real time;
[0159] Perform three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data;
[0160] Generate vehicle forward driving trajectory data based on 3D point cloud data and vehicle operating condition data;
[0161] Determine scratch risk information based on the vehicle's forward driving trajectory data and three-dimensional point cloud data.
[0162] Optionally, before performing three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data, the method further includes:
[0163] The camera coordinate system corresponding to the initial image data is mapped to the vehicle coordinate system of the vehicle.
[0164] Optionally, the three-dimensional point cloud data includes a time-continuous point cloud; and the step of generating vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data includes:
[0165] Generate a first vehicle displacement vector and a first rotation matrix based on the point cloud and vehicle operating condition data at a previous moment;
[0166] Generate a second vehicle displacement vector and a second rotation matrix based on the current point cloud and vehicle operating condition data;
[0167] Vehicle forward driving trajectory data is generated according to the first vehicle displacement vector, the first rotation matrix, the second vehicle displacement vector, and the second rotation matrix.
[0168] Optionally, the method further comprises:
[0169] Get vehicle size information;
[0170] The vehicle body size information is combined with the vehicle forward driving trajectory data to correct the vehicle forward driving trajectory data.
[0171] Optionally, the step of determining the scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data includes:
[0172] When the trajectory of the vehicle's forward driving trajectory data overlaps with the point cloud at the next moment, the scratch risk information is determined to indicate that there is a scratch risk;
[0173] When the trajectory of the vehicle's forward driving trajectory data and the point cloud at the next moment do not overlap, the scratch risk information is determined to be that there is no scratch risk.
[0174] Optionally, the method further comprises:
[0175] Read the wall and ground point clouds from the 3D point cloud data;
[0176] The road curvature is calculated based on the position of the wall ground point cloud;
[0177] The point cloud at the next moment is fitted according to the road curvature to generate a virtual point cloud. The virtual point cloud is the subsequent point cloud of the point cloud at the next moment in time. The virtual point cloud is used to increase the point cloud at the next moment.
[0178] Optionally, the method further comprises:
[0179] When the scratch risk information indicates that there is a scratch risk, the operation of the vehicle is controlled.
[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0181] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0182] The present application embodiment is described with reference to the flow chart and / or block diagram of the method, terminal device (system), and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flow chart and / or block diagram and the combination of the process and / or box in the flow chart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for realizing the function specified in one process or multiple processes and / or one box or multiple boxes of the flow chart.
[0183] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0185] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0186] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0187] The above is a detailed introduction to the vehicle scratch recognition method and device based on binocular stereo vision provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A vehicle scratch recognition method based on binocular stereo vision, wherein a binocular stereo vision sensor is installed on a vehicle, characterized in that: The method comprises: When the vehicle reaches a narrow turning space, initial image data and vehicle operating condition data collected by the binocular stereo vision sensor are acquired in real time; Performing three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data; Generating vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data; Based on the vehicle forward driving trajectory data and the three-dimensional point cloud data, scratch risk information is determined.
2. The method according to claim 1, characterized in that Before the step of performing three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data, the method further includes: The camera coordinate system corresponding to the initial image data is mapped to the vehicle coordinate system of the vehicle.
3. The method according to claim 1, characterized in that The three-dimensional point cloud data includes a time-continuous point cloud; the step of generating vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data includes: Generate a first vehicle displacement vector and a first rotation matrix according to the point cloud at the last moment and the vehicle operating condition data; Generate a second vehicle displacement vector and a second rotation matrix according to the current point cloud and the vehicle operating condition data; The vehicle forward driving trajectory data is generated according to the first vehicle displacement vector, the first rotation matrix, the second vehicle displacement vector and the second rotation matrix.
4. The method according to claim 3, characterized in that The method further comprises: Get vehicle size information; The vehicle body size information is combined with the vehicle forward driving trajectory data to correct the vehicle forward driving trajectory data.
5. The method according to claim 3 or 4, characterized in that: The step of determining the scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data comprises: When the trajectory of the forward driving trajectory data of the vehicle overlaps with the point cloud at the next moment, determining that the scratch risk information indicates that there is a scratch risk; When the trajectory of the vehicle's forward driving trajectory data and the point cloud at the next moment do not overlap, the scratch risk information is determined to be that there is no scratch risk.
6. The method according to claim 5, characterized in that The method further comprises: Reading wall and ground point clouds from the three-dimensional point cloud data; Calculating the road curvature according to the position of the wall ground point cloud; The point cloud at the next moment is fitted according to the road curvature to generate a virtual point cloud, where the virtual point cloud is a subsequent point cloud of the point cloud at the next moment in time, and the virtual point cloud is used to increase the point cloud at the next moment.
7. The method according to claim 1, characterized in that The method further comprises: When the scratch risk information indicates that there is a scratch risk, the operation of the vehicle is controlled.
8. A vehicle scratch recognition device based on binocular stereo vision, wherein a binocular stereo vision sensor is installed on a vehicle, characterized in that: The device comprises: A first acquisition module is used to acquire, in real time, initial image data and vehicle operating condition data collected by the binocular stereo vision sensor when the vehicle reaches a narrow turning space; A three-dimensional matching module, used for performing three-dimensional stereo matching on the initial image data to generate three-dimensional point cloud data; A trajectory determination module, used to generate vehicle forward driving trajectory data based on the three-dimensional point cloud data and the vehicle operating condition data; The scratch recognition module is used to determine the scratch risk information based on the vehicle forward driving trajectory data and the three-dimensional point cloud data.
9. A vehicle, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the vehicle scratch recognition method based on binocular stereo vision as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the vehicle scratch recognition method based on binocular stereo vision as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Auxiliary system for travelling on narrow road of vehicle
CN106945668A
A vehicle passability detection system based on vision and laser sensors
CN109359409A
Automatic driving control method
CN110262521A
Vehicle driving system adjusting method and system based on cross-country scene
CN115179920A
Vehicle rubbing identification method and device based on binocular stereo vision
CN116757918A
