A road surface detection method and system, a vehicle and a storage medium

Abstract

The application provides a road surface detection method and system, a vehicle and a storage medium, comprising: acquiring a target image, determining whether a target obstacle exists on a current road surface based on the target image; under the condition that the target obstacle exists on the current road surface, determining the world coordinates of feature points in the target image in a world coordinate system; and performing curve fitting according to the world coordinates of the feature points, and determining a target obstacle region based on the curve fitting result. The application identifies whether at least one of a protrusion and a depression exists on the current road surface by using a target image obtained by photographing the current road surface by using a binocular camera, and performs curve fitting according to the world coordinates of feature points in the target image to determine the target obstacle region, so that the protrusion and the depression in the current ground surface can be accurately identified, more road surface information is provided for a parking assistance system or a driver in the vehicle, and the vehicle can normally travel on more complex road surfaces.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more particularly to a road surface detection method and system, a vehicle, and a storage medium. Background Technology

[0002] Some vehicles are equipped with parking assistance systems, but these systems may have blind spots and problems such as missed or incorrect recognition. For example, they may not accurately recognize low obstacles or fail to recognize potholes and depressions, thus failing to meet the vehicle's driving needs on complex road surfaces. Summary of the Invention

[0003] This application provides a road surface detection method and system, a vehicle, and a storage medium to solve or alleviate the problems described above.

[0004] In a first aspect, this application provides a road surface detection method, comprising the following steps: acquiring a target image, wherein the target image is obtained by a binocular camera capturing the current road surface; determining whether a target obstacle exists on the current road surface based on the target image; wherein the target obstacle includes at least one of protrusions and depressions; determining the world coordinates of feature points in the target image in the world coordinate system under the condition that a target obstacle exists on the current road surface; and performing curve fitting based on the world coordinates of the feature points, and determining the target obstacle region based on the curve fitting result.

[0005] Compared with related technologies, this road surface detection method has at least the following beneficial effects: by capturing target images of the current road surface using a binocular camera, it identifies whether there are at least one of the following: bumps and depressions on the current road surface. Simultaneously, by performing curve fitting based on the world coordinates of feature points in the target image, it determines the target obstacle area. This method can accurately identify bumps and depressions on the current ground, thereby providing more road surface information to the parking assistance system or driver in the vehicle. It solves or alleviates the detection blind spots caused by the application of ultrasonic sensors during parking, avoids the problem of missed or incorrect identification of different road obstacles, and enables vehicles to drive normally on more complex road surfaces.

[0006] In one possible implementation, the process of curve fitting based on the world coordinates of the feature points and determining the target obstacle region based on the curve fitting result includes: forming a feature point set of the binocular camera based on the world coordinates of the feature points; performing spline fitting on the feature point set of the binocular camera to form a first spline curve and a second spline curve; wherein the first spline curve is obtained by spline fitting of the feature point set of one of the binocular cameras, and the second spline curve is obtained by spline fitting of the feature point set of the other binocular camera; sampling data points on the first spline curve and the second spline curve respectively, and forming a closed polygon according to the data point sampling result, which serves as the target obstacle region. Thus, this method achieves a fast approximate representation of a closed projection region by sampling and connecting two curves to form a closed polygon. Through this discretized polygon approximation, complex curve geometry operations can be transformed into simple polygon vertex vector operations, greatly simplifying the execution complexity of subsequent algorithms and enabling the algorithm to maintain a very high frame rate even on low-computing-power automotive chips.

[0007] In one possible implementation, after determining the target obstacle area based on curve fitting results, the method further includes: determining the obstacle attributes of the target obstacle based on the geometric parameters of the target obstacle area; and determining the threat level of the target obstacle to the target vehicle during driving based on the vehicle parameters of the target vehicle; and forming a driving strategy for the target vehicle based on the obstacle attributes and the threat level, so that the target vehicle can avoid or safely pass through the target obstacle area after executing the driving strategy. Therefore, this method, by determining the obstacle attributes and threat level of the target obstacle, can then form a driving strategy based on the obstacle attributes and threat level, ensuring that the target vehicle can directly avoid or safely pass through the target obstacle area after executing the driving strategy, thereby enabling the target vehicle to perform intelligent obstacle avoidance.

[0008] In one possible implementation, the geometric parameters include at least one of longitudinal span, lateral width, maximum height, and curvature extremum; and / or, the vehicle parameters include at least one of vehicle width, safety distance threshold, vehicle chassis ground clearance, and wheel diameter. Therefore, this method can determine the obstacle attributes of a target obstacle using at least one of longitudinal span, lateral width, maximum height, and curvature extremum, and can determine the threat level of the target obstacle using at least one of vehicle width, safety distance threshold, vehicle chassis ground clearance, and wheel diameter.

[0009] In one possible implementation, the process of acquiring the target image includes: acquiring a first image and a second image; wherein the first image is obtained by the first camera of the binocular camera capturing the current road surface, and the second image is obtained by the second camera of the binocular camera capturing the current road surface, the first camera being located above the second camera and in a straight line with the second camera, with a distance between the first camera and the second camera; preprocessing the first image and the second image, and using the preprocessed first image and the second image as the target image; wherein the preprocessing includes grayscale processing, enhancement processing, filtering processing, and subtraction processing. Therefore, this method obtains a first image and a second image by capturing the current road surface with a binocular camera, then performs image preprocessing on the obtained first image and the second image, converting the RGB (red, green, blue) signals in the image into grayscale signals, performing image enhancement operations on the grayscale signals, and then performing filtering and subtraction processing on the enhanced image information, thereby effectively removing noise in the image while preserving image edge and detail information. Therefore, this method uses the preprocessed image as the target image, which can eliminate the texture, noise, and light and shadow reflection interference of the background road surface, making it easier to accurately identify target obstacles such as protrusions and / or depressions in the future.

[0010] In one possible implementation, the process of determining whether a target obstacle exists on the current road surface based on the target image includes: acquiring near-end feature point projections and far-end feature point projections of the current road surface detection range by a first camera based on the first image, and acquiring near-end feature point projections and far-end feature point projections of the current road surface detection range by a second camera based on the second image; wherein, the near-end feature point projection is the feature point projection closer to the camera, and the far-end feature point projection is the feature point projection farther from the camera; the distance between the near-end feature point projection and the far-end feature point projection of the first camera is recorded as a first distance, and the distance between the near-end feature point projection and the far-end feature point projection of the second camera is recorded as a second distance; under the condition that the first distance and the second distance are equal, and the near-end feature point projections of the first camera and the second camera coincide, and the far-end feature point projections of the first camera and the second camera coincide, it is determined that there is no target obstacle on the current road surface; under the condition that the first distance and the second distance are not equal, and the near-end feature point projections of the first camera and the second camera do not coincide, and the far-end feature point projections of the first camera and the second camera do not coincide, it is determined that there is a target obstacle on the current road surface. Therefore, this method can determine whether there is a target obstacle on the current road surface by using the principle of edge projection drift. If the current road surface is completely flat, the detection range of each camera in the binocular camera is exactly the same, and the ground projection area will overlap. If the current road surface is uneven, the detection range of each camera in the binocular camera will have projection drift, and the ground projection area will not overlap.

[0011] In one possible implementation, when a target obstacle exists on the current road surface, the method further includes: recording the distance between the projection of the near-end feature point of the first camera and the projection of the near-end feature point of the second camera as the near-end line projection distance, and recording the distance between the projection of the far-end feature point of the first camera and the projection of the far-end feature point of the second camera as the far-end line projection distance; determining that the target obstacle is a protrusion when both the near-end line projection distance and the far-end line projection distance are greater than a preset threshold; and determining that the target obstacle is a depression when the near-end line projection distance is greater than the preset threshold and the far-end line projection distance is less than or equal to the preset threshold. Therefore, this method can determine the specific obstacle type of the target obstacle through the principle of edge projection drift. When the target obstacle is a protrusion, the three-dimensional height structure of the protrusion causes the feature point projection in the field of view of the binocular camera to drift; and when the target obstacle is a depression, the depth structure of the depression also causes the feature point projection in the field of view of the binocular camera to drift.

[0012] Secondly, this application provides a road surface detection system, comprising: an image acquisition module for acquiring a target image, the target image being obtained by a binocular camera capturing the current road surface; a road surface detection module for determining whether a target obstacle exists on the current road surface based on the target image, and, under the condition that a target obstacle exists on the current road surface, determining the world coordinates of feature points in the target image in the world coordinate system; and performing curve fitting based on the world coordinates of the feature points, and determining the target obstacle region based on the curve fitting result; wherein the target obstacle includes at least one of protrusions and depressions.

[0013] Compared with related technologies, this road surface detection system has at least the following beneficial effects: it identifies the presence of at least one of the following on the current road surface by capturing target images of the current road surface using a binocular camera; at the same time, it determines the target obstacle area by curve fitting based on the world coordinates of feature points in the target image; it can accurately identify the bumps and depressions in the current ground, thereby providing more road surface information to the parking assistance system in the vehicle or the driver, solving or mitigating the detection blind spots caused by the application of ultrasonic sensors during parking, avoiding the problems of missed or incorrect identification of different road obstacles, and enabling vehicles to drive normally on more complex road surfaces.

[0014] Thirdly, this application also provides a vehicle that includes the road surface detection system as described above.

[0015] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the road surface detection method described in any one of the above. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] In the attached diagram: Figure 1 This is a schematic flowchart of a road surface detection method provided in one embodiment of this application; Figure 2 This is a projection diagram of a road surface without target obstacles, provided in one embodiment of this application. Figure 3 This is a projection diagram of a road surface with protrusions, provided in one embodiment of this application. Figure 4This is a projection diagram of a road surface with depressions, provided in one embodiment of this application. Figure 5 A schematic diagram illustrating the determination of world coordinates of a feature point according to an embodiment of this application; Figure 6 A schematic flowchart of a road surface detection method provided in another embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of a road surface detection system provided in one embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of a computer device suitable for implementing one or more embodiments of this application. Detailed Implementation

[0018] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0019] It is understood that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0021] The inventors discovered that some vehicles are equipped with parking assistance systems (such as those based on ultrasonic detection principles). However, these systems may have blind spots, leading to issues like missed or incorrect identification. For example, they may inaccurately identify low obstacles such as parking posts, curbs, and speed bumps that are below the sensor's detection limit, and they may fail to detect potholes, depressions, and cracks in the road surface, thus failing to meet the vehicle's driving needs on complex road conditions. In practical applications, accurately detecting protruding obstacles and depressions on the road surface is crucial for parking assistance systems. However, related detection methods often rely on the vehicle's motion positioning, which can easily lead to accumulated errors, affecting the intelligence and safety of automatic parking.

[0022] Figure 1 A schematic flowchart of a road surface detection method is shown. Specifically, in an exemplary embodiment, as follows... Figure 1 As shown, this embodiment provides a road surface detection method, including the following steps: S110, acquire the target image, which is obtained by the binocular camera capturing the current road surface.

[0023] S120, determine whether there is a target obstacle on the current road surface based on the target image; wherein the target obstacle includes at least one of a protrusion and a depression.

[0024] S130, under the condition that there is a target obstacle on the current road surface, determine the world coordinates of the feature points in the target image in the world coordinate system; and, perform curve fitting based on the world coordinates of the feature points, and determine the target obstacle area based on the curve fitting result.

[0025] Therefore, this method uses a binocular camera to capture a target image of the current road surface to identify whether there are at least one of the bumps and depressions on the current road surface. At the same time, curve fitting is performed based on the world coordinates of feature points in the target image to determine the target obstacle area. This method can accurately identify bumps and depressions in the current ground, thereby providing more road information to the parking assistance system or driver in the vehicle. It solves or alleviates the detection blind spots caused by the use of ultrasonic sensors during parking, avoids the problem of missed or incorrect identification of different road obstacles, and enables the vehicle to drive normally on more complex road surfaces.

[0026] In an exemplary embodiment, step S110, acquiring the target image, may include: acquiring a first image and a second image; wherein the first image is obtained by the first camera of the binocular camera capturing the current road surface, and the second image is obtained by the second camera of the binocular camera capturing the current road surface, the first camera being located above the second camera and in the same straight line as the second camera, and there is a gap between the first camera and the second camera; preprocessing the first image and the second image, and using the preprocessed first image and the second image as the target image; wherein the preprocessing includes grayscale processing, enhancement processing, filtering processing, and subtraction processing.

[0027] In some embodiments, the first camera may also be referred to as the upper camera or the top camera, and the second camera may also be referred to as the lower camera or the bottom camera. A binocular camera consisting of the upper camera and the lower camera may also be referred to as an upper and lower binocular camera.

[0028] In some embodiments, the two cameras in the binocular camera can have the same parameters, for example, a horizontal FOV (field of view) of 200°±3°, a vertical FOV (field of view) of 146°±3°, and a resolution of 1920*1080.

[0029] In some embodiments, both cameras in a binocular camera may be short-range cameras.

[0030] In some embodiments, the two cameras in the binocular camera can be vertically mounted above and below the rear bumper beam of the vehicle, respectively. The camera located above the rear bumper beam can be referred to as the first camera, the upper camera, or the top camera, and the camera located below the rear bumper beam can be referred to as the second camera, the lower camera, or the bottom camera.

[0031] In some embodiments, the first image may also be referred to as the upper image, and the second image may also be referred to as the lower image.

[0032] In some embodiments, the process of converting the first image or the second image to grayscale can be as follows: converting the RGB image to a grayscale image using a weighted average method, that is, converting the RGB image to a monochrome image containing only brightness information, with the grayscale value ranging from 0 to 255. The following expression is obtained by weighting the RGB components based on human eye sensitivity to calculate the grayscale value: In the formula, Represents the x-coordinate of a pixel in the image pixel coordinate system, in pixels; Represents the ordinate of a pixel in the image pixel coordinate system, in pixels; Represents the grayscale value of the image; Indicates the value of the red pixel; Indicates the value of the green pixel; This represents the value of the blue pixel.

[0033] In some embodiments, the process of enhancing the grayscale first image or the second image may be: sharpening and contrast stretching the grayscale image to improve the image's clarity and contrast.

[0034] In some embodiments, the process of filtering the enhanced first or second image can be as follows: Gaussian filtering is applied to the enhanced image information, with a filter kernel size of 5×5, to remove image noise. Gaussian filtering is a linear filter that achieves a smoothing effect by convolving the image with a Gaussian kernel. The Gaussian kernel is a two-dimensional Gaussian function, as shown in the formula: In the formula, σ is the standard deviation of the Gaussian distribution, which determines the diffusion degree of the filter. Represents the Gaussian function value. This represents the coordinates of a pixel in the image, expressed in pixels. Gaussian filtering assigns weights to pixels based on their distance from the center, with pixels farther from the center receiving lower weights. This allows the image to be smoothed while preserving its main contours.

[0035] In some embodiments, the process of subtracting the filtered first or second image can be as follows: the subtraction operation is based on the image projection of the identified road surface protrusions or depressions to identify the unevenness and corresponding depth data of the ground. Specifically, the gray value of each pixel is subtracted from the corresponding reference gray value, and the difference is used as the ground unevenness signal. The reference gray value is the gray value of a flat surface, which is obtained by repeatedly capturing images of the same area after the vehicle has come to a complete stop, and taking the average value as the reference gray value. Furthermore, the ground unevenness signal identified by the binocular camera and the information identified by the ultrasonic radar can be fused together.

[0036] Therefore, this method uses a binocular camera to capture the current road surface, obtaining a first image and a second image. Then, it performs image preprocessing on both images, converting the RGB (red, green, blue) signals into grayscale signals, performing image enhancement operations on the grayscale signals, and then filtering and subtracting the enhanced image information to effectively remove noise while preserving image edges and details. Thus, this method uses the preprocessed image as the target image, eliminating textures, noise, and light reflections from the background road surface, facilitating accurate identification of obstacles such as protrusions and / or depressions.

[0037] In an exemplary embodiment, step S120, which determines whether a target obstacle exists on the current road surface based on the target image, may include: acquiring near-end feature point projections and far-end feature point projections of the current road surface detection range from a first camera based on a first image, and acquiring near-end feature point projections and far-end feature point projections of the current road surface detection range from a second camera based on a second image; recording the distance between the near-end feature point projections and far-end feature point projections of the first camera as a first distance, and recording the distance between the near-end feature point projections and far-end feature point projections of the second camera as a second distance; determining that no target obstacle exists on the current road surface when the first distance and the second distance are equal, and the near-end feature point projections of the first camera and the second camera coincide, and the far-end feature point projections of the first camera and the second camera coincide; determining that a target obstacle exists on the current road surface when the first distance and the second distance are not equal, and the near-end feature point projections of the first camera and the second camera do not coincide, and the far-end feature point projections of the first camera and the second camera do not coincide. In some examples, the near-end feature point projection is the feature point projection closer to the camera, and the far-end feature point projection is the feature point projection farther from the camera.

[0038] In some embodiments, such as Figure 2 As shown, a projection diagram is provided when there are no target obstacles on the road surface. Figure 2 In this diagram, the first camera is designated O, and the second camera is designated O'. Cameras O and O' can be installed above and below the rear bumper beam of a vehicle. The distance between the first and second cameras can be referred to as the optical center distance. Figure 2 The optical center distance between the first and second cameras is h, and the height distance between the first camera O and the ground is H. Figure 2 In the diagram, 'a' represents the projection of near-end feature points of the current road surface detection range by the first camera O, and 'b' represents the projection of far-end feature points of the current road surface detection range by the first camera O. 'a' represents the projection of near-end feature points of the current road surface detection range by the second camera O', and 'b' represents the projection of far-end feature points of the current road surface detection range by the second camera O'. 'ab' represents the distance between the near-end and far-end feature point projections of the first camera O, i.e., the first distance. 'b' represents the distance between the near-end and far-end feature point projections of the first camera O', i.e., the second distance. For example... Figure 2 As shown, if there is no target obstacle on the current road surface, the detection range of the first camera and the second camera on the road surface is the same, that is, ab=a'b', a and a' overlap, and b and b' overlap.

[0039] In some embodiments, such as Figure 3 As shown, a projection diagram is provided when there is a bump in the road surface. Figure 3 In the text, the meanings of the parameters ab, a'b', a, a', b, b', O, O', H, and h are the same as... Figure 2 The same applies, so it will not be repeated here. For example... Figure 3 As shown, if there is a bump on the current road surface, the three-dimensional structure of the bump causes the projection of feature points in the field of view of the two cameras to shift. At this time, the detection range of the road surface by the upper camera and the lower camera is different, that is, ab≠a'b', a and a' do not coincide, and b and b' do not coincide.

[0040] In some embodiments, such as Figure 4 As shown, a projection diagram of a road surface with depressions is provided. Figure 4 In the text, the meanings of the parameters ab, a'b', a, a', b, b', O, O', H, and h are the same as... Figure 2 The same applies, so it will not be repeated here. For example... Figure 4 As shown, if there is a depression on the current road surface, the depth structure of the depression will also cause the projection of feature points in the field of view of the two cameras to shift. At this time, the detection range of the road surface by the upper camera and the lower camera is different, that is, ab≠a'b', a and a' do not coincide, and b and b' do not coincide.

[0041] Therefore, this method can determine whether there is a target obstacle on the current road surface by using the principle of edge projection drift. If the current road surface is completely flat, the detection range of each camera in the binocular camera is exactly the same, and the ground projection area will overlap. If the current road surface is uneven, the detection range of each camera in the binocular camera will have projection drift, and the ground projection area will not overlap.

[0042] In an exemplary embodiment, when a target obstacle exists on the current road surface, the road surface detection method may further include: recording the distance between the projection of a near-end feature point of the first camera and the projection of a near-end feature point of the second camera as the near-end line projection distance, and recording the distance between the projection of a far-end feature point of the first camera and the projection of a far-end feature point of the second camera as the far-end line projection distance; determining the target obstacle as a protrusion when both the near-end line projection distance and the far-end line projection distance are greater than a preset threshold; and determining the target obstacle as a depression when the near-end line projection distance is greater than the preset threshold and the far-end line projection distance is less than or equal to the preset threshold. In some examples, the specific value of the preset threshold can be selected or set according to the actual situation, and no specific value is limited here.

[0043] In some embodiments, such as Figure 3 As shown, a projection diagram is provided when there is a bump in the road surface. Figure 3In this diagram, aa' represents the distance between the projections of the near-end feature points of the first camera and the near-end feature points of the second camera, i.e., the near-end line projection distance; bb' represents the distance between the projections of the far-end feature points of the first camera and the far-end feature points of the second camera, i.e., the far-end line projection distance. aa' can also be called the front edge projection separation distance, and bb' can also be called the rear edge projection separation distance. According to the principle of similar triangles, when there are protruding obstacles on the road surface, the closer the top edge of the obstacle is to the camera, the larger the values ​​of aa' and bb'. In real-world environments, protruding obstacles all possess certain height information. Therefore, by detecting the edge drift of protruding obstacles with front edge drift (e.g., the front of a step) and protruding obstacles with rear edge drift (e.g., gullies, the rear of steps), the obstacle detection problem can be transformed into the problem of identifying the top edge projection drift aa' and bb' of the obstacle.

[0044] In some embodiments, such as Figure 4 As shown, a projection diagram of a road surface with depressions is provided. Figure 4 In Chinese, the parameters aa' and bb' have the same meaning as... Figure 3 The same principle applies here, so it will not be repeated. According to the principle of similar triangles, when there is a depression on the road surface, the farther the top edge of the depression is from the camera, the larger the value of aa'. In real-world environments, depressions all possess a certain depth information. By detecting the edge drift of depressions with front and rear edge drift, the depression detection problem can be transformed into the problem of identifying the projection drift aa' of the top edge of the depression.

[0045] Therefore, this method can determine the specific type of obstacle by using the principle of edge projection drift. When the target obstacle is raised, the three-dimensional height structure of the raised object causes the feature point projection in the field of view of the binocular camera to drift. When the target obstacle is recessed, the depth structure of the recess also causes the feature point projection in the field of view of the binocular camera to drift.

[0046] In an exemplary embodiment, step S130, determining the world coordinates of a feature point in the target image in the world coordinate system, may involve: obtaining the imaging point position of the feature point on the CCD (charge coupled device camera) imaging plane of the first camera, denoted as the first imaging point position; and obtaining the imaging point position of the feature point on the CCD imaging plane of the second camera, denoted as the second imaging point position; calculating the camera coordinates of the feature point based on the first imaging point position, the second imaging point position, the distance between the camera focus and the CCD imaging plane, the optical center distance between the first camera and the second camera, the camera imaging parallax, and the pixel coordinates of the feature point; and converting the camera coordinates of the feature point into world coordinates in the world coordinate system using the inverse transformation of the extrinsic parameter matrix.

[0047] In some embodiments, such as Figure 5 The diagram illustrates how to determine the world coordinates of a feature point. Figure 5 In the diagram, P1(x1, y1) is the image point position of feature point P on the CCD imaging plane of the first camera O1, and P2(x2, y2) is the image point position of feature point P on the CCD imaging plane of the second camera O2. The distance between the camera focal point and the CCD imaging plane is f, the optical center distance between the first camera O1 and the second camera O2 is b, and the camera imaging parallax is denoted as d. The calculation process for the camera coordinates P(X, Y, Z) of feature point P can be: Z / f = Y / y1 = (Yb) / (y2-b) = X / x1 = X / x2, d = y1 - (y2-b), Z = b*f / d, Y = Z*y1 / d, X = Z*x1 / f = Z*x2 / f. Once the camera coordinates P(X, Y, Z) of feature point P are known, they need to be converted to world coordinates P. w (X w Y w Z w If the extrinsic parameter matrix is ​​inversely transformed, the camera coordinates of the feature point can be converted to world coordinates in the world coordinate system, resulting in: P w =R T *P c +C; where R is the formula. T P represents the transpose of the rotation matrix R. c Let P be the camera coordinates of feature point P, C be the camera's position in the world coordinate system, and denot be the translation vector. The inverse of the extrinsic parameter matrix can be expressed as: M ext -1 = .

[0048] Therefore, this method can locate the target obstacle by determining the world coordinates of the feature points in the target image in the world coordinate system, which facilitates the subsequent formation of the target obstacle area.

[0049] In an exemplary embodiment, step S130, which involves performing curve fitting based on the world coordinates of feature points and determining the target obstacle region based on the curve fitting result, may include: forming a feature point set of the binocular cameras based on the world coordinates of the feature points; performing spline fitting on the feature point set of the binocular cameras to form a first spline curve and a second spline curve; sampling data points on the first spline curve and the second spline curve, and forming a closed polygon according to the data point sampling result, which serves as the target obstacle region. In some examples, B-spline curve fitting technology can be used to perform spline fitting on the feature point set of the binocular cameras to form a first B-spline curve and a second B-spline curve. In some examples, the first spline curve can be obtained by spline fitting on the feature point set of one of the binocular cameras, and the second spline curve can be obtained by spline fitting on the feature point set of the other binocular camera.

[0050] In some embodiments, if the feature point set formed based on the world coordinates of the feature points includes the front edge feature point set P front and the feature point set P of the back edge back , where P front ={ Pf 1, Pf 2, ..., Pfp}, Pfi =( xfi , yfi ); P back ={ Pb 1, Pb 2, ..., Pbq}, Pbj =( xbj , ybj ).

[0051] The parameter value corresponding to each point in the feature point set is determined using the chord length parameterization method. And control points are obtained by back-calculation of control points. Pi After obtaining the control points P 0, P 1, ..., Pn After 1, for u ∈[0,1] with a fixed step size (such as Δ) u Sampling is performed at 0.01, the coordinates of points on the curve are calculated, and a smooth fitting edge is obtained: .in, Defined by the following recursive formula: ; .

[0052] Let the front edge B-spline curve be Cf( u The back edge B-spline curve is Cb ( u ), u ∈[0,1]; then the obstacle region Ω is: Ω={( x , y )∣ u ∈[0,1], y ∈[ yCb ( u ), yCf ( u For protruding obstacles, the front edge B-spline curve is closer to the vehicle, i.e. yCf ( u )< yCb ( u ).

[0053] Sample M points from both the front and back B-spline curves, and construct closed polygons according to a preset order. Here, M is a positive integer, which can be selected or set according to actual conditions; no specific numerical limit is specified here. For example, M can be 20. The closed polygons can be constructed in the following order: Cf ( u 0)→ Cf ( u 1)→...→ Cf ( u M → Cb ( u M →...→ Cb ( u 0)→ Cf ( u 0).

[0054] Therefore, this method achieves a fast approximate representation of a closed projection region by sampling and connecting two curves to form a closed polygon. This discretized polygon approximation transforms complex curve geometry operations into simple polygon vertex vector operations, greatly simplifying the execution complexity of subsequent algorithms and enabling the algorithm to maintain a very high frame rate even on low-computing-power automotive chips.

[0055] In an exemplary embodiment, after determining the target obstacle area based on the curve fitting results, the road surface detection method may further include: determining the obstacle attributes of the target obstacle based on the geometric parameters of the target obstacle area; and determining the threat level of the target obstacle to the target vehicle during driving based on the vehicle parameters of the target vehicle; and forming a driving strategy for the target vehicle based on the obstacle attributes and threat level, so that the target vehicle can avoid or safely pass through the target obstacle area after executing the driving strategy.

[0056] In some embodiments, the geometric parameters include, but are not limited to, at least one of longitudinal span, lateral width, maximum height, and curvature extremum.

[0057] In some embodiments, vehicle parameters include, but are not limited to, at least one of vehicle width, safety distance threshold, vehicle chassis ground clearance, and wheel diameter.

[0058] In some embodiments, the process of determining the obstacle attributes of a target obstacle based on the geometric parameters of the target obstacle region may include: Calculate the longitudinal span of the target obstacle area ,have: In the formula, This indicates the number of points sampled for the B-spline curve. This represents the parameters of the B-spline curve. Evenly distributed; This indicates that the front edge B-spline curve has the following parameters. The vertical coordinate (y-direction coordinate) at the location. This indicates that the back edge B-spline curve has the following parameters. The vertical coordinate at the location. The vertical span D is measured in centimeters (cm).

[0059] Calculate the lateral width of the target obstacle area W ,have: W =max( xCf ) min( xCf In the formula, xCf This represents the set of lateral coordinates of all points on the front edge B-spline curve. `max()` and `min()` represent the maximum and minimum value operations, respectively, used to obtain the extreme values ​​of the lateral coordinates. The lateral width... W The unit is centimeters or cm.

[0060] Calculate the maximum height of the target obstacle area Hmax ,have: Hmax =max(z fi ∪z bj In the formula, Z fi Z represents the set of heights of feature points on the front edge B-spline curve. bj This represents the set of heights of feature points on the back-edge B-spline curve. Where, degrees... Hmax The unit is centimeters or cm.

[0061] Calculate the curvature extrema of the target obstacle region ,have: This is used to determine the cusp of an obstacle. In the formula, This indicates that the B-spline curve has the following parameters. The first derivative at that point, i.e., the tangent vector component; : indicates that the B-spline curve has the following parameters The second derivative at that point, i.e., the acceleration vector component; This indicates all parameters The corresponding curvature is taken as the maximum value. On the entire B-spline curve, find the point with the maximum curvature. Among these, the curvature extrema are... The unit is cm -1 .

[0062] The most dangerous points in the target obstacle area are: Pdanger = w 1 + w 2 κ ( u ),in u =argmin yCf ( u ), that is, the front edge B-spline curve The parameter corresponding to the point closest to the vehicle; where, : indicates a B-spline curve based on the front edge The longitudinal distance from the point closest to the vehicle. κ ( u ) represents a B-spline curve based on the front edge. The local curvature of the point closest to the vehicle, where w1 and w2 are non-negative weight coefficients, satisfying w1+w2=1.

[0063] Based on the calculated geometric parameters, the obstacle attributes are determined according to the rules set in Table 1.

[0064] Table 1

[0065] In some embodiments, the process of determining the threat level of a target obstacle to a target vehicle during its operation, based on vehicle parameters of the target vehicle, may include: The threat level is calculated based on vehicle width, safe distance threshold, vehicle ground clearance, or wheel diameter. ,have: In the formula, α 1. α 2. α 3 represents the weighting coefficients. α 1. α 2 and α The sum of 3 is 1; This indicates the width of the vehicle, measured in centimeters (cm). The unit for indicating the safe distance threshold is centimeters (cm). When there is a raised obstacle, it indicates the ground clearance of the vehicle chassis; when there is a dent, it indicates the wheel diameter, which is in centimeters or cm.

[0066] Threat levels can be categorized as follows: T ≤0.3: classified as low risk; 0.3< T ≤0.7: Classified as medium risk; T >0.7: Classified as high risk.

[0067] In some embodiments, curvature extrema can also be used. Determine whether there are sharp points on the surface of the obstacle, as shown in Table 2 below.

[0068] Table 2

[0069] In some embodiments, the process of forming a driving strategy for a target vehicle based on obstacle attributes and threat levels may include: when the threat level is low risk, the corresponding driving strategy may be to only record obstacle attributes and drive normally; when the threat level is medium risk, the corresponding driving strategy may be to trigger a warning based on obstacle attributes and perform routine avoidance of the target obstacle; when the threat level is high risk, the corresponding driving strategy may be to trigger emergency braking or expand the safety radius by 10% to avoid scraping the target obstacle.

[0070] Therefore, this method can determine the obstacle attributes of a target obstacle by its longitudinal span, lateral width, maximum height, and curvature extreme value, and determine the threat level of the target obstacle by its vehicle width, safe distance threshold, ground clearance, and wheel diameter. Furthermore, by determining the obstacle attributes and threat level of the target obstacle, this method can formulate a driving strategy based on these attributes and threat levels, ensuring that the target vehicle can directly avoid or safely pass through the target obstacle area after executing the driving strategy, thus enabling the target vehicle to perform intelligent obstacle avoidance.

[0071] In an exemplary embodiment of this application, as Figure 6 As shown, a road surface detection method is provided, including the following steps: Images are captured using dual cameras, one above and one below, for example, images of the vehicle from the rear.

[0072] The acquired images are preprocessed, including grayscale conversion, enhancement, filtering, and subtraction.

[0073] The preprocessed image is input into the top edge projection drift model for analysis to identify ground concavity and convexity information and extract the front edge separation distance aa' and the rear edge separation distance bb'. Edge projection drift analysis can reduce the computational power and real-time requirements of traditional monocular or binocular cameras.

[0074] The world coordinates of feature points are located using the disparity map method, and the feature point sets of the front and rear edges are extracted based on these world coordinates.

[0075] B-spline curves are fitted to the feature point sets of the front and rear edges to construct the closed contour region of the obstacle, and geometric parameters such as the longitudinal span, lateral width, maximum height, and curvature extremum of the closed contour region are calculated.

[0076] Based on the geometric parameters of the closed contour region, attribute determination is performed to identify the obstacle type. Furthermore, based on vehicle parameters such as vehicle width, safe distance threshold, ground clearance, or wheel diameter, obstacle threat assessment is conducted to determine the threat level. The cusp is identified based on curvature extrema to adjust the avoidance strategy, thereby outputting the final driving decision. This includes: for low-risk situations, the corresponding driving strategy may be to simply record obstacle attributes and drive normally; for medium-risk situations, the corresponding driving strategy may be to trigger a warning based on obstacle attributes and perform conventional avoidance of the target obstacle; for high-risk situations, the corresponding driving strategy may be to trigger emergency braking or increase the safety radius by 10% to avoid collision with the target obstacle. This final driving decision is used as the vehicle's driving strategy so that the target vehicle, after executing the driving strategy, avoids or safely passes through the target obstacle area.

[0077] In some examples, Figure 1 or Figure 6 The road surface detection method shown can be used after the driver shifts the gear to reverse (R).

[0078] In some examples, Figure 1 or Figure 6 The road surface detection method shown can be the detection process for parking assistance systems or obstacle detection systems in vehicles.

[0079] In summary, this application provides a road surface detection method that uses a binocular camera to capture a target image of the current road surface to identify whether there are at least one of the following: bumps and depressions. Simultaneously, curve fitting is performed based on the world coordinates of feature points in the target image to determine the target obstacle area. This method can accurately identify bumps and depressions in the current ground, thereby providing more road surface information to the parking assistance system or driver in the vehicle. It solves or alleviates the detection blind spots caused by the use of ultrasonic sensors during parking, avoids the problem of missed or incorrect identification of different road obstacles, and enables vehicles to drive normally on more complex road surfaces.

[0080] In an exemplary embodiment of this application, as Figure 7 As shown, a road surface detection system is provided, including: The image acquisition module 710 is used to acquire the target image, which is obtained by the binocular camera capturing the current road surface. The road surface detection module 720 is used to determine whether there is a target obstacle on the current road surface based on the target image, and under the condition that there is a target obstacle on the current road surface, to determine the world coordinates of the feature points in the target image in the world coordinate system; and to perform curve fitting based on the world coordinates of the feature points, and to determine the target obstacle area based on the curve fitting result; wherein the target obstacle includes at least one of protrusions and depressions.

[0081] In summary, this application provides a road surface detection system that uses a binocular camera to capture a target image of the current road surface to identify whether there are at least one of the following: bumps and depressions. Simultaneously, it performs curve fitting based on the world coordinates of feature points in the target image to determine the target obstacle area. This system can accurately identify bumps and depressions in the current ground, thereby providing more road surface information to the parking assistance system or driver in the vehicle. It solves or alleviates the detection blind spots caused by the use of ultrasonic sensors during parking, avoids the problem of missed or incorrect identification of different road obstacles, and enables vehicles to drive normally on more complex road surfaces.

[0082] It is understood that the road surface detection system provided in this embodiment and the road surface detection method provided in the above embodiments belong to the same concept. The specific way in which the road surface detection method performs its operations has been described in detail in the above method embodiments, and will not be repeated here. In practical applications, the road surface detection system provided in the above embodiments can be assigned to different functional modules as needed. That is, the internal structure of the road surface detection system can be divided into different functional modules, and then all or part of the functions of the corresponding functional modules can be implemented through the road surface detection method described in the above embodiments. No specific limitations are imposed here.

[0083] In another exemplary embodiment of this application, a vehicle is provided that includes a road surface detection system as described in some of the embodiments above. Since the specific manner in which the road surface detection system performs its operation has been described in detail in the embodiments, the technical functions and effects of the vehicle provided in this embodiment can be found in the above embodiments, and will not be repeated here.

[0084] In an exemplary embodiment of this application, a computer device is also provided. The computer device may include a memory, a processor, and a computer program stored in the memory. The processor can execute the computer program to cause the computer device to perform actions such as... Figure 1 or Figure 6 The steps of the road surface inspection method are shown. Figure 8 A schematic diagram of the structure of a computer device 1000 is shown. (See attached diagram.) Figure 8 As shown, the computer device 1000 includes: a processor 1010, a memory 1020, a power supply 1030, a display unit 1040, and an input unit 1060.

[0085] The processor 1010 is the control center of the computer device 1000. It connects various components via interfaces and lines, and performs various functions of the computer device 1000 by running or executing computer programs / instructions stored in the memory 1020, thereby providing overall monitoring of the computer device 1000. In some embodiments, when the processor 1010 calls a computer program stored in the memory 1020, it can execute, for example... Figure 1 or Figure 6 The steps of the road surface detection method are shown. Optionally, the processor 1010 may include one or more processing units; preferably, the processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. In some embodiments, the processor 1010 and the memory 1020 may be implemented on a single chip; in other embodiments, they may be implemented on separate chips.

[0086] The memory 1020 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, various applications, etc.; the data storage area can store instruction data created according to the use of the computer device 1000. In addition, the memory 1020 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0087] The computer device 1000 also includes a power supply 1030 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 1010 through a power management system, thereby enabling the management of functions such as charging, discharging, and power consumption through the power management system.

[0088] The display unit 1040 can be used to display information input by the user or information provided to the user, and can also be used to display various menus of the computer device 1000, etc. In this embodiment, it is mainly used to display the display interface of each application in the computer device 1000, as well as text, pictures, and other objects displayed in the display interface. The display unit 1040 may include a display panel 1050. The display panel 1050 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0089] The input unit 1060 can be used to receive information such as numbers or characters input by the user. The input unit 1060 may include a touch panel 1070 and other input devices 1080. The touch panel 1070 can also be referred to as a touch screen, and the touch panel 1070 can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1070).

[0090] Specifically, the touch panel 1070 can detect user touch operations and the signals generated by these operations, convert these signals into touch point coordinates and send them to the processor 1010, and receive and execute commands transmitted by the processor 1010. Furthermore, the touch panel 1070 can employ various input methods such as resistive, capacitive, infrared, and surface acoustic waves to achieve interaction. Other input devices 1080 include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, and joystick.

[0091] Of course, the touch panel 1070 can also cover the display panel 1050. When the touch panel 1070 detects a touch operation on or near it, it can transmit the information to the processor 1010 to determine the type of touch event. Subsequently, the processor 1010 provides corresponding visual output on the display panel 1050 based on the type of touch event. Although in Figure 8 In this embodiment, the touch panel 1070 and the display panel 1050 are two separate components to realize the input and output functions of the computer device 1000. However, in some embodiments, the touch panel 1070 and the display panel 1050 can be integrated to realize the input and output functions of the computer device 1000.

[0092] The computer device 1000 may also include one or more sensors, such as pressure sensors, gravity acceleration sensors, proximity sensors, etc. Of course, depending on the specific application scenario, the computer device 1000 may also include other components such as cameras.

[0093] In an exemplary embodiment of this application, a computer-readable storage medium is also provided, which stores a computer program / instructions. When executed by a processor, the computer program / instructions enable the aforementioned computer device to perform the functions described in this application. Figure 1 or Figure 6 The steps of the road surface inspection method are shown.

[0094] It will be understood by those skilled in the art that Figure 8 This is merely an example of a computer device and does not constitute a limitation on the device. The device may include more or fewer components than illustrated, or a combination of certain components, or different components. For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components. For example, as some examples, the aforementioned computer device may be a vehicle, in-vehicle infotainment system, vehicle-mounted terminal, etc.

[0095] It is understood that although terms such as "first," "second," etc., may be used in this application to describe cameras, these terms are only used to distinguish cameras from each other. For example, without departing from the scope of the embodiments of this application, a first camera may also be referred to as a second camera, and similarly, a second camera may also be referred to as a first camera.

[0096] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A road surface detection method, characterized in that, The method includes the following steps: Acquire a target image, which is obtained by a binocular camera capturing the current road surface; Based on the target image, determine whether there is a target obstacle on the current road surface; wherein, the target obstacle includes at least one of a protrusion and a depression; Given the presence of a target obstacle on the current road surface, determine the world coordinates of the feature points in the target image in the world coordinate system; and perform curve fitting based on the world coordinates of the feature points, and determine the target obstacle area based on the curve fitting result.

2. The road surface detection method according to claim 1, characterized in that, The process of performing curve fitting based on the world coordinates of the feature points and determining the target obstacle region based on the curve fitting results includes: The feature point set of the binocular camera is formed based on the world coordinates of the feature points; Spline fitting is performed on the feature point set of the binocular cameras to form a first spline curve and a second spline curve, respectively; wherein, the first spline curve is obtained by spline fitting of the feature point set of one of the binocular cameras, and the second spline curve is obtained by spline fitting of the feature point set of the other binocular camera. Data points are sampled from the first spline curve and the second spline curve respectively, and a closed polygon is formed according to the data point sampling results, which serves as the target obstacle region.

3. The road surface detection method according to claim 1 or 2, characterized in that, After determining the target obstacle region based on the curve fitting results, the method further includes: Based on the geometric parameters of the target obstacle area, the obstacle attributes of the target obstacle are determined; and based on the vehicle parameters of the target vehicle, the threat level of the target obstacle to the target vehicle during its operation is determined. A driving strategy for the target vehicle is formed based on the obstacle attributes and the threat level, so that the target vehicle can avoid or safely pass through the target obstacle area after executing the driving strategy.

4. The road surface detection method according to claim 3, characterized in that, The geometric parameters include at least one of longitudinal span, lateral width, maximum height, and curvature extremum; and / or, the vehicle parameters include at least one of vehicle width, safety distance threshold, vehicle chassis ground clearance, and wheel diameter.

5. The road surface detection method according to claim 1, characterized in that, The process of acquiring the target image includes: Acquire a first image and a second image; wherein the first image is obtained by the first camera of the binocular camera capturing the current road surface, and the second image is obtained by the second camera of the binocular camera capturing the current road surface, the first camera is located above the second camera and is in the same straight line as the second camera, and there is a gap between the first camera and the second camera; The first image and the second image are preprocessed, and the preprocessed first image and the second image are used as the target image; wherein, the preprocessing includes grayscale processing, enhancement processing, filtering processing and subtraction processing.

6. The road surface detection method according to claim 5, characterized in that, The process of determining whether there is a target obstacle on the current road surface based on the target image includes: Based on the first image, the near-end feature point projection and far-end feature point projection of the first camera on the current road surface detection range are obtained, and based on the second image, the near-end feature point projection and far-end feature point projection of the second camera on the current road surface detection range are obtained; wherein, the near-end feature point projection is the feature point projection close to the camera, and the far-end feature point projection is the feature point projection far from the camera. The distance between the projection of the near-end feature point and the projection of the far-end feature point of the first camera is recorded as the first distance, and the distance between the projection of the near-end feature point and the projection of the far-end feature point of the second camera is recorded as the second distance. Under the conditions that the first distance and the second distance are equal, and the projections of the near-end feature points of the first camera and the second camera coincide, and the projections of the far-end feature points of the first camera and the second camera coincide, it is determined that there is no target obstacle on the current road surface. Under the conditions that the first distance and the second distance are not equal, the near-end feature point projections of the first camera and the second camera do not overlap, and the far-end feature point projections of the first camera and the second camera do not overlap, it is determined that there is a target obstacle on the current road surface.

7. The road surface detection method according to claim 6, characterized in that, Given that there is a target obstacle on the current road surface, the method further includes: The distance between the projection of the near-end feature point of the first camera and the projection of the near-end feature point of the second camera is denoted as the near-end line projection distance, and the distance between the projection of the far-end feature point of the first camera and the projection of the far-end feature point of the second camera is denoted as the far-end line projection distance. If both the near-end line projection distance and the far-end line projection distance are greater than a preset threshold, the target obstacle is determined to be a protrusion. If the projection distance of the near end line is greater than the preset threshold and the projection distance of the far end line is less than or equal to the preset threshold, the target obstacle is determined to be a depression.

8. A road surface detection system, characterized in that, The system includes: The image acquisition module is used to acquire a target image, which is obtained by a binocular camera capturing the current road surface. The road surface detection module is used to determine whether there is a target obstacle on the current road surface based on the target image, and, if there is a target obstacle on the current road surface, to determine the world coordinates of the feature points in the target image in the world coordinate system. Furthermore, curve fitting is performed based on the world coordinates of the feature points, and the target obstacle region is determined based on the curve fitting result; wherein the target obstacle includes at least one of protrusions and depressions.

9. A vehicle, characterized in that, The vehicle includes the road surface detection system as described in claim 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the road surface detection method according to any one of claims 1 to 7.

Images