Road element detection method and device and storage medium

Abstract

The invention discloses a road element detection method and device and a storage medium, relates to the technical field of image analysis, and can realize road element detection for a monocular image and reduce the requirement on hardware. The method comprises the following steps: training an initial neural network model according to a sample image and modeling information of road elements in the pre-labeled sample image to obtain a modeling prediction model; inputting a target image of a to-be-detected road element into the modeling prediction model, and outputting modeling information of the road element in the target image; the sample image and the target image are monocular images; and generating a fitting equation of the road elements in a world coordinate system corresponding to the target image according to the modeling information of the road elements in the target image and the camera parameters.

Description

Technical Field

[0001] The present application relates to the field of image analysis technology, and in particular to a road element detection method, device and storage medium. Background Art

[0002] The development of automation technology, such as driverless technology, relies on accurate judgment of road elements, such as sidewalk curbs, flower bed curbs, concrete walls, railings, water barriers and other non-drivable area boundaries.

[0003] At present, three-dimensional (3D) road element detection usually requires the collection of multi-eye images or laser point clouds of road elements, and the fitting of 3D images of road elements through the logic algorithm of stereo matching. However, since the collection of multi-eye images or laser point clouds requires the deployment of additional multi-eye cameras or laser radar systems, the general method has at least a technical difficulty of high hardware requirements, which is not conducive to the promotion of technology. Summary of the invention

[0004] The embodiments of the present application provide a road element detection method, device and storage medium, which can realize road element detection for monocular images and reduce the requirements for hardware.

[0005] In a first aspect, an embodiment of the present application provides a road element detection method, the method comprising: training an initial neural network model based on a sample image and modeling information of road elements in a pre-labeled sample image to obtain a modeling prediction model; inputting a target image of a road element to be detected into the modeling prediction model, and outputting modeling information of the road element in the target image; the sample image and the target image are monocular images; generating a fitting equation for the road element in a world coordinate system corresponding to the target image based on the modeling information of the road element in the target image and camera parameters; the camera parameters include intrinsic and extrinsic parameters of a camera that captures the sample image.

[0006] The technical solution provided by the embodiment of the present application brings at least the following beneficial effects: On the one hand, it can realize the training and application of the neural network model for the modeling information of the monocular image detection of road elements. When detecting 3D road elements, the input is the same as that of the traditional 2D road element detection, only the image input of the monocular camera is required, and no additional hardware support is required. On the other hand, replacing the logical algorithm with a deep learning neural network can further improve the accuracy and robustness of road element detection.

[0007] In some embodiments, the above method also includes: acquiring a sample image and point cloud data of the sample image; obtaining a mask image of the road element from the sample image; obtaining a key point set of the road element in the mask image based on the point cloud data corresponding to the road element in the mask image and prior information of the road element; and annotating the modeling information of the road element in the sample image for the key point set of the road element.

[0008] It should be understood that based on the mask image and point cloud data of the road elements, combined with prior information, a set of key points is determined, and then the modeling information of the road elements in the sample image is annotated for the key point set of the road elements. This can achieve a unified sample label format and improve the standardization and accuracy of the sample labeling method.

[0009] In some embodiments, the a priori information includes empirical data of height changes corresponding to road elements; the method for obtaining a set of key points of road elements in a mask image based on point cloud data corresponding to road elements in a mask image and a priori information of road elements includes: mapping the point cloud data corresponding to the road elements into the mask image to obtain point cloud coordinates of pixel points corresponding to the road elements in a pixel coordinate system; the pixel coordinate system includes a depth direction, a height direction and a width direction perpendicular to each other; determining a plurality of sampling depths at intervals of a preset depth along the depth direction; determining a plurality of sampling widths at intervals of a preset width along the width direction in each sampling depth; determining, based on the point cloud coordinates, height change data of a first pixel point corresponding to a first sampling width and a second pixel point corresponding to a second sampling width in a height direction in each sampling depth; the first sampling width and the second sampling width are any two adjacent sampling widths in the plurality of sampling widths; when the height change data is greater than or equal to the empirical data, determining a key point of the road element in each sampling depth based on the first pixel point and the second pixel point to obtain a set of key points of the road element; the height of the key point is the height of the first pixel point or the second pixel point, and the width is the average width of the first sampling width and the second sampling width.

[0010] Based on this, it is possible to accurately mark the key points of sudden height changes of road elements according to the height change data of adjacent pixels of road elements, so as to achieve accurate labeling of sample image labels.

[0011] In some embodiments, the above-mentioned height change data includes: the height difference between the first pixel point and the second pixel point, or the height change rate of the second pixel point relative to the first pixel point; when the height change data includes the height change rate, before determining the height change data in the height direction of the first pixel point corresponding to the first sampling width and the second pixel point corresponding to the second sampling width in each sampling depth according to the point cloud coordinates, the above-mentioned method also includes: mapping the height of the point cloud coordinates to a preset interval.

[0012] It should be understood that mapping the height of the point cloud coordinates to a preset interval, which can be limited according to actual applications, can effectively improve computing efficiency.

[0013] In some embodiments, the above method for obtaining a mask image of a road element from a sample image includes: inputting the sample image into an instance segmentation model to obtain a segmentation image of the road elements in the sample image; the instance segmentation model is trained based on the image of the road element; upsampling the segmentation image of the road element to an image pixel greater than or equal to a preset pixel to obtain a mask image of the road element.

[0014] Based on this, on the one hand, the instance segmentation model can improve the accuracy and robustness of image instance segmentation. On the other hand, upsampling the segmented image can improve the clarity of the mask image, and further improve the accuracy of the subsequent point cloud data projection on the mask image.

[0015] In some embodiments, the above modeling information includes: a key point set of road elements, and depth information of the key point set.

[0016] Based on this, it is possible to implement a model prediction of the key point set of a single-path element in a monocular image, as well as the depth information of the key point set.

[0017] In some embodiments, the above modeling information includes: a mask area of ​​the road element and depth information of the mask area; the mask area is the area after pixel expansion processing is performed on each key point in the key point set of the road element.

[0018] Based on this, it is possible to implement a model prediction of the mask area of ​​a single-path element in a monocular image, as well as the depth information of the mask area.

[0019] In a second aspect, an embodiment of the present application provides a road element detection device, comprising: a model training unit, used to train an initial neural network model based on a sample image and modeling information of road elements in a pre-labeled sample image to obtain a modeling prediction model; a model application unit, used to input a target image of a road element to be detected into the modeling prediction model, and output the modeling information of the road element in the target image; the sample image and the target image are monocular images; an equation fitting unit, used to generate a fitting equation of the road element in the world coordinate system corresponding to the target image based on the modeling information of the road elements in the target image and camera parameters; the camera parameters include the intrinsic parameters and extrinsic parameters of the camera that collects the sample image.

[0020] In some embodiments, the above-mentioned device also includes: a sample annotation unit, which is used to obtain a sample image and point cloud data of the sample image; obtain a mask image of the road element from the sample image; obtain a key point set of the road element in the mask image based on the point cloud data corresponding to the road element in the mask image and prior information of the road element; and annotate the modeling information of the road element in the sample image for the key point set of the road element.

[0021] In some embodiments, the a priori information includes empirical data on height changes corresponding to road elements; the sample annotation unit is specifically used to map the point cloud data corresponding to the road elements into the mask image to obtain the point cloud coordinates of the pixel points corresponding to the road elements in the pixel coordinate system; the pixel coordinate system includes a depth direction, a height direction and a width direction that are perpendicular to each other; a plurality of sampling depths are determined by spacing a preset depth along the depth direction; a plurality of sampling widths are determined by spacing a preset width along the width direction in each sampling depth of the plurality of sampling depths; according to the point cloud coordinates, the height change data of the first pixel point corresponding to the first sampling width and the second pixel point corresponding to the second sampling width in the height direction in each sampling depth are determined; the first sampling width and the second sampling width are any two adjacent sampling widths in the plurality of sampling widths; when the height change data is greater than or equal to the empirical data, the key points of the road elements in each sampling depth are determined according to the first pixel point and the second pixel point to obtain a set of key points of the road elements; the height of the key point is the height of the first pixel point or the second pixel point, and the width is the average width of the first sampling width and the second sampling width.

[0022] In some embodiments, the above-mentioned height change data includes: the height difference between the first pixel and the second pixel, or the height change rate of the second pixel relative to the first pixel; when the height change data includes the height change rate, the above-mentioned sample annotation unit is also used to map the height of the point cloud coordinates to a preset interval.

[0023] In some embodiments, the sample annotation unit is specifically used to input the sample image into an instance segmentation model to obtain a segmentation image of the road elements in the sample image; the instance segmentation model is trained based on the image of the road elements; the segmentation image of the road elements is upsampled to an image pixel greater than or equal to a preset pixel to obtain a mask image of the road elements.

[0024] In some embodiments, the above modeling information includes: a key point set of road elements, and depth information of the key point set.

[0025] In some embodiments, the above modeling information includes: the modeling information includes: the mask area of ​​the road element, and the depth information of the mask area; the mask area is the area after the pixel expansion processing is performed on each key point in the key point set of the road element.

[0026] In a third aspect, an embodiment of the present application provides a road element detection device, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; wherein, when the processor executes the computer instructions, the road element detection device executes the road element detection method as in the first aspect and any possible design method thereof.

[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes computer instructions. When the computer instructions are executed on a computer, the computer executes the method provided in the first aspect and possible implementations.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer instructions, which, when executed on a computer, enables the computer to execute the method provided in the first aspect and possible implementations described above.

[0029] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the controller, or may be packaged separately from the processor of the controller, which is not limited in this application.

[0030] The specific description of the second to fifth aspects and their various implementations in this application can refer to the detailed description of the first aspect and its various implementations. The beneficial effects of the second to fifth aspects and their various implementations can refer to the beneficial effect analysis of the first aspect and its various implementations, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A road element detection method according to some embodiments is shown in FIG. Figure 1 ;

[0032] Figure 2 A road element detection method according to some embodiments is shown in FIG. Figure 2 ;

[0033] Figure 3 A projection diagram of a laser point cloud projected onto a mask image according to some embodiments;

[0034] Figure 4 A road element detection method according to some embodiments is shown in FIG. Figure 3 ;

[0035] Figure 5 is a schematic diagram of sampling pixel points of road elements according to some embodiments;

[0036] Figure 6 A road element detection method according to some embodiments is shown in FIG. Figure 4 ;

[0037] Figure 7 A schematic diagram of an application of a modeling prediction model according to some embodiments Figure 1 ;

[0038] Figure 8 A schematic diagram of an application of a modeling prediction model according to some embodiments Figure 2 ;

[0039] Fig. 9 is a schematic diagram of a fitting equation according to some embodiments;

[0040] Fig.10 FIG. 1 is a schematic diagram of a road element detection device according to some embodiments. Figure 1 ;

[0041] Fig.11 FIG. 1 is a schematic diagram of a road element detection device according to some embodiments. Figure 2 . DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] It should be noted that, in the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "multiple" means two or more.

[0044] As described in the background technology, three-dimensional (3D) road element detection usually requires the collection of multi-eye images or laser point clouds of road elements, and the fitting of three-dimensional images of road elements through a stereo matching logic algorithm. However, since the collection of multi-eye images or laser point clouds requires the deployment of additional multi-eye cameras or laser radar systems, the general method at least has the technical difficulty of high hardware requirements, which is not conducive to the promotion of technology.

[0045] In this regard, an embodiment of the present application provides a road element detection method, the core idea of ​​which is: first, according to the sample image and the modeling information of the road elements in the pre-labeled sample image, an initial neural network model is trained to obtain a modeling prediction model. Subsequently, the target image of the road element to be detected is input into the modeling prediction model, and the modeling information of the road element in the target image is output. Among them, the sample image and the target image are monocular images. Then, according to the modeling information of the road elements in the target image and the camera parameters, a fitting equation of the road element in the world coordinate system corresponding to the target image is generated. Based on this, the training and application of the neural network model for the modeling information of the road elements detected by the monocular image can be realized. When detecting 3D road elements, the input is the same as that of the traditional two-dimensional (two dimensions, 2D) road element detection, only the image input of the monocular camera is required, and no additional hardware support is required. At the same time, replacing the logical algorithm with a deep learning neural network can further improve the accuracy and robustness of road element detection.

[0046] It should be noted that the embodiments of the present application do not limit the specific application scenarios of road element detection, and can be applied to vehicle-mounted front-view cameras, involving any road scenarios such as urban roads, highways, suburban roads, and parking lots. In addition, the road element detection method provided in the embodiments of the present application can be executed by a road element detection device. For example, the road element detection device can be a computer device. For another example, the road element detection device can be a server. For another example, the road element detection device may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof; for another example, the road element detection device may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, etc. The present application does not impose any special restrictions on the specific form of the road element detection device.

[0047] For ease of understanding, the road element detection method provided in this application is specifically introduced in conjunction with the accompanying drawings, taking the execution subject as a vehicle-mounted front-view camera as an example.

[0048] A road element detection method proposed in an embodiment of the present application is applied to the field of image analysis technology. The method includes: a labeling process of modeling information of road elements in sample images (hereinafter referred to as "sample label labeling process"), and a training and application process of a modeling prediction model (hereinafter referred to as "model training and use process").

[0049] Figure 1 A road element detection method proposed in the embodiment of the present application specifically relates to a sample labeling process. Figure 1 As shown, the sample labeling process includes the following steps S101 to S104:

[0050] S101, obtaining a sample image and point cloud data of the sample image.

[0051] In the embodiment of the present application, the sample image is a monocular image, such as a front-view image captured by a vehicle-mounted front-view camera.

[0052] In some embodiments, the point cloud data may be a laser point cloud collected by a laser radar. In order to ensure the matching accuracy between the point cloud data and the sample image, the timestamp of the point cloud data and the timestamp of the sample image may be obtained respectively. If the timestamps are the same, the point cloud data is considered to be the point cloud data of the sample image.

[0053] S102: Obtain a mask image of the road element from the sample image.

[0054] In some embodiments, the road element may be a curb, a flower bed curb, a cement wall, a railing, a water barrier, etc.

[0055] Furthermore, image instance segmentation aims to label each pixel of an image as an instance, and deep learning networks have greatly promoted the development of image instance segmentation technology. Therefore, the embodiment of the present application trains an end-to-end, pixel-to-pixel instance segmentation model based on a deep learning network. Figure 2 As shown, the above step S102 can be specifically implemented as the following steps S1021 to S1022:

[0056] S1021. Input the sample image into the instance segmentation model to obtain a segmented image of the road elements in the sample image.

[0057] In some embodiments, the instance segmentation model is trained based on images of road elements and can be used to extract preliminary features of road elements. Then, during forward prediction, deep features of the input sample image are extracted and compared with the preliminary features of the road elements, the areas in the sample image showing the road elements are screened out, and a segmented image of the road elements is output.

[0058] S1022: Up-sample the segmented image of the road element until the image pixels are greater than or equal to the preset pixels, to obtain a mask image of the road element.

[0059] It should be understood that the image pixels can be defined by at least one of the image size and the image resolution. The image pixels are greater than or equal to the preset pixels, which are used to define the clarity of the mask image to improve the accuracy of subsequent point cloud data projection. For example, when the image pixels are defined by the image size, when the preset pixels correspond to the original image size, a mask image with the same size as the original image can be obtained.

[0060] S103 . Obtain a set of key points of the road elements in the mask image according to the point cloud data corresponding to the road elements in the mask image and the prior information of the road elements.

[0061] In some embodiments, the embodiments of the present application can project a laser point cloud with the same timestamp as the sample image into a mask image using a matrix transformation from a point cloud coordinate system to a camera coordinate system and then to a pixel coordinate system to obtain point cloud data corresponding to road elements in the mask image.

[0062] For example, Figure 3 As shown, the projection of the laser point cloud onto the mask image of the curb can reflect the change in the point cloud height (height 1 < height 2 < height 3). Among them, part of the mask image of the curb will fall on the curb and part will fall on the road surface.

[0063] In some embodiments, the point cloud data may include the three-dimensional coordinates of each sampling point, and may also contain color information (RGB) or reflection intensity information (Intensity). Accordingly, the prior information may include empirical data on the shape, color, and reflection intensity of road elements.

[0064] Furthermore, when the point cloud data includes the three-dimensional coordinates of each sampling point and the prior information includes empirical data of the height change corresponding to the road element, such as Figure 4 As shown, the above step S103 can be specifically implemented as the following steps S1031 to S1035:

[0065] S1031. Map the point cloud data corresponding to the road element into the mask image to obtain the point cloud coordinates of the pixel points corresponding to the road element in the pixel coordinate system.

[0066] It should be understood that the pixel coordinate system (xyz) includes a depth direction (y direction), a height direction (z direction), and a width direction (x direction) which are perpendicular to each other.

[0067] S1032 , determining a plurality of sampling depths at intervals of preset depths along a depth direction.

[0068] S1033 . In each of the plurality of sampling depths, determine a plurality of sampling widths by spacing them by a preset width in a width direction.

[0069] S1034. Determine, according to the point cloud coordinates, height change data in the height direction of a first pixel point corresponding to the first sampling width and a second pixel point corresponding to the second sampling width in each sampling depth.

[0070] The first sampling width and the second sampling width are any two adjacent sampling widths among the plurality of sampling widths.

[0071] For example, Figure 5As shown on the left side of the figure, after obtaining the point cloud data in the mask image of the curb, multiple curb point cloud areas corresponding to multiple sampling depths are determined at preset depth intervals along the y direction, and an outward traversal in the x direction is performed under each area (left curb to the left, right curb to the right).

[0072] Then, if Figure 5 As shown on the right side, at the sampling depth y j In the process of traversing outward in the x direction, multiple pixel points corresponding to multiple sampling widths are determined at intervals of preset width to obtain the xz coordinate set {(x1,z1),(x2,z2),……,(x n ,z n )}. Then, according to the height of each pixel (i.e., z value), the height change data of any two adjacent pixels corresponding to the sampling width in the height direction is calculated. j ∈[y near ,y far ],y near is the nearest depth in the mask image, y far is the farthest depth in the mask image.

[0073] In some embodiments, the height change data is the height difference between the first pixel point (x1, z1) and the second pixel point (x2, z2), that is, (z2-z1).

[0074] In other embodiments, the height change data is the height change rate of the second pixel point (x1, z1) relative to the first pixel point (x2, z2), that is, (z2-z1) / z1.

[0075] Furthermore, when the height change data is a height change rate, the embodiment of the present application may map the height of the point cloud coordinates to a preset interval before step S1034. The preset interval may be limited according to actual applications to improve calculation efficiency.

[0076] It should be understood that the linear change of height is mapped to the preset interval, and the minimum height in the area can be calculated as z min , and subtract z from the height of all pixels in the area min And add the minimum value of the preset interval. For example, if the coordinates of the original point cloud are (x, y, z), the linear change f mapped to the interval with the minimum value of 1 satisfies the following formula:

[0077] f(x,y,z)=(x,y,z–z min +1).

[0078] S1035. When the height change data is greater than or equal to the empirical data, determine the key points of the road element in each sampling depth according to the first pixel point and the second pixel point to obtain a key point set of the road element.

[0079] The height of the key point is the height of the first pixel point or the second pixel point, and the width is the average width of the first sampling width and the second sampling width.

[0080] Exemplarily, when the road element is a curb, Figure 5 As shown on the right side of the figure, when the height change data is greater than or equal to the empirical data, it can be considered that the first pixel point (x1, z1) corresponds to the road surface, and the second pixel point (x2, z2) corresponds to the curb.

[0081] The empirical data of the preset curb height is 10 cm. If z2-z1≥0.1, it can be marked as a height mutation point of the curb (i.e., a key point), and its coordinates are T(v)=((x1+x2) / 2,y j ,z1). Repeating the above operation can obtain any sampling depth y among t sampling depths. j Multiple key points P(V j ), the specific expansion formula is as follows:

[0082]

[0083] i=(1,2,…,n)and x i ∈[x start ,x end ],j=(1,2,…,t)and y j ∈[y near ,y far ].

[0084] As another example, the empirical data of the preset curb height change rate is 1.1 times. If (z2-z1) / z1≥1.1, it can be marked as a height mutation point (i.e., a key point) of the curb, and its coordinates are T(v)=((x1+x2) / 2,y j ,z1). Repeating the above operation can obtain any sampling depth y among t sampling depths. j Multiple key points P(V j ), the specific expansion formula is as follows:

[0085]

[0086] i=(1,2,…,n)and x i ∈[x start ,x end ],j=(1,2,…,t)and yj ∈[y near ,y far ].

[0087] Furthermore, by combining any of the above examples, we can obtain the key point set S(V j ), that is, there exists the following formula:

[0088] S(V j )=Set[P(V j )],j=(1,2,…,t)and y j ∈[y near ,y far ].

[0089] S104: labeling modeling information of road elements in the sample image according to the key point set for the road elements.

[0090] The embodiments of the present application can generate modeling information in a variety of ways, such as directly using key point detection (also known as key point regression), or mask and deep multi-task modeling.

[0091] In some embodiments, key point regression can directly perform model training and application of key point sets. In this case, the modeling information includes: key point sets of road elements and depth information of the key point sets.

[0092] In other embodiments, the depth and instance masking method is to project into the pixel coordinate system, and perform pixel dilation processing on each key point in the key point set of the road element, such as covering the area of ​​the curb wall, thereby expanding the mask area and improving the pixel detection of the road element. In this case, the modeling information includes: the mask area of ​​the road element, and the depth information of the mask area. The mask area is the area after the pixel dilation processing is performed on each key point in the key point set of the road element.

[0093] It should be understood that Figure 1 The provided technical solution brings at least the following beneficial effects: based on the mask image and point cloud data of the road elements, combined with prior information, a set of key points is determined, and then the modeling information of the road elements in the sample image is annotated for the key point set of the road elements, which can achieve a unified sample label format and improve the standardization and accuracy of the sample labeling method.

[0094] Figure 6 A road element detection method proposed in the embodiment of the present application specifically involves model training and use process. Figure 6 As shown, the model training and use process includes the following steps S201 to S203:

[0095] S201: Based on the sample image and the modeling information of the road elements in the pre-labeled sample image, an initial neural network model is trained to obtain a modeling prediction model.

[0096] It should be understood that by inputting sample images and modeling information of road elements in labeled sample images into the initial neural network model and training the initial neural network model, a modeling prediction model for predicting modeling information of road elements can be obtained.

[0097] In some embodiments, when the modeling information includes: a key point set of road elements, and depth information of the key point set, the modeling prediction model is a key point regression and deep multi-task network model.

[0098] In other embodiments, when the modeling information includes: a mask area of ​​a road element, and depth information of the mask area, the modeling prediction model is an instance segmentation and deep multi-task network model.

[0099] S202: Input a target image of the road element to be detected into a modeling prediction model, and output modeling information of the road element in the target image.

[0100] The target image is a monocular image.

[0101] In some embodiments, Figure 7 As shown, the target image is input into the key point regression and deep multi-task network model, which can output the key point set and depth information of the road elements in the target image (the darker the color, the greater the depth).

[0102] In other embodiments, Figure 8 As shown, the target image is input into the instance segmentation and deep multi-task network model, and the mask area and depth information of the road elements in the target image can be output (the darker the color, the greater the depth).

[0103] S203: Generate a fitting equation of the road element in the world coordinate system corresponding to the target image according to the modeling information of the road element in the target image and the camera parameters.

[0104] In some embodiments, the camera parameters include the intrinsic parameters and extrinsic parameters of the camera that collects the sample image. The key point coordinates or the edge point coordinates of the mask area in the modeling information are coordinates in the pixel coordinate system, which can be converted to the camera coordinate system through the camera's intrinsic parameters and distortion coefficients, and then further converted to the world coordinate system through the camera's extrinsic parameters. Then, the coordinates of the road elements in the world coordinate system are fitted with polynomial equations using the least squares method.

[0105] For example, Fig. 9As shown, when the model predicts, it outputs a set of key points or a mask area, and depth information, and combines the camera parameters of the camera to perform a conversion from the pixel coordinate system to the world coordinate system, and then uses the least squares method in the world coordinate system to generate two cubic polynomials of depth and height (yz) and depth and width (yx).

[0106] It should be understood that Figure 6 The technical solution provided brings at least the following beneficial effects: it can realize the training and application of the neural network model for modeling information of road element detection based on monocular image. When detecting 3D road elements, the input is the same as that of traditional 2D road element detection, only the image input of the monocular camera is needed, and no additional hardware support is required. At the same time, replacing the logical algorithm with a deep learning neural network can further improve the accuracy and robustness of road element detection.

[0107] It can be seen that the above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, the embodiment of the present application provides a hardware structure and / or software module corresponding to each function. It should be easily appreciated by those skilled in the art that, in combination with the modules and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0108] like Fig.10 As shown, the embodiment of the present application provides a road element detection device for performing Figure 1 and Figure 6 The road element detection method shown. The road element detection device 300 includes:

[0109] The model training unit 301 is used to train the initial neural network model according to the sample image and the modeling information of the road elements in the pre-labeled sample image to obtain the modeling prediction model; the model application unit 302 is used to input the target image of the road element to be detected into the modeling prediction model and output the modeling information of the road element in the target image; the sample image and the target image are monocular images; the equation fitting unit 303 is used to generate the fitting equation of the road element in the world coordinate system corresponding to the target image according to the modeling information of the road element in the target image and the camera parameters.

[0110] In some embodiments, the above-mentioned road element detection device 300 also includes: a sample annotation unit 304, which is used to obtain a sample image and point cloud data of the sample image; obtain a mask image of the road element from the sample image; obtain a key point set of the road element in the mask image based on the point cloud data corresponding to the road element in the mask image and the prior information of the road element; and annotate the modeling information of the road element in the sample image for the key point set of the road element.

[0111] In some embodiments, the a priori information includes empirical data on height changes corresponding to road elements; the sample annotation unit 304 is specifically used to map the point cloud data corresponding to the road elements to the mask image to obtain the point cloud coordinates of the pixel points corresponding to the road elements in the pixel coordinate system; the pixel coordinate system includes a depth direction, a height direction and a width direction that are perpendicular to each other; a plurality of sampling depths are determined by spacing a preset depth along the depth direction; in each of the plurality of sampling depths, a plurality of sampling widths are determined by spacing a preset width along the width direction; according to the point cloud coordinates, the height change data of the first pixel point corresponding to the first sampling width and the second pixel point corresponding to the second sampling width in the height direction are determined in each sampling depth; the first sampling width and the second sampling width are any two adjacent sampling widths in the plurality of sampling widths; when the height change data is greater than or equal to the empirical data, the key points of the road elements in each sampling depth are determined according to the first pixel point and the second pixel point to obtain a set of key points of the road elements; the height of the key point is the height of the first pixel point or the second pixel point, and the width is the average width of the first sampling width and the second sampling width.

[0112] In some embodiments, the height change data includes: the height difference between the first pixel and the second pixel, or the height change rate of the second pixel relative to the first pixel; when the height change data includes the height change rate, the sample annotation unit 304 is also used to map the height of the point cloud coordinates to a preset interval.

[0113] In some embodiments, the sample annotation unit 304 is specifically used to input the sample image into an instance segmentation model to obtain a segmentation image of the road elements in the sample image; the instance segmentation model is trained based on the image of the road elements; the segmentation image of the road elements is upsampled to an image pixel greater than or equal to a preset pixel to obtain a mask image of the road elements.

[0114] It should be noted that Fig.10 The division of modules in the example is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or software function modules.

[0115] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the embodiment of the present application provides another possible structural diagram of the road element detection device involved in the above-mentioned embodiment. Fig.11 As shown, the road element detection device 400 includes: a processor 402 and a bus 404. Optionally, the road element detection device may further include a memory 401; optionally, the road element detection device may further include a communication interface 403.

[0116] The processor 402 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor 402 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor 402 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0117] The communication interface 403 is used to connect with other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0118] The memory 401 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0119] As a possible implementation, the memory 401 may exist independently of the processor 402, and the memory 401 may be connected to the processor 402 via a bus 404 to store instructions or program codes. When the processor 402 calls and executes the instructions or program codes stored in the memory 401, the road element detection method provided in the embodiment of the present application can be implemented.

[0120] In another possible implementation, the memory 401 may also be integrated with the processor 402 .

[0121] The bus 404 may be an extended industry standard architecture (EISA) bus, etc. The bus 404 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0122] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the road element detection device is divided into different functional modules to complete all or part of the functions described above.

[0123] The embodiment of the present application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be completed by computer instructions to instruct the relevant hardware, and the program can be stored in the above computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be the memory or memory of any of the above embodiments. The above computer-readable storage medium can also be the above external storage device based on the road element detection device, such as the plug-in hard disk equipped on the above road element detection device, a smart memory card (smartmedia card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. Further, the above computer-readable storage medium can also include both the internal storage unit of the above road element detection device and an external storage device. The above computer-readable storage medium is used to store the above computer program and other programs and data required by the above road element detection device. The above computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0124] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program product is run on a computer, the computer is enabled to execute any one of the road element detection methods provided in the above embodiments.

[0125] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other changes to the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0126] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0127] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A road element detection method, characterized in that: include: According to the sample image and the modeling information of the road elements in the sample image that are pre-labeled, an initial neural network model is trained to obtain a modeling prediction model; Inputting a target image of the road element to be detected into the modeling prediction model, and outputting modeling information of the road element in the target image; the sample image and the target image are monocular images; A fitting equation of the road element in the world coordinate system corresponding to the target image is generated according to the modeling information of the road element in the target image and camera parameters; the camera parameters include intrinsic parameters and extrinsic parameters of the camera that collects the sample image.

2. The method according to claim 1, characterized in that Also includes: Acquire the sample image and point cloud data of the sample image; Obtaining a mask image of the road element from the sample image; Obtaining a key point set of the road element in the mask image according to the point cloud data corresponding to the road element in the mask image and the prior information of the road element; For the key point set of the road element, the modeling information of the road element in the sample image is annotated.

3. The method according to claim 2, characterized in that The prior information includes empirical data of height changes corresponding to the road elements; The step of obtaining a key point set of the road element in the mask image according to the point cloud data corresponding to the road element in the mask image and the prior information of the road element comprises: Mapping the point cloud data corresponding to the road element to the mask image to obtain the point cloud coordinates of the pixel points corresponding to the road element in a pixel coordinate system; the pixel coordinate system includes a depth direction, a height direction and a width direction that are perpendicular to each other; Determine a plurality of sampling depths at intervals of preset depths along the depth direction; In each of the plurality of sampling depths, a preset width is spaced along the width direction to determine a plurality of sampling widths; Determine, according to the point cloud coordinates, height variation data of a first pixel point corresponding to a first sampling width and a second pixel point corresponding to a second sampling width in the height direction at each sampling depth; the first sampling width and the second sampling width are any two adjacent sampling widths among the multiple sampling widths; When the height change data is greater than or equal to the empirical data, the key points of the road element in each sampling depth are determined according to the first pixel point and the second pixel point to obtain a key point set of the road element; the height of the key point is the height of the first pixel point or the second pixel point, and the width is the average width of the first sampling width and the second sampling width.

4. The method according to claim 3, characterized in that The height change data includes: a height difference between the first pixel and the second pixel, or a height change rate of the second pixel relative to the first pixel; When the height change data includes the height change rate, determining, according to the point cloud coordinates, in each sampling depth, a first pixel point corresponding to the first sampling width and a second pixel point corresponding to the second sampling width before the height change data in the height direction further includes: The height of the point cloud coordinates is mapped to a preset interval.

5. The method according to claim 2, characterized in that: The step of obtaining the mask image of the road element from the sample image comprises: Inputting the sample image into an instance segmentation model to obtain a segmented image of the road element in the sample image; the instance segmentation model is trained based on the image of the road element; The segmented image of the road element is up-sampled until the image pixels are greater than or equal to the preset pixels, so as to obtain the mask image of the road element.

6. The method according to any one of claims 1 to 5, characterized in that: The modeling information includes: a key point set of the road element, and depth information of the key point set.

7. The method according to any one of claims 1 to 5, characterized in that: The modeling information includes: a mask area of ​​the road element and depth information of the mask area; the mask area is an area after pixel expansion processing is performed on each key point in a key point set of the road element.

8. A road element detection device, characterized in that: include: A model training unit, used to train an initial neural network model based on a sample image and modeling information of road elements in the sample image that is pre-labeled, to obtain a modeling prediction model; A model application unit, used for inputting a target image of the road element to be detected into the modeling prediction model, and outputting modeling information of the road element in the target image; the sample image and the target image are monocular images; An equation fitting unit is used to generate a fitting equation of the road element in the world coordinate system corresponding to the target image according to the modeling information of the road element in the target image and camera parameters; the camera parameters include intrinsic parameters and extrinsic parameters of the camera that collects the sample image.

9. The device according to claim 8, characterized in that Also includes: A sample annotation unit, used to obtain the sample image and point cloud data of the sample image; Obtaining a mask image of the road element from the sample image; Obtaining a key point set of the road element in the mask image according to the point cloud data corresponding to the road element in the mask image and the prior information of the road element; and marking the modeling information of the road element in the sample image for the key point set of the road element; The prior information includes empirical data of height variation corresponding to the road element; the sample annotation unit is specifically used to map the point cloud data corresponding to the road element to the mask image to obtain the point cloud coordinates of the pixel points corresponding to the road element in the pixel coordinate system; the pixel coordinate system includes a depth direction, a height direction and a width direction which are perpendicular to each other; a plurality of sampling depths are determined by spacing a preset depth along the depth direction; a plurality of sampling widths are determined by spacing a preset width along the width direction in each sampling depth of the plurality of sampling depths; according to the point cloud coordinates, the height variation data of the first pixel point corresponding to the first sampling width and the second pixel point corresponding to the second sampling width in the height direction in each sampling depth are determined; the first sampling width and the second sampling width are any two adjacent sampling widths in the plurality of sampling widths; when the height variation data is greater than or equal to the empirical data, the key points of the road element in each sampling depth are determined according to the first pixel point and the second pixel point to obtain the key point set of the road element; the height of the key point is the height of the first pixel point or the second pixel point, and the width is the average width of the first sampling width and the second sampling width; The height change data includes: a height difference between the first pixel and the second pixel, or a height change rate of the second pixel relative to the first pixel; when the height change data includes the height change rate, the sample annotation unit is further used to map the height of the point cloud coordinates to a preset interval; The sample annotation unit is specifically used to input the sample image into an instance segmentation model to obtain a segmentation image of the road element in the sample image; the instance segmentation model is trained based on the image of the road element; the segmentation image of the road element is upsampled to an image pixel greater than or equal to a preset pixel to obtain a mask image of the road element; The modeling information includes: a key point set of the road element and depth information of the key point set; or, the modeling information includes: a mask area of ​​the road element and depth information of the mask area; the mask area is an area after pixel expansion processing is performed on each key point in the key point set of the road element.

10. A road element detection device, characterized in that: include: A memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code includes computer instructions; When the processor executes the computer instruction, the road element detection device executes the road element detection method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions; Wherein, when the computer instructions are executed on the road element detection device, the road element detection device executes the road element detection method as claimed in any one of claims 1 to 7.

Images