Road slope preview method and device, vehicle-mounted equipment, storage medium and program product

Abstract

The invention relates to a road gradient preview method and device, vehicle-mounted equipment, a storage medium and a program product. The method comprises the following steps: inputting image data of a road in front of a vehicle to a road line detection model for road line detection to obtain lane line point location information, determining an actual vanishing point of the road in front of the vehicle according to the lane line point location information, and determining a preview road slope value of the road in front of the vehicle according to the actual vanishing point, the road line detection model can perform detection based on captured multi-scale context information and long-distance spatial correlation information, can enhance the global perception ability of deep features, further effectively makes up for the problem of insufficient spatial resolution of high-level features, can improve the accuracy of road line detection by the road line detection model, and improves the accuracy of road line detection. And meanwhile, the accuracy of road slope preview based on the road line detection result in the later period is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a road slope preview method, device, vehicle-mounted equipment, storage medium and program product. Background Art

[0002] With the development of intelligent vehicle technology, intelligent chassis technology has emerged. Leveraging onboard sensors, electronic control systems, and intelligent algorithms, it enables vehicles to perceive road conditions in real time, adapt to driver habits, and even meet the personalized needs of passengers, significantly enhancing vehicle intelligence. For example, intelligent vehicles can use intelligent chassis technology to predict the slope ahead. This prediction provides critical data support for the intelligent chassis, improving vehicle performance and safety in complex road conditions.

[0003] In related technologies, the road slope preview method mainly relies on the vehicle dynamics model for real-time calculation. However, this method has the problem of inaccurate slope preview. Summary of the Invention

[0004] Based on this, it is necessary to provide a road slope preview method, device, vehicle-mounted equipment, storage medium and program product that can improve the accuracy of slope preview in response to the above technical problems.

[0005] In a first aspect, the present application provides a road slope preview method, comprising:

[0006] Inputting image data of the road ahead of the vehicle into a road line detection model to perform road line detection to obtain lane line point information; the road line detection model is configured to perform detection based on multi-scale contextual information and / or long-range spatial correlation information of lane lines captured from the image data;

[0007] An actual shadow vanishing point of the road in front of the vehicle is determined according to the lane line point information, and a preview road slope value of the road in front of the vehicle is determined according to the actual shadow vanishing point.

[0008] In some embodiments, the road line detection model includes a backbone network, a feature enhancement network, and a detection network; inputting image data of the road in front of the vehicle into the road line detection model to perform road line detection to obtain lane line point information includes:

[0009] Inputting image data of the road ahead of the vehicle into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the final level;

[0010] Inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial correlation features of lane lines;

[0011] The enhanced features and the second features of the multiple intermediate levels are input into the detection network for feature fusion and detection to obtain the lane line point information.

[0012] In some embodiments, the feature enhancement network includes a pyramid pooling subnetwork and an attention mechanism subnetwork, and inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features includes:

[0013] Inputting the third feature into the pyramid pooling subnetwork to capture multi-scale context information to obtain the multi-scale context feature;

[0014] The third feature is input into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line to obtain the long-distance spatial correlation feature of the lane line.

[0015] In some embodiments, the detection network includes a first feature pyramid, a second feature pyramid, and a detection module. Inputting the enhanced features and the plurality of second features at the intermediate levels into the detection network for feature fusion and detection to obtain the lane point information includes:

[0016] Inputting the multi-scale contextual features and the plurality of second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features;

[0017] Inputting the plurality of first fused features and the long-distance spatial correlation features of the lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features;

[0018] The multiple second fusion features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0019] In some embodiments, the detection network further includes an information collection module, and the method further includes:

[0020] Inputting the plurality of second fusion features into the information collection module for feature aggregation to obtain aggregated features;

[0021] Inputting the plurality of second fused features into the detection module to perform lane line detection to obtain the lane line point information includes:

[0022] The aggregated features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0023] In some embodiments, the method further comprises:

[0024] Acquire sample image data; the sample image data includes image data of road lines in various scenarios;

[0025] An initial road line detection model is trained according to the sample image data to obtain the road line detection model.

[0026] In some embodiments, determining the actual shadow disappearance point of the road in front of the vehicle based on the lane line point information includes:

[0027] Determining a target lane line of the road in front of the vehicle based on the lane line point information; the target lane line is a lane line within the peripheral area of ​​the vehicle;

[0028] The target lane line is fitted, and the actual shadow vanishing point is determined based on the fitted lane line.

[0029] In some embodiments, the method further comprises:

[0030] Performing point interception processing on the target lane line to obtain an intercepted target lane line;

[0031] The fitting process for the target lane line and determining the actual shadow vanishing point based on the fitted lane line includes:

[0032] The intercepted target lane line is fitted, and the actual shadow vanishing point is determined based on the lane line generated after fitting.

[0033] In some embodiments, determining the preview road slope value of the road in front of the vehicle based on the actual shadow disappearance point includes:

[0034] Analyzing the image data to determine a theoretical vanishing point in the image data;

[0035] Performing camera coordinate transformation on the theoretical vanishing point and the actual vanishing point to obtain transformed theoretical vanishing point and actual vanishing point;

[0036] Based on the pinhole imaging principle, the preview road slope value of the road in front of the vehicle is determined according to the converted theoretical shadow vanishing point and the actual shadow vanishing point.

[0037] In some embodiments, the method of determining the preview road slope value of the road in front of the vehicle based on the converted theoretical vanishing point and the actual vanishing point based on the pinhole imaging principle includes:

[0038] Based on the pinhole imaging principle, determining the initial preview road slope value of the road in front of the vehicle according to the converted theoretical shadow vanishing point and the actual shadow vanishing point;

[0039] The current pitch angle information of the vehicle is obtained, and the initial preview road slope value is compensated according to the pitch angle information to obtain the preview road slope value.

[0040] In a second aspect, the present application further provides a road slope preview device, comprising:

[0041] a detection module configured to input image data of the road ahead of the vehicle into a road line detection model to perform road line detection and obtain lane line point information; the road line detection model is configured to perform detection based on captured multi-scale contextual information and / or long-range spatial correlation information of lane lines;

[0042] A determination module is used to determine the actual shadow disappearance point of the road in front of the vehicle based on the lane line point information, and to determine the preview road slope value of the road in front of the vehicle based on the actual shadow disappearance point.

[0043] In a third aspect, the present application also provides a vehicle-mounted device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the road slope preview method described in the first aspect is implemented.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the road slope preview method described in the first aspect above.

[0045] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the road slope preview method described in the first aspect above.

[0046] The above-mentioned road slope preview method, device, vehicle-mounted equipment, storage medium and program product input the image data of the road in front of the vehicle into the road line detection model to perform road line detection, obtain lane line point information, and determine the actual shadow disappearance point of the road in front of the vehicle based on the lane line point information, and determine the preview road slope value of the road in front of the vehicle based on the actual shadow disappearance point. On the one hand, the road line detection model can perform detection based on the captured multi-scale context information, which can enhance the global perception ability of deep features, thereby effectively compensating for the problem of insufficient spatial resolution of high-level features, and can improve the accuracy of the road line detection model in detecting road lines, and at the same time improve the accuracy of road slope preview based on the road line detection results in the later stage; on the other hand, the road line detection model can also perform detection based on the captured long-distance spatial correlation information of the lane lines, which can enhance the contextual interaction between features, improve the perception ability of the global structure of lane lines in complex scenes, thereby improving the accuracy of road line detection of the vehicle in complex environments, and at the same time improve the accuracy of road slope preview based on the road line detection results in the later stage. In addition, the above method performs road slope preview based on the shadow vanishing point, which can improve the accuracy of obtaining the preview road slope value. Compared with the complex method of using the vehicle dynamics model for slope preview in the related technology, the above method is simple and easy to implement, and can also improve the efficiency of slope preview. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is an application environment diagram of the road slope preview method in an embodiment of the present application;

[0049] Figure 2 This is one of the flow charts of the road slope preview method in the embodiment of the present application;

[0050] Figure 3 This is the second flow chart of the road slope preview method in the embodiment of the present application;

[0051] Figure 3A This is one of the structural diagrams of the road line detection model in the embodiment of this application;

[0052] Figure 4 This is the third flow chart of the road slope preview method in the embodiment of the present application;

[0053] Figure 4AThis is the second structural diagram of the road line detection model in the embodiment of the present application;

[0054] Figure 5 This is a fourth flow chart of the road slope preview method in an embodiment of the present application;

[0055] Figure 5A This is the third structural diagram of the road line detection model in the embodiment of this application;

[0056] Figure 5B This is the fourth structural diagram of the road line detection model in the embodiment of the present application;

[0057] Figure 6 This is the fifth flow chart of the road slope preview method in the embodiment of the present application;

[0058] Figure 7 This is the sixth flow chart of the road slope preview method in the embodiment of the present application;

[0059] Figure 7A This is a schematic diagram of lane lines in an embodiment of the present application;

[0060] Figure 8 This is the seventh flow chart of the road slope preview method in the embodiment of the present application;

[0061] Figure 8A Schematic diagram of the positions of the actual shadow vanishing point and the theoretical shadow vanishing point in the embodiment of the present application;

[0062] Figure 9 This is the eighth flow chart of the road slope preview method in the embodiment of the present application;

[0063] Figure 10 This is a ninth flowchart of the road slope preview method according to an embodiment of the present application;

[0064] Figure 11 This is a schematic diagram of the structure of the road slope preview device in an embodiment of the present application;

[0065] Figure 12 This is a diagram of the internal structure of the vehicle-mounted equipment in an embodiment of the present application. DETAILED DESCRIPTION

[0066] To facilitate understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.

[0067] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, they do not indicate any order, quantity or importance, but are simply used to distinguish different components. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. "Include" or "comprising" and similar words mean that the elements or objects that appear before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0069] The road slope preview method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the preview device 102 is installed on a vehicle 104. The preview device 102 can interact with the sensor system, electronic control system, etc. on the vehicle 104 and perceive and analyze the conditions of the road or environment on which the vehicle 104 is located based on various sensory data or other data. In this embodiment of the present application, the preview device 102 can preview the slope of the road on which the vehicle 104 is traveling while driving, thereby determining the preview road slope value of the road on which the vehicle 104 is traveling in real time. The preview device 102 can be an onboard device on the vehicle 104. The preview device 102 can also be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, projectors, etc. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0070] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the preview device to which the solution of the present application is applied. The specific preview device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0071] In some exemplary embodiments, Figure 2 As shown, a road slope preview method is provided, which is applied to Figure 1 The preview device 102 in FIG. 1 is used as an example to illustrate the method, which includes the following steps, wherein:

[0072] S201: Inputting image data of the road in front of the vehicle into a road line detection model to perform road line detection to obtain lane line point information.

[0073] Among them, the road line detection model can be a neural network or a machine learning model, which can be pre-trained based on road image sample data in various environments. The road line detection model is used to identify or detect lane lines on the road. The road line detection model in the embodiment of the present application can capture multi-scale context information and / or long-distance spatial correlation information of lane lines, and perform detection based on the captured multi-scale context information and / or long-distance spatial correlation information of lane lines. The lane line point information includes multiple lane line spatial coordinate sequences, each lane line spatial coordinate sequence represents the corresponding lane line, for example, each lane line spatial coordinate sequence: {(x1,y1),(x2,y2),...,(x n ,y n )}, where each coordinate in the sequence represents a lane line point.

[0074] In an embodiment of the present application, the preview device can obtain in real time image data of the road in front of the vehicle collected by an image acquisition device (e.g., a monocular camera, a front-facing camera, a driving recorder, etc.) during the vehicle's driving process. When the vehicle is driving on an uphill or downhill road, the road slope preview function can be turned on. Based on this, the preview device can obtain a trained road line detection model and obtain image data of the road in front of the current vehicle, and input the image data of the road in front of the vehicle into the road line detection model. The first scenario is that the road line detection model can capture multi-scale contextual information from the image data during the detection process, and perform road line detection based on the multi-scale contextual information to obtain lane line point information. The multi-scale contextual information is used to enhance the joint perception of local details and global semantics of lane lines through feature fusion of different spatial granularities, thereby coping with scale changes, occlusions, and lighting interference in complex road scenes. Optionally, the multi-scale contextual information includes small-scale features (fine-grained local information), medium-scale features and large-scale features (global semantic information), wherein the small-scale features include lane line edge information, color and texture information, etc., for example, the gap between dashed line segments, transition information between solid lines and dashed lines, yellow solid lines, white dashed lines, etc.; the medium-scale features may include local curvature change information of lane lines, the positional relationship between lane lines and surrounding road elements, for example, the curvature of turning lane lines, the relative position of lane lines and isolation belts, etc.; the large-scale features may include the shape (direction) of lane lines, or the topological structure of the road, for example, the lane lines are straight lines, S-shaped curve lines, one-way lanes, lanes at forked intersections, etc.

[0075] The second scenario is that the road line detection model can capture the long-distance spatial correlation information of the lane line during the detection process, and perform road line detection based on the long-distance spatial correlation information of the lane line to obtain lane line point information; wherein, the long-distance spatial correlation information is used to represent the logical relationship between lane line areas that are far apart, enhance the contextual interaction between features, and improve the perception of the global structure of lane lines in complex scenes; the long-distance spatial correlation information may include key information that ensures the spatial, semantic and topological continuity of the lane line, wherein the key information that ensures the spatial continuity of the lane line may include spatial continuity features, geometric feature transformation information, etc., for example, the curvature gradient information of the lane line, the slope continuity information of the lane line, and the spacing information between lane lines; the key information that ensures the semantic continuity of the lane line may include the type of the same lane line, the semantic relationship between the lane line and its surrounding environment (such as shoulders, medians, traffic signs), etc., for example, the type of the proximal lane line and the type of the distal lane line. Key information required to ensure lane topological continuity includes lane parallelism and lane orientation, such as the parallel relationship between multiple lanes, lane bifurcation information, lane merging information, and lane turning information. In a third scenario, the road detection model can simultaneously capture multi-scale contextual information and long-range spatial correlation information about lanes during detection. This information is used to detect lane points and determine lane position information.

[0076] S202: determining an actual shadow vanishing point of the road in front of the vehicle based on the lane line position information, and determining a preview road slope value of the road in front of the vehicle based on the actual shadow vanishing point.

[0077] The actual vanishing point is the point where parallel lines on the road converge in the distance, also known as the vanishing point. This is typically where the lane lines on either side of the vehicle converge in the distance. The preview road slope value is a key parameter used in intelligent vehicle control systems to predict the slope of the road ahead. It directly impacts vehicle power distribution, energy optimization, and driving safety.

[0078] In an embodiment of the present application, when the preview device obtains the lane line point information of the road in front of the vehicle based on the aforementioned steps, since the lane line point information includes multiple lane line spatial coordinate sequences, the preview device can first filter out the lane lines on both sides of the vehicle from the multiple lane line spatial coordinate sequences, and determine the actual shadow vanishing point of the road in front of the vehicle based on the lane lines on both sides of the vehicle. Since the actual shadow vanishing point is the point where the lane lines on both sides of the vehicle converge in the distance, the road slope can be analyzed by analyzing the distant convergence point. Therefore, when calculating the actual shadow vanishing point, the preview device can combine the horizon of the road and the position of the actual shadow vanishing point to predict the preview road slope value of the road in front of the vehicle. For example, the offset of the actual shadow vanishing point relative to the horizon can be calculated, and then the preview road slope value is determined based on the offset.

[0079] The road slope preview method described in the above embodiment inputs the image data of the road in front of the vehicle into the road line detection model to perform road line detection, obtain lane line point information, and determine the actual shadow disappearance point of the road in front of the vehicle based on the lane line point information, and determine the preview road slope value of the road in front of the vehicle based on the actual shadow disappearance point. On the one hand, the road line detection model can perform detection based on the captured multi-scale context information, which can enhance the global perception ability of deep features, thereby effectively compensating for the problem of insufficient spatial resolution of high-level features, and can improve the accuracy of the road line detection model in detecting road lines, and at the same time improve the accuracy of road slope preview based on the road line detection results in the later stage; on the other hand, the road line detection model can also perform detection based on the long-distance spatial correlation information of the captured lane lines, which can enhance the contextual interaction between features, improve the perception ability of the global structure of lane lines in complex scenes, thereby improving the accuracy of road line detection of vehicles in complex environments, and at the same time improve the accuracy of road slope preview based on the road line detection results in the later stage. In addition, the above method performs road slope preview based on the shadow vanishing point, which can improve the accuracy of obtaining the preview road slope value. Compared with the complex method of using the vehicle dynamics model for slope preview in the related technology, the above method is simple and easy to implement, which can improve the efficiency of slope preview.

[0080] In some exemplary embodiments, a type of road line detection model is provided, and a method for lane line detection based on the type of road line detection model is provided, such as Figure 3 As shown, the method includes:

[0081] S301: Input image data of the road in front of the vehicle into the backbone network for multi-scale feature extraction to obtain first features of multiple levels.

[0082] Among them, the first features of multiple levels include the second features of multiple intermediate levels and the third features of the last level. The backbone network is a pre-trained network (Backbone), which is responsible for extracting features of images or other data. It can usually be a convolutional neural network CNN or a residual neural network ResNet, etc. In the embodiment of the present application, the backbone network is the core feature extraction module in the road line detection model, which is responsible for capturing multi-level and multi-scale visual features from the input image, and providing rich semantic and geometric information for subsequent lane line positioning and refinement; in the embodiment of the present application, a deep residual neural network resnet34, which is lighter than the deep residual network resnet50, is used as the backbone network for image feature extraction, which can improve the efficiency of feature extraction. Accordingly, other types of deep residual neural networks can also be used as backbone networks in the embodiment of the present application, such as ResNet18, HRNetW18, and HRNetW32. The first features of multiple levels may include multi-stage features, for example, the first features of multiple levels include multiple low-level features and multiple high-level features, wherein the low-level features and the high-level features are semantic information represented by the outputs of different levels of the neural network. In the embodiment of the present application, the low-level features and the high-level features are features output by different feature extraction convolution layers in the backbone network 10. For example, see Figure 3A As shown, the backbone network is constructed by multiple layers of cascaded feature extraction layers. The backbone network 10 includes 4 layers of cascaded feature extraction convolutional layers (C3, C4, C5, C6 from bottom to top). Among them, the C3 and C4 feature extraction convolutional layers belong to the line layer network, so they output low-level features, and the C5 and C6 feature extraction convolutional layers belong to the deep network, so they output high-level features.

[0083] In the embodiment of the present application, when the preview device obtains image data of the road in front of the vehicle, the image data can be input into the backbone network for multi-scale feature extraction. Each convolutional layer in the backbone network outputs features of the corresponding level to obtain first features of multiple levels, including second features of multiple intermediate levels and third features of the last level; for example, see Figure 3A As shown, the backbone network 10 in the lane detection model 1 includes four cascaded feature extraction convolutional layers layer1 (C3), layer2 (C4), layer3 (C5), and layer4 (C6). The multiple first features output by the backbone network include the first feature output by layer1, the first feature output by layer2, the first feature output by layer3, and the first feature output by layer4. The first feature output by layer1, the first feature output by layer2, and the first feature output by layer3 are the second features of the above-mentioned intermediate layers, and the first feature output by layer4 is the third feature of the above-mentioned last layer.

[0084] S302: Input the third feature into a feature enhancement network for feature enhancement to obtain an enhanced feature.

[0085] The enhanced features include multi-scale contextual features and / or long-range spatial correlation features of lane lines. The feature enhancement network is a pre-trained network used to capture multi-scale contextual information and / or long-range spatial correlation information of lane lines. When used to capture multi-scale contextual information, it can be a golden pooling sub-network; when used to capture long-range spatial correlation information of lane lines, it can be an attention mechanism sub-network.

[0086] In the embodiment of the present application, when the preview device obtains the first features of multiple levels, the third features of the last level included therein can be input into the enhancement network for feature enhancement. On the one hand, since the enhancement network can capture multi-scale context information, multi-scale context features can be obtained after feature enhancement by the enhancement network; on the other hand, since the enhancement network can also capture the long-distance spatial correlation information of the lane line, long-distance spatial correlation features of the lane line can be obtained after feature enhancement by the enhancement network. For example, see Figure 3A As shown, the first feature output by layer4 (C6) in the backbone network 10 in the lane detection model 1 is input into the feature enhancement network 20 to obtain the enhanced feature Z.

[0087] S303: Input the enhanced features and multiple intermediate-level second features into the detection network for feature fusion and detection to obtain lane line point information.

[0088] The detection network can include a feature fusion structure and a detection module. The feature fusion structure can be a cross-layer refinement network (CLRNet) or an optimized CLRNet, namely FASP-CLRNet. This optimized CLRNet is the feature fusion structure of PAFPN optimized from the FPN structure in CLRNet. The feature fusion structure of PAFPN is a "bottom-up + top-down" bidirectional connection structure, which can improve feature transfer efficiency by 40% and reduce information loss by 34% compared to the FPN structure in the traditional CLRNet. The feature fusion structure is used to effectively fuse high-level semantics with low-level details to output multiple fused features. The detection module is used to perform lane line detection based on the multiple fused features output by the feature fusion structure to obtain lane line point information.

[0089] In the embodiment of the present application, when the preview device obtains the enhanced features and the second features of multiple intermediate levels based on the above steps, the enhanced features and the second features of multiple intermediate levels can be input into the feature fusion structure in the detection network for feature fusion to obtain multiple fused features, and then the multiple fused features are input into the detection module for lane line detection to obtain lane line point information. For example, see Figure 3A As shown, the enhanced feature Z output by the feature enhancement network 20 and the multiple second features output by the backbone network 10 (the second features output by C3, C4 and C5) are input into the detection network 30 for feature fusion and detection to obtain lane line point information.

[0090] The method described in the above embodiment can capture multi-scale contextual information and / or long-distance spatial correlation information of lane lines through the feature enhancement network, so that the detection network can perform feature fusion and detection based on the multi-scale contextual information and / or long-distance spatial correlation information of lane lines, which can enhance the global perception ability of deep features and the contextual interaction between features, thereby improving the detection accuracy of multi-scale lane lines, especially the detection of lane lines with small, curved or occluded parts, thereby improving the accuracy of lane line detection in complex environments.

[0091] In some exemplary embodiments, a feature enhancement network is provided, which includes a pyramid pooling subnetwork and an attention mechanism subnetwork. The above S302 "inputs the third feature into the feature enhancement network for feature enhancement to obtain an enhanced feature" is as follows: Figure 4 As shown, including:

[0092] S401: Input the third feature into the pyramid pooling sub-network to capture multi-scale context information to obtain a multi-scale context feature.

[0093] The pyramid pooling subnetwork performs spatial pyramid pooling on input features, capturing multi-scale contextual information, enhancing the global perception of deep features, and effectively compensating for the lack of spatial resolution of high-level features. In practical applications, the pyramid pooling subnetwork can be a spatial pyramid pooling network (SPP), and the attention mechanism subnetwork can be a full attention network.

[0094] In the embodiment of the present application, when the preview device obtains the first features of multiple levels, the third features of the last level can be input into the pyramid pooling sub-network in the enhancement network for spatial pyramid pooling processing. Since the pyramid pooling sub-network can capture multi-scale context information, multi-scale context features can be obtained after feature enhancement by the pyramid pooling sub-network. For example, see Figure 4AAs shown, the third feature output by C6 in the backbone network 10 in the lane detection model 1 is input into the pyramid pooling subnetwork 21 in the feature enhancement network 20 to obtain the multi-scale context feature Z1.

[0095] S402: Input the third feature into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line, and obtain the long-distance spatial correlation feature of the lane line.

[0096] Among them, the attention mechanism sub-network is used to enhance the input features, which can effectively capture the long-distance spatial correlation information of lane lines (such as curvature and occlusion shapes), enhance the contextual interaction between features, and improve the perception of the global structure of lane lines in complex scenes.

[0097] In the embodiment of the present application, when the preview device obtains the first features of multiple levels output by the backbone network, the third feature of the last level can be input into the attention mechanism sub-network in the enhancement network for enhancement processing. Since the attention mechanism sub-network can capture the long-distance spatial correlation information of the lane line, after the feature enhancement by the attention mechanism sub-network, the long-distance spatial correlation features of the lane line can be obtained, and the background noise interference can be suppressed at the same time, thereby enhancing the model's ability to focus on subtle or blurred lane lines, thereby improving the model's ability to perceive the global structure of the lane line and further improving the detection accuracy and robustness. For example, see Figure 4A As shown, the third feature output by layer4 (C6) in the backbone network 10 in the lane detection model 1 is input into the attention mechanism sub-network 22 in the feature enhancement network 20 to obtain the long-distance spatial correlation feature Z2 of the lane line.

[0098] The method described in the above embodiment, by setting up a pyramid pooling sub-network and an attention mechanism sub-network, can, on the one hand, capture and detect based on multi-scale contextual information, enhance the global perception ability of deep features, and effectively compensate for the problem of insufficient spatial resolution of high-level features; on the other hand, it can also capture and detect based on long-distance spatial correlation information, thereby improving the perception ability of the global structure of lane lines in complex scenes; at the same time, it suppresses background noise interference, strengthens the model's ability to focus on subtle or blurred lane lines, improves the model's ability to perceive the global structure of lane lines, and further improves detection accuracy and robustness.

[0099] In some exemplary embodiments, a detection network is provided, which includes a first feature pyramid, a second feature pyramid, and a detection module. Based on this, the above S303 "inputs the enhanced features and the second features of the multiple intermediate levels into the detection network for feature fusion and detection to obtain lane line point information" is as follows: Figure 5 As shown, including:

[0100] S501: Input the multi-scale contextual features and the plurality of second features into a first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features.

[0101] Among them, the first feature pyramid is also called the Feature Pyramid Network (FPN), which is the core module of multi-scale feature fusion. Its core function is to solve the scale change problem in lane line detection (such as the difference between wide lane lines in the near distance and thin lane lines in the far distance), while enhancing the model's perception of slender structures and small targets. The first feature pyramid is used to fuse multi-scale features through top-down and lateral connections to generate high-resolution and semantically rich feature maps. The first feature pyramid is constructed by multiple layers of cascaded convolutional fusion layers and is laterally connected to some feature extraction convolutional layers in the backbone network. For example, see Figure 5A As shown, the first feature pyramid 31 includes four cascaded convolutional fusion layers.

[0102] In the embodiment of the present application, when the preview device obtains the second features of multiple intermediate levels output by the backbone network and the multi-scale context features output by the pyramid pooling sub-network, the preview device can input the multi-scale context features and multiple second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain the first fused features output by each convolutional fusion layer in the first feature pyramid. For example, see Figure 5A As shown, the backbone network 10 includes 4 layers of feature extraction convolution layers (from bottom to top C3, C4, C5, C6), and the first feature pyramid 31 includes 4 layers of cascaded convolution fusion layers (from top to bottom P5, P4, P3, P2). The preview device can input the multi-scale context feature Z1 output by the pyramid pooling subnetwork 21 into the convolution fusion layer P5 in the first feature pyramid 31, and input the fusion feature output by the convolution fusion layer P5 and the second feature output by the feature extraction convolution layer C5 in the backbone network 10 into the convolution fusion layer P4 in the first feature pyramid 31 for feature convolution fusion, and then the convolution fusion layer The fused features output by P4 and the second features output by the feature extraction convolution layer C4 in the backbone network 10 are input into the convolution fusion layer P3 in the first feature pyramid 31 for feature convolution fusion, and then the fused features output by the convolution fusion layer P3 and the second features output by the feature extraction convolution layer C3 in the backbone network 10 are input into the convolution fusion layer P2 in the first feature pyramid 31 for feature convolution fusion. In this process, each convolution fusion layer in the first feature pyramid 31 outputs the first fusion feature, and the first fusion features output by the convolution fusion layer P5, the convolution fusion layer P4, the convolution fusion layer P3, and the convolution fusion layer P2 can be obtained.

[0103] S502: Input the plurality of first fused features and the long-distance spatial correlation features of the lane lines into a second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features.

[0104] Among them, the second feature pyramid is also called the cross-layer refinement module, which is the core module of multi-scale feature fusion. Its core function is to transfer strong positioning features of the lower layers, that is, to solve the positioning accuracy problem in lane line detection (especially curved, occluded or small-scale lane lines) through layer-by-layer progressive optimization. The second feature pyramid is used for "bottom-up" feature fusion, and the second feature pyramid and the first feature pyramid are constructed to form a path aggregation feature pyramid network (PathAggregation Feature Pyramid Network, PAFPN), which not only retains the semantic enhancement capability of the FPN network, but also supplements the transmission of positioning information, making the entire network more comprehensive and accurate when processing multi-scale targets. The second feature pyramid is constructed by multiple layers of cascaded convolutional fusion layers and is laterally connected with the convolutional fusion layers in the first feature pyramid. For example, see Figure 5A As shown, the second feature pyramid 32 includes four cascaded convolutional fusion layers.

[0105] In the embodiment of the present application, when the preview device obtains multiple first fused features output by the first feature pyramid and the long-distance spatial correlation features output by the attention mechanism sub-network, the preview device can input the long-distance spatial correlation features and the multiple first fused features into the second feature pyramid for layer-by-layer feature fusion from bottom to top, and obtain the second fused features output by each convolutional fusion layer in the second feature pyramid. For example, see Figure 5AAs shown, the first feature pyramid 31 includes 4 layers of cascaded convolution fusion layers (P5, P4, P3, P2 from top to bottom). The preview device can input the first fusion feature output by the convolution fusion layer P2 in the first feature pyramid 31 into the convolution fusion layer N2 in the second feature pyramid 32 for feature fusion, and then input the fusion feature output by the convolution fusion layer N2 and the first fusion feature output by the convolution fusion layer P3 in the first feature pyramid 31 into the convolution fusion layer N3 in the second feature pyramid 32 for feature fusion, and then input the fusion feature output by the convolution fusion layer N3 and the first fusion feature output by the convolution fusion layer P4 in the first feature pyramid 31 into the convolution fusion layer N3 in the second feature pyramid 32 for feature fusion. The feature is input to the convolution fusion layer N4 in the second feature pyramid 32 for feature fusion, and then the fusion feature output by the convolution fusion layer N4 and the first fusion feature output by the convolution fusion layer P5 in the first feature pyramid 31 are input to the convolution fusion layer N5 in the second feature pyramid 32. At the same time, the long-distance spatial correlation feature Z2 output by the attention mechanism sub-network 22 is also input to the convolution fusion layer N5 for feature fusion. In this process, each convolution fusion layer in the second feature pyramid 32 outputs a second fusion feature, and the second fusion feature output by the convolution fusion layer N5, the convolution fusion layer N4, the convolution fusion layer N3, and the convolution fusion layer N2 can be obtained. It should be noted that the process of feature fusion of each convolution fusion layer in the first feature pyramid 31 can also be understood as an upsampling process. The process of feature fusion of each convolution fusion layer in the second feature pyramid 32 can also be understood as a downsampling process.

[0106] S503: Input the plurality of second fused features into a detection module to perform lane line detection to obtain lane line point information.

[0107] Among them, the detection module is used to decode the second fusion features output by each feature fusion layer in the second feature pyramid into specific lane line parameters and classification results.

[0108] In an embodiment of the present application, when the preview device obtains multiple second fused features output by the second feature pyramid, the multiple second fused features can be input into the detection module for classification and regression processing to obtain lane line point information.

[0109] The detection network involved in the above embodiment provides a "bottom-up + top-down" bidirectional connection structure, which can improve the feature transfer efficiency (for example, it can be improved by about 40%) and reduce information loss by 34% compared to FPN. By enhancing the path aggregation capability between different feature levels, it effectively integrates high-level semantics and low-level details, and improves the detection accuracy of multi-scale lane lines (especially small, curved or occluded parts); at the same time, it optimizes the feature transfer efficiency and reduces information loss, making it more suitable for lane line recognition tasks in complex scenarios. The structure of the detection network combined with the pyramid pooling sub-network further strengthens the complementarity of features at different levels, thereby improving the model's detection accuracy and robustness for multi-scale lane lines in complex scenarios.

[0110] In some exemplary embodiments, a detection network is further provided. The detection network further includes an information collection module based on the network structure described in the above embodiment. Based on this type of detection network, the above Figure 5 The method described in the embodiment further includes: inputting the plurality of second fusion features into the information collection module for feature aggregation to obtain aggregated features; correspondingly, the preview device executes the above Figure 5 In S503 of the method described in the embodiment, the following steps may be specifically performed: the aggregated features are input into the detection module to perform lane line detection to obtain lane line point information.

[0111] The information collection module is an attention mechanism that enhances the global context of lane features. It can efficiently extract features of local lane areas from the feature map and aggregate them to enhance the positioning accuracy and semantic consistency of lane detection. The information collection module is also known as the RoIGather module.

[0112] In the embodiment of the present application, when the preview device obtains multiple second fused features output by the second feature pyramid, it can input the multiple second fused features into the information collection module for feature aggregation to obtain aggregated features; and then input the aggregated features into the detection module for lane line detection to obtain lane line point information. Figure 5B As shown, the information collection module 50 performs feature aggregation on the multiple second fused features output by the second feature pyramid, and the output aggregated features are input to the detection module 40 for detection to obtain lane line point information. By adding an information collection module to perform feature aggregation and further performing lane line detection based on the aggregated features, the embodiment of the present application can greatly improve the accuracy of lane line detection, especially in complex lane environments, and can solve the problem of poor detection effect when the lane line is curved, obscured, or blurred.

[0113] In summary, the above Figure 2-Figure 5The road line detection model involved in the embodiment has the following advantages: First, through the joint network architecture design of a lightweight backbone network, a pyramid pooling sub-network, an attention mechanism sub-network, and a detection network including a first feature pyramid, a second feature pyramid, and a detection module, the road line detection model significantly improves its capabilities in terms of inference speed, global structure perception, feature transfer, lane line focus, and background noise suppression. This effectively solves the problem of poor detection results when encountering dense, small, curved, blurred, or partially obscured lane lines during road lane line detection. Second, by setting up a feature enhancement network including a pyramid pooling sub-network and an attention mechanism sub-network, the model can be driven to learn cross-domain feature invariance and local missing complement capabilities. Ultimately, in multi-scenario field measurements, it achieves improved detection accuracy in extreme weather scenarios, improved recall rate in occluded scenarios, and reduced generalization variance across data sets, comprehensively improving the model's detection accuracy and robustness, ensuring high stability and accuracy in complex environments.

[0114] In some exemplary embodiments, a method for training a road line detection model is also provided, such as Figure 6 As shown, the method includes:

[0115] S601, acquiring sample image data.

[0116] The sample image data includes image data of road lines in various scenarios, including annotated image data. These scenarios can include both general and complex scenarios, while complex scenarios can include occlusion, blur, curves, and other complex scenarios. The sample image data described in the embodiments of this application includes image data of various scenarios, which can enrich the sample data types and improve the detection accuracy of lane detection models trained based on these sample data types.

[0117] In an embodiment of the present application, the preview device can pre-train a road line detection model. Prior to this, the initial image data can be obtained and processed. For example, the preview device introduces random perturbations, such as noise, blur, rotation, and scaling, to process the initial image data to obtain sample image data containing complex scenes. The preview device can also use the CycleGAN network to perform style transfer (for example, converting a sunny day image to a rainy day, night, etc.) to generate sample image data of complex scenes with high realism. In addition, the CutOut method is also used to randomly block the data samples to simulate the scene where the lane line is blocked. In summary, the generalization and robustness of the model are improved, so that it can adapt to lane line detection problems in different scenarios and improve the real-time and accuracy of lane line detection. In addition, the preview device can also annotate part of the initial image data, that is, deframe and filter the initial image data, open the image using the corresponding graphical interface, select an annotation tool (such as a polyline tool), draw lane lines on the image, and add labels to them. Upon completion, the annotation results will be saved as a JSON file containing image information and annotation data. Finally, these annotated data will be organized to form a structured Tusimple format dataset, and the structured dataset will be used as sample image data. It should be noted that when the preview device obtains the initial image data, it can use the corresponding software to perform simulation modeling and build a variety of lane simulation scenes with different slopes. Finally, the image data collected by the image acquisition device during the vehicle's driving in the lane simulation scene will be output as sample image data.

[0118] S602: Train the initial road line detection model according to the sample image data to obtain a road line detection model.

[0119] The initial road line detection model includes an initial backbone network, an initial feature enhancement network, and an initial detection network; optionally, the initial road line detection model may also include an initial backbone network, an initial feature enhancement network, an initial information collection network, and an initial detection network.

[0120] In the embodiment of the present application, when the preview device acquires sample image data based on the aforementioned steps, the sample image data can be input into the initial road line detection model. During this process, a target loss is determined based on the output of the initial road line detection model and a preset training loss method. The parameters of the initial road line detection model are adjusted based on the target loss until training is complete, thereby obtaining a trained road line detection model. The target loss can include classification loss and regression loss, and the type of loss is not limited.

[0121] The method described in the above embodiment involves a training method for a road line detection model, and provides sample image data for various scenarios, including sample image data for complex scenes. Training the road line detection model based on this can improve the detection accuracy of the trained road line detection model, especially the detection accuracy in complex scenarios.

[0122] In some exemplary embodiments, a method for determining a shadow vanishing point is provided, such as Figure 7 As shown, the method includes:

[0123] S701: Determine a target lane line of the road ahead of the vehicle based on lane line point information.

[0124] The target lane lines are the lane lines within the outer area of ​​the vehicle, that is, the target lane lines are the lane lines at both ends of the vehicle.

[0125] In the embodiment of the present application, when the preview device obtains lane line point information, it can filter the lane line positioning information to filter out the target lane line, so as to facilitate the subsequent slope preview based on the target lane line. Specifically, when filtering the lane line point information, the two lane lines containing the most points can be filtered out, that is, the lane lines at the left and right ends in front of the vehicle, for example, see Figure 7A As shown in the figure, L1 and L2 are the lane lines at both ends of vehicle A, and these two lane lines contain the most points. For another example, the lane line point information includes multiple lane line spatial coordinate sequences, and each lane line spatial coordinate sequence includes multiple points. The lane line corresponding to the lane line spatial coordinate sequence containing the most points is selected as the target lane line.

[0126] S702: Fit the target lane line and determine the actual shadow vanishing point based on the fitted lane line.

[0127] The fitting process may be least squares fitting.

[0128] In an embodiment of the present application, the preview device can perform a least-squares fit on the target lane lines to determine the slopes of the two target lane lines at the left and right ends of the vehicle. Finally, based on the slopes and intersection of the two target lane lines, the location of the actual vanishing point in the image data of the road ahead of the vehicle is determined, providing the necessary parameters for calculating the road slope. The method described in the above embodiment provides a method for determining the actual vanishing point based on the lane lines at both ends of the vehicle. This method is simple and easy to implement, and can improve the efficiency and accuracy of vanishing point determination.

[0129] In some exemplary embodiments, a method for determining a shadow vanishing point is also provided. Figure 7The embodiment also includes the steps of: performing point interception processing on the target lane line to obtain the intercepted target lane line. Correspondingly, when the preview device executes the above-mentioned step S702, it specifically performs: performing fitting processing on the intercepted target lane line, and determining the actual shadow disappearance point based on the lane line generated after fitting.

[0130] In the embodiment of the present application, when the preview device obtains the lane line point information, it can filter the lane line positioning information to filter out the target lane line, so as to facilitate the subsequent slope preview based on the target lane line. Specifically, when filtering the lane line point information, the two lane lines containing the most points can be filtered out, that is, the lane lines at the left and right ends in front of the vehicle, and then the points of the two filtered lane lines are intercepted at a preset distance. Because the lane lines far away from the vehicle are not very meaningful for calculating the shadow cancellation point, it is necessary to intercept the points of the two lane lines at a preset distance before calculating the shadow cancellation point. For example, only the lane line at 1 / 3 near the vehicle end is taken, or the lane line 20 to 30 meters away from the vehicle end is taken. See Figure 7A As shown, L11 and L21 are the intercepted target lane lines. The preview device can perform a least squares fit on the intercepted target lane lines to determine the slopes of the two intercepted target lane lines at the left and right ends of the vehicle. Finally, based on the slopes and intersection of the two intercepted target lane lines, the position of the actual vanishing point in the image data of the road in front of the vehicle is determined, providing the necessary parameters for calculating the road slope. The method described in the above embodiment provides a method for determining the actual vanishing point based on the intercepted target lane lines. This method is simple and easy to implement, and can improve the efficiency and accuracy of vanishing point determination.

[0131] In some exemplary embodiments, a method for determining the slope value of the preview road according to the actual shadow disappearance point is also provided. Figure 8 As shown, the method includes:

[0132] S801, analyzing the image data to determine the theoretical vanishing point in the image data.

[0133] The theoretical vanishing point is the center point of the horizon in the image data, and is also the theoretical vanishing point when the road has no slope. Figure 8A As shown, L0 is the horizon, P0 is the theoretical shadow vanishing point, and P1 is the actual shadow vanishing point.

[0134] In an embodiment of the present application, when the preview device acquires image data, since the image data includes image data of the road where the vehicle is located, the preview device can first identify the horizon line of the road in the image data when locating the theoretical shadow vanishing point, then determine the center point of the horizon line, and finally use the center point as the theoretical shadow vanishing point in the image data.

[0135] S802, performing camera coordinate transformation on the theoretical vanishing point and the actual vanishing point to obtain the transformed theoretical vanishing point and the actual vanishing point.

[0136] In an embodiment of the present application, when the preview device obtains the theoretical vanishing point and the actual vanishing point, since these two vanishing points are points in the pixel coordinate system, it is necessary to combine the parameters of the image acquisition device (for example, the parameters of the monocular camera on the vehicle) to convert the theoretical vanishing point and the actual vanishing point into points in the image coordinate system, and then convert the theoretical vanishing point and the actual vanishing point in the image coordinate system into points in the camera coordinate system.

[0137] S803 , based on the pinhole imaging principle, determine the preview road slope value of the road in front of the vehicle according to the converted theoretical vanishing point and the actual vanishing point.

[0138] In the embodiment of the present application, when the preview device obtains the theoretical shadow vanishing point and the actual shadow vanishing point after coordinate conversion, the preview road slope value can be calculated based on the pinhole imaging principle, that is, the following relationship (1):

[0139]

[0140] Where y0 represents the actual vanishing point; y1 represents the theoretical vanishing point; f represents the focal length of the camera; and α represents the preview road slope value. The method described in the above embodiment utilizes the pinhole imaging principle to calculate the preview road slope value based on the vanishing point. This method is simple and easy to implement, and can improve the efficiency of vehicle slope preview.

[0141] In some exemplary embodiments, a method for determining the slope value of the preview road is also provided, such as Figure 9 As shown, the above S803 "determining the preview road slope value of the road in front of the vehicle based on the converted theoretical shadow vanishing point and the actual shadow vanishing point based on the pinhole imaging principle" includes:

[0142] S901, based on the pinhole imaging principle, determine the initial preview road slope value of the road in front of the vehicle according to the converted theoretical shadow vanishing point and the actual shadow vanishing point.

[0143] The method described in the embodiment of the present application is basically the same as the method described in the aforementioned step S803. For details, please refer to the aforementioned description and will not be repeated here. The method described in the embodiment of the present application obtains the initial preview road slope value.

[0144] S902 , obtaining current pitch angle information of the vehicle, and compensating the initial preview road slope value according to the pitch angle information to obtain a preview road slope value.

[0145] In an embodiment of the present application, when the preview device obtains the initial preview road slope value of the road in front of the vehicle based on the aforementioned steps, it can further obtain the vehicle's current pitch angle information (for example, the pitch angle of the vehicle at time K), and then perform timestamp alignment, that is, obtain the above-mentioned initial preview road slope value and pitch angle information at the same time, and finally add the vehicle's current pitch angle information as information compensation to the initial preview road slope value to obtain the final preview road slope value. The method described in the above embodiment takes into account the impact of the vehicle's posture on the slope estimation, so obtaining the real-time initial preview road slope value and pitch angle information, and using the pitch angle information to compensate for the slope estimation can improve the accuracy of the preview road slope value estimation.

[0146] In summary, Figure 7-Figure 9 The road slope preview method described in the embodiment proposes a method for slope preview using shadow vanishing points, which has the advantages of low latency and high precision, and can realize real-time slope preview; moreover, it also incorporates the posture compensation strategy of the vehicle body dynamics. On the basis of timestamp alignment, the vehicle pitch angle information is compensated to the slope data in real time, the preview slope value is accurately calculated, and the slope preview accuracy is optimized.

[0147] Based on the methods described in all the above embodiments, a road slope preview method is provided, such as Figure 10 As shown, the method includes:

[0148] S1001: Input image data of the road in front of the vehicle into the backbone network for multi-scale feature extraction to obtain first features of multiple levels.

[0149] The first features of the multiple levels include the second features of the multiple intermediate levels and the third features of the last level.

[0150] S1002: Input the third feature into the pyramid pooling sub-network to capture multi-scale context information to obtain a multi-scale context feature.

[0151] S1003: Input the multi-scale contextual features and the plurality of second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features.

[0152] S1004: Input the third feature into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line, and obtain the long-distance spatial correlation feature of the lane line.

[0153] S1005: Input the plurality of first fused features and the long-distance spatial correlation features of the lane lines into a second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features.

[0154] S1006: Input the plurality of second fusion features into the information collection module for feature aggregation to obtain aggregated features.

[0155] S1007: Input the aggregated features into the detection module to perform lane line detection to obtain lane line point information.

[0156] S1008: Determine the target lane line of the road ahead of the vehicle based on the lane line point information.

[0157] The target lane line is the lane line in the outer area of ​​the vehicle.

[0158] S1009: Perform point-position interception processing on the target lane line to obtain an intercepted target lane line.

[0159] S1010: Fitting is performed on the intercepted target lane line, and an actual shadow vanishing point is determined based on the lane line generated after fitting.

[0160] S1011, analyzing the image data to determine the theoretical vanishing point in the image data.

[0161] S1012, performing camera coordinate transformation on the theoretical vanishing point and the actual vanishing point to obtain the transformed theoretical vanishing point and the actual vanishing point.

[0162] S1013 , based on the pinhole imaging principle, determine an initial preview road slope value of the road in front of the vehicle according to the converted theoretical vanishing point and the actual vanishing point.

[0163] S1014 , obtaining the current pitch angle information of the vehicle, and compensating the initial preview road slope value according to the pitch angle information to obtain the preview road slope value.

[0164] The methods described in the above embodiments are all described in the above content. Please refer to the above description for details and will not be repeated here.

[0165] The method described in the embodiment of the present application is intended to solve the problem of road slope preview, and improve the accuracy and real-time performance of road slope preview, thereby improving the applicability and safety of the preview equipment on the vehicle under complex road conditions. In addition, the lane line detection model proposed in the embodiment of the present application improves the reasoning speed by setting a lightweight backbone network, and by setting a feature enhancement network for feature enhancement, it can improve the global structural perception ability of the lane line, and can improve the detection accuracy of lane lines on complex roads such as crossing lane lines, curves, blur, and occlusion, and also correspondingly improve the detection recall rate. Moreover, the lane line detection model also sets up an efficient path aggregation detection network (such as a first feature pyramid and a second feature pyramid connected laterally), which greatly reduces information loss; in addition, by setting an information collection module in the lane detection model, it can focus on the lane line area, and in fuzzy scenes, it can suppress non-lane line texture interference, achieve background noise suppression, and greatly reduce the false detection rate. In summary, the method described in the embodiment of the present application achieves a triple breakthrough in accuracy, speed and robustness in the lane line detection task, effectively solving the problem of poor detection effect when encountering dense, small, curved, blurred or partially obscured lane lines, more accurately identifying lane lines, and providing accurate basic data for subsequent slope calculation, so that the slope of the road ahead can be effectively and accurately predicted in the later stage in combination with the shadow vanishing point of the lane line, making up for the shortcomings of slope pre-aiming in the related art. Finally, the method described in the embodiment of the present application can allow the pre-aiming device on the vehicle to pre-adjust the vehicle suspension damping and body height in advance by estimating the slope value of the pre-aiming road, reducing the impact and bumps caused by slope changes, and enhancing the applicability of the pre-aiming device to different road conditions. Moreover, the above method also realizes slope pre-aiming based on visual information, reduces dependence on other external sensors (such as GPS), reduces system cost and complexity, and also illustrates that the pre-aiming device of this embodiment can be integrated into existing intelligent transportation systems and autonomous driving vehicles, with good scalability and compatibility.

[0166] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0167] Based on the same inventive concept, embodiments of the present application also provide a road slope preview device for implementing the aforementioned road slope preview method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more road slope preview device embodiments provided below can be found in the aforementioned limitations of the road slope preview method and will not be further elaborated here.

[0168] In some exemplary embodiments, Figure 11 As shown, a road slope preview device is provided, comprising:

[0169] A detection module 110 is configured to input image data of the road ahead of the vehicle into a road line detection model to perform road line detection and obtain lane line point information; the road line detection model is configured to perform detection based on multi-scale contextual information and / or long-range spatial correlation information of lane lines captured from the image data;

[0170] The determination module 111 is configured to determine an actual shadow vanishing point of the road in front of the vehicle based on the lane line position information, and determine a preview road slope value of the road in front of the vehicle based on the actual shadow vanishing point.

[0171] In some exemplary embodiments, the detection module 110 includes:

[0172] an extraction unit, configured to input image data of the road ahead of the vehicle into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the final level;

[0173] an enhancement unit, configured to input the third feature into the feature enhancement network for feature enhancement to obtain an enhanced feature; the enhanced feature includes a multi-scale context feature and / or a long-distance spatial correlation feature of the lane line;

[0174] A detection unit is used to input the enhanced features and the second features of the multiple intermediate levels into the detection network for feature fusion and detection to obtain the lane line point information.

[0175] In some exemplary embodiments, the enhancement unit includes:

[0176] A first capturing subunit is configured to input the third feature into the pyramid pooling subnetwork to capture multi-scale context information to obtain the multi-scale context feature;

[0177] The second capture subunit is used to input the third feature into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line to obtain the long-distance spatial correlation feature of the lane line.

[0178] In some exemplary embodiments, the detection unit includes:

[0179] A first input subunit is configured to input the multi-scale contextual feature and the plurality of second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features;

[0180] A second input subunit is configured to input the plurality of first fused features and the long-distance spatial correlation features of the lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features;

[0181] The third input subunit is used to input the multiple second fusion features into the detection module to perform lane line detection to obtain the lane line point information.

[0182] In some exemplary embodiments, the detection unit further includes:

[0183] an aggregation subunit, configured to input the plurality of second fusion features into the information collection module for feature aggregation to obtain an aggregated feature;

[0184] Correspondingly, the third input subunit is specifically configured to input the aggregated features into the detection module for lane line detection to obtain the lane line point information.

[0185] In some exemplary embodiments, the road slope preview device further includes:

[0186] An acquisition module, configured to acquire sample image data, wherein the sample image data includes image data of road lines in various scenarios;

[0187] A training module is used to train an initial road line detection model according to the sample image data to obtain the road line detection model.

[0188] In some exemplary embodiments, the determination module includes:

[0189] A first determining unit is configured to determine a target lane line of the road ahead of the vehicle based on the lane line point information; the target lane line is a lane line within a peripheral area of ​​the vehicle;

[0190] The second determining unit is used to perform fitting processing on the target lane line and determine the actual shadow disappearance point based on the fitted lane line.

[0191] In some exemplary embodiments, the determining module further includes:

[0192] An interception unit, configured to perform point interception processing on the target lane line to obtain an intercepted target lane line;

[0193] Correspondingly, the second determining unit is specifically configured to perform fitting processing on the intercepted target lane line, and determine the actual shadow vanishing point based on the lane line generated after fitting.

[0194] In some exemplary embodiments, the determining module further includes:

[0195] a third determining unit, configured to analyze the image data and determine a theoretical vanishing point in the image data;

[0196] a conversion unit, configured to perform camera coordinate conversion on the theoretical vanishing point and the actual vanishing point to obtain converted theoretical vanishing point and actual vanishing point;

[0197] The fourth determining unit is used to determine the preview road slope value of the road in front of the vehicle based on the pinhole imaging principle and according to the converted theoretical shadow vanishing point and the actual shadow vanishing point.

[0198] In some exemplary embodiments, the fourth determining unit includes:

[0199] a determination subunit, configured to determine an initial preview road slope value of the road in front of the vehicle based on the converted theoretical shadow vanishing point and the actual shadow vanishing point based on the pinhole imaging principle;

[0200] The compensation subunit is configured to obtain current pitch angle information of the vehicle and compensate the initial preview road slope value according to the pitch angle information to obtain the preview road slope value.

[0201] Each module in the aforementioned road slope preview device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0202] In some exemplary embodiments, a vehicle-mounted device is provided. The vehicle-mounted device may be a terminal, and its internal structure diagram may be as follows: Figure 12As shown. The vehicle-mounted device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the vehicle-mounted device is used to provide computing and control capabilities. The memory of the vehicle-mounted device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the vehicle-mounted device is used to exchange information between the processor and an external device. The communication interface of the vehicle-mounted device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a road slope preview method is implemented. The display unit of the vehicle-mounted device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the vehicle-mounted device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the vehicle-mounted device housing, or an external keyboard, touchpad or mouse.

[0203] Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0204] In some exemplary embodiments, a vehicle-mounted device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0205] Inputting image data of the road ahead of the vehicle into a road line detection model to perform road line detection to obtain lane line point information; the road line detection model is configured to perform detection based on multi-scale contextual information and / or long-range spatial correlation information of lane lines captured from the image data;

[0206] An actual shadow vanishing point of the road in front of the vehicle is determined according to the lane line point information, and a preview road slope value of the road in front of the vehicle is determined according to the actual shadow vanishing point.

[0207] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0208] Inputting image data of the road ahead of the vehicle into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the final level;

[0209] Inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial correlation features of lane lines;

[0210] The enhanced features and the second features of the multiple intermediate levels are input into the detection network for feature fusion and detection to obtain the lane line point information.

[0211] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0212] Inputting the third feature into the pyramid pooling subnetwork to capture multi-scale context information to obtain the multi-scale context feature;

[0213] The third feature is input into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line to obtain the long-distance spatial correlation feature of the lane line.

[0214] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0215] Inputting the multi-scale contextual features and the plurality of second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features;

[0216] Inputting the plurality of first fused features and the long-distance spatial correlation features of the lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features;

[0217] The multiple second fusion features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0218] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0219] Inputting the plurality of second fusion features into the information collection module for feature aggregation to obtain aggregated features;

[0220] Inputting the plurality of second fused features into the detection module to perform lane line detection to obtain the lane line point information includes:

[0221] The aggregated features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0222] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0223] Acquire sample image data; the sample image data includes image data of road lines in various scenarios;

[0224] An initial road line detection model is trained according to the sample image data to obtain the road line detection model.

[0225] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0226] Determining a target lane line of the road in front of the vehicle based on the lane line point information; the target lane line is a lane line within the peripheral area of ​​the vehicle;

[0227] The target lane line is fitted, and the actual shadow vanishing point is determined based on the fitted lane line.

[0228] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0229] Performing point interception processing on the target lane line to obtain an intercepted target lane line;

[0230] The fitting process for the target lane line and determining the actual shadow vanishing point based on the fitted lane line includes:

[0231] The intercepted target lane line is fitted, and the actual shadow vanishing point is determined based on the lane line generated after fitting.

[0232] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0233] Analyzing the image data to determine a theoretical vanishing point in the image data;

[0234] Performing camera coordinate transformation on the theoretical vanishing point and the actual vanishing point to obtain transformed theoretical vanishing point and actual vanishing point;

[0235] Based on the pinhole imaging principle, the preview road slope value of the road in front of the vehicle is determined according to the converted theoretical shadow vanishing point and the actual shadow vanishing point.

[0236] In some embodiments, when the processor executes the computer program, it further implements the following steps:

[0237] Based on the pinhole imaging principle, determining the initial preview road slope value of the road in front of the vehicle according to the converted theoretical shadow vanishing point and the actual shadow vanishing point;

[0238] The current pitch angle information of the vehicle is obtained, and the initial preview road slope value is compensated according to the pitch angle information to obtain the preview road slope value.

[0239] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0240] Inputting image data of the road ahead of the vehicle into a road line detection model to perform road line detection to obtain lane line point information; the road line detection model is configured to perform detection based on multi-scale contextual information and / or long-range spatial correlation information of lane lines captured from the image data;

[0241] An actual shadow vanishing point of the road in front of the vehicle is determined according to the lane line point information, and a preview road slope value of the road in front of the vehicle is determined according to the actual shadow vanishing point.

[0242] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0243] Inputting image data of the road ahead of the vehicle into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the final level;

[0244] Inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial correlation features of lane lines;

[0245] The enhanced features and the second features of the multiple intermediate levels are input into the detection network for feature fusion and detection to obtain the lane line point information.

[0246] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0247] Inputting the third feature into the pyramid pooling subnetwork to capture multi-scale context information to obtain the multi-scale context feature;

[0248] The third feature is input into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line to obtain the long-distance spatial correlation feature of the lane line.

[0249] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0250] Inputting the multi-scale contextual features and the plurality of second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features;

[0251] Inputting the plurality of first fused features and the long-distance spatial correlation features of the lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features;

[0252] The multiple second fusion features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0253] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0254] Inputting the plurality of second fusion features into the information collection module for feature aggregation to obtain aggregated features;

[0255] Inputting the plurality of second fused features into the detection module to perform lane line detection to obtain the lane line point information includes:

[0256] The aggregated features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0257] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0258] Acquire sample image data; the sample image data includes image data of road lines in various scenarios;

[0259] An initial road line detection model is trained according to the sample image data to obtain the road line detection model.

[0260] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0261] Determining a target lane line of the road in front of the vehicle based on the lane line point information; the target lane line is a lane line within the peripheral area of ​​the vehicle;

[0262] The target lane line is fitted, and the actual shadow vanishing point is determined based on the fitted lane line.

[0263] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0264] Performing point interception processing on the target lane line to obtain an intercepted target lane line;

[0265] The fitting process for the target lane line and determining the actual shadow vanishing point based on the fitted lane line includes:

[0266] The intercepted target lane line is fitted, and the actual shadow vanishing point is determined based on the lane line generated after fitting.

[0267] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0268] Analyzing the image data to determine a theoretical vanishing point in the image data;

[0269] Performing camera coordinate transformation on the theoretical vanishing point and the actual vanishing point to obtain transformed theoretical vanishing point and actual vanishing point;

[0270] Based on the pinhole imaging principle, the preview road slope value of the road in front of the vehicle is determined according to the converted theoretical shadow vanishing point and the actual shadow vanishing point.

[0271] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0272] Based on the pinhole imaging principle, determining the initial preview road slope value of the road in front of the vehicle according to the converted theoretical shadow vanishing point and the actual shadow vanishing point;

[0273] The current pitch angle information of the vehicle is obtained, and the initial preview road slope value is compensated according to the pitch angle information to obtain the preview road slope value.

[0274] In some embodiments, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0275] Inputting image data of the road ahead of the vehicle into a road line detection model to perform road line detection to obtain lane line point information; the road line detection model is configured to perform detection based on multi-scale contextual information and / or long-range spatial correlation information of lane lines captured from the image data;

[0276] An actual shadow vanishing point of the road in front of the vehicle is determined according to the lane line point information, and a preview road slope value of the road in front of the vehicle is determined according to the actual shadow vanishing point.

[0277] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0278] Inputting image data of the road ahead of the vehicle into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the final level;

[0279] Inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial correlation features of lane lines;

[0280] The enhanced features and the second features of the multiple intermediate levels are input into the detection network for feature fusion and detection to obtain the lane line point information.

[0281] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0282] Inputting the third feature into the pyramid pooling subnetwork to capture multi-scale context information to obtain the multi-scale context feature;

[0283] The third feature is input into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line to obtain the long-distance spatial correlation feature of the lane line.

[0284] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0285] Inputting the multi-scale contextual features and the plurality of second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features;

[0286] Inputting the plurality of first fused features and the long-distance spatial correlation features of the lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features;

[0287] The multiple second fusion features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0288] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0289] Inputting the plurality of second fusion features into the information collection module for feature aggregation to obtain aggregated features;

[0290] Inputting the plurality of second fused features into the detection module to perform lane line detection to obtain the lane line point information includes:

[0291] The aggregated features are input into the detection module to perform lane line detection to obtain the lane line point information.

[0292] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0293] Acquire sample image data; the sample image data includes image data of road lines in various scenarios;

[0294] An initial road line detection model is trained according to the sample image data to obtain the road line detection model.

[0295] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0296] Determining a target lane line of the road in front of the vehicle based on the lane line point information; the target lane line is a lane line within the peripheral area of ​​the vehicle;

[0297] The target lane line is fitted, and the actual shadow vanishing point is determined based on the fitted lane line.

[0298] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0299] Performing point interception processing on the target lane line to obtain an intercepted target lane line;

[0300] The fitting process for the target lane line and determining the actual shadow vanishing point based on the fitted lane line includes:

[0301] The intercepted target lane line is fitted, and the actual shadow vanishing point is determined based on the lane line generated after fitting.

[0302] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0303] Analyzing the image data to determine a theoretical vanishing point in the image data;

[0304] Performing camera coordinate transformation on the theoretical vanishing point and the actual vanishing point to obtain transformed theoretical vanishing point and actual vanishing point;

[0305] Based on the pinhole imaging principle, the preview road slope value of the road in front of the vehicle is determined according to the converted theoretical shadow vanishing point and the actual shadow vanishing point.

[0306] In some embodiments, when the computer program is executed by a processor, the following steps are further implemented:

[0307] Based on the pinhole imaging principle, determining the initial preview road slope value of the road in front of the vehicle according to the converted theoretical shadow vanishing point and the actual shadow vanishing point;

[0308] The current pitch angle information of the vehicle is obtained, and the initial preview road slope value is compensated according to the pitch angle information to obtain the preview road slope value.

[0309] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile memory and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable logic unit (PLC), a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, and the like.

[0310] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0311] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A road slope preview method, characterized in that: The method comprises: Inputting image data of the road ahead of the vehicle into a road line detection model to perform road line detection to obtain lane line point information; the road line detection model is configured to perform detection based on multi-scale contextual information and / or long-range spatial correlation information of lane lines captured from the image data; An actual shadow vanishing point of the road in front of the vehicle is determined according to the lane line point information, and a preview road slope value of the road in front of the vehicle is determined according to the actual shadow vanishing point.

2. The method according to claim 1, characterized in that The road line detection model includes a backbone network, a feature enhancement network, and a detection network. The image data of the road in front of the vehicle is input into the road line detection model to perform road line detection to obtain lane line point information, including: Inputting image data of the road ahead of the vehicle into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the final level; Inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial correlation features of lane lines; The enhanced features and the second features of the multiple intermediate levels are input into the detection network for feature fusion and detection to obtain the lane line point information.

3. The method according to claim 2, characterized in that The feature enhancement network includes a pyramid pooling subnetwork and an attention mechanism subnetwork. Inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features includes: Inputting the third feature into the pyramid pooling subnetwork to capture multi-scale context information to obtain the multi-scale context feature; The third feature is input into the attention mechanism sub-network to capture the long-distance spatial correlation information of the lane line to obtain the long-distance spatial correlation feature of the lane line.

4. The method according to claim 2, characterized in that The detection network includes a first feature pyramid, a second feature pyramid, and a detection module. The enhanced features and the second features of the plurality of intermediate levels are input into the detection network for feature fusion and detection to obtain the lane line point information, including: Inputting the multi-scale contextual features and the plurality of second features into the first feature pyramid for layer-by-layer feature fusion from top to bottom to obtain a plurality of first fused features; Inputting the plurality of first fused features and the long-distance spatial correlation features of the lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain a plurality of second fused features; The multiple second fusion features are input into the detection module to perform lane line detection to obtain the lane line point information.

5. The method according to claim 4, characterized in that The detection network further includes an information collection module, and the method further includes: Inputting the plurality of second fusion features into the information collection module for feature aggregation to obtain aggregated features; Inputting the plurality of second fused features into the detection module to perform lane line detection to obtain the lane line point information includes: The aggregated features are input into the detection module to perform lane line detection to obtain the lane line point information.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire sample image data; the sample image data includes image data of road lines in various scenarios; An initial road line detection model is trained according to the sample image data to obtain the road line detection model.

7. The method according to any one of claims 1 to 5, characterized in that Determining the actual shadow disappearance point of the road in front of the vehicle based on the lane line point information includes: Determining a target lane line of the road in front of the vehicle based on the lane line point information; the target lane line is a lane line within the peripheral area of ​​the vehicle; The target lane line is fitted, and the actual shadow vanishing point is determined based on the fitted lane line.

8. The method according to claim 7, characterized in that The method further comprises: Performing point interception processing on the target lane line to obtain an intercepted target lane line; The fitting process for the target lane line and determining the actual shadow vanishing point based on the fitted lane line includes: The intercepted target lane line is fitted, and the actual shadow vanishing point is determined based on the lane line generated after fitting.

9. The method according to any one of claims 1 to 5, characterized in that The step of determining the preview road slope value of the road in front of the vehicle according to the actual shadow disappearance point includes: Analyzing the image data to determine a theoretical vanishing point in the image data; Performing camera coordinate transformation on the theoretical vanishing point and the actual vanishing point to obtain transformed theoretical vanishing point and actual vanishing point; Based on the pinhole imaging principle, the preview road slope value of the road in front of the vehicle is determined according to the converted theoretical shadow vanishing point and the actual shadow vanishing point.

10. The method according to claim 9, characterized in that The method of determining the preview road slope value of the road in front of the vehicle based on the pinhole imaging principle and the converted theoretical shadow vanishing point and the actual shadow vanishing point includes: Based on the pinhole imaging principle, determining the initial preview road slope value of the road in front of the vehicle according to the converted theoretical shadow vanishing point and the actual shadow vanishing point; The current pitch angle information of the vehicle is obtained, and the initial preview road slope value is compensated according to the pitch angle information to obtain the preview road slope value.

11. A road slope preview device, characterized in that: The device comprises: a detection module configured to input image data of the road ahead of the vehicle into a road line detection model to perform road line detection and obtain lane line point information; the road line detection model is configured to perform detection based on captured multi-scale contextual information and / or long-range spatial correlation information of lane lines; A determination module is used to determine the actual shadow disappearance point of the road in front of the vehicle based on the lane line point information, and to determine the preview road slope value of the road in front of the vehicle based on the actual shadow disappearance point.

12. A vehicle-mounted device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images