Road slope preview method and device, vehicle-mounted equipment, storage medium and program product

Abstract

The application relates to a road slope preview method and device, a vehicle-mounted equipment, a storage medium and a program product. The method comprises the following steps: inputting image data of a road in front of a vehicle into a road line detection model to detect the road line, obtaining lane line point information, determining an actual vanishing point of the road in front of the vehicle according to the lane line point information, and determining a preview road slope value of the road in front of the vehicle according to the actual vanishing point. The road line detection model can detect based on captured multi-scale context information and long-distance spatial correlation information, can enhance the global perception ability of deep features, can effectively make up for the problem of insufficient spatial resolution of high-level features, can improve the accuracy of the road line detection model in detecting the road line, and can improve the accuracy of the road slope preview based on the road line detection result.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a road slope prediction method, device, vehicle-mounted equipment, storage medium, and program product. Background Technology

[0002] With the development of vehicle intelligence technology, intelligent chassis technology has emerged. Utilizing sensors, electronic control systems, and intelligent algorithms, it enables vehicles to perceive road conditions in real time, adapt to driver habits, and even meet personalized passenger needs, greatly enhancing vehicle intelligence. For example, intelligent vehicles can use intelligent chassis technology to predict the slope ahead, and the prediction results can provide crucial data support for the intelligent chassis to improve vehicle performance and safety in complex road conditions.

[0003] In related technologies, road slope prediction methods mainly rely on vehicle dynamics models for real-time calculations; however, this method suffers from inaccurate slope prediction. Summary of the Invention

[0004] Therefore, it is necessary to provide a road slope prediction method, device, vehicle-mounted equipment, storage medium, and program product that can improve the accuracy of slope prediction in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a road slope prediction method, including:

[0006] Image data of the road in front of the vehicle is input into a road line detection model to detect road lines and obtain lane line location information; the road line detection model is used to perform detection based on multi-scale contextual information and / or long-distance spatial correlation information of lane lines captured from the image data;

[0007] The actual vanishing point of the road in front of the vehicle is determined based on the lane line location information, and the predicted road slope value of the road in front of the vehicle is determined based on the actual vanishing point.

[0008] In some embodiments, the road line detection model includes a backbone network, a feature enhancement network, and a detection network; the step of inputting image data of the road in front of the vehicle into the road line detection model for road line detection to obtain lane line location information includes:

[0009] Image data of the road in front of the vehicle is input into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the last level;

[0010] The third feature is input into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial association features of lane lines.

[0011] The enhanced features and the second features of the multiple intermediate layers are input into the detection network for feature fusion and detection to obtain the lane line location information.

[0012] In some embodiments, the feature enhancement network includes a pyramid pooling subnetwork and an attention mechanism subnetwork. The step of inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features includes:

[0013] The third feature is input into the pyramid pooling subnetwork to capture multi-scale contextual information, thereby obtaining the multi-scale contextual feature.

[0014] The third feature is input into the attention mechanism subnetwork to capture long-distance spatial association information of lane lines, thereby obtaining the long-distance spatial association features of lane lines.

[0015] In some embodiments, the detection network includes a first feature pyramid, a second feature pyramid, and a detection module. The step of inputting the enhanced features and the multiple intermediate-level second features into the detection network for feature fusion and detection to obtain the lane line location information includes:

[0016] The multi-scale context features and the multiple second features are input into the first feature pyramid for top-to-bottom layer-by-layer feature fusion to obtain multiple first fused features;

[0017] The multiple first fusion features and the long-distance spatial correlation features of the lane lines are input into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features;

[0018] The multiple second fusion features are input into the detection module to detect lane lines and obtain the lane line location information.

[0019] In some embodiments, the detection network further includes an information collection module, and the method further includes:

[0020] The multiple second fusion features are input into the information collection module for feature aggregation to obtain aggregated features;

[0021] The step of inputting the plurality of second fused features into the detection module for lane line detection to obtain the lane line location information includes:

[0022] The aggregated features are input into the detection module to detect lane lines and obtain the lane line location information.

[0023] In some embodiments, the method further includes:

[0024] Acquire sample image data; the sample image data includes road line image data in various scenarios;

[0025] The initial road line detection model is trained based on the sample image data to obtain the road line detection model.

[0026] In some embodiments, determining the actual vanishing point of the road in front of the vehicle based on the lane line location information includes:

[0027] The target lane line of the road in front of the vehicle is determined based on the lane line location information; the target lane line is the lane line within the outer area of ​​the vehicle.

[0028] The target lane line is fitted, and the actual vanishing point is determined based on the fitted lane line.

[0029] In some embodiments, the method further includes:

[0030] The target lane line is subjected to point interception processing to obtain the intercepted target lane line;

[0031] The process of fitting the target lane line and determining the actual vanishing point based on the fitted lane line includes:

[0032] The extracted target lane line is fitted, and the actual vanishing point is determined based on the lane line generated after fitting.

[0033] In some embodiments, determining the target road slope value based on the actual vanishing point includes:

[0034] The image data is analyzed to determine the theoretical cancellation points in the image data;

[0035] The theoretical and actual cancellation points are transformed using camera coordinate systems to obtain the transformed theoretical and actual cancellation points.

[0036] Based on the pinhole imaging principle, the predicted road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0037] In some embodiments, determining the pre-aimed road slope value based on the pinhole imaging principle, according to the converted theoretical and actual vanishing points, includes:

[0038] Based on the pinhole imaging principle, the initial pre-aimed road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0039] The vehicle's current pitch angle information is obtained, and the initial preview road slope value is compensated based on the pitch angle information to obtain the preview road slope value.

[0040] Secondly, this application also provides a road slope prediction device, comprising:

[0041] The detection module is used to input image data of the road in front of the vehicle into the road line detection model to detect road lines and obtain lane line position information; the road line detection model is used to perform detection based on captured multi-scale context information and / or long-distance spatial correlation information of lane lines.

[0042] The determination module is used to determine the actual vanishing point of the road in front of the vehicle based on the lane line location information, and to determine the predicted road slope value of the road in front of the vehicle based on the actual vanishing point.

[0043] Thirdly, this application also provides an in-vehicle device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the road slope prediction method described in the first aspect above.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the road slope prediction method described in the first aspect above.

[0045] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the road slope prediction method described in the first aspect above.

[0046] The aforementioned road slope prediction method, device, vehicle-mounted equipment, storage medium, and program product input image data of the road in front of the vehicle into a road line detection model for road line detection, obtain lane line position information, determine the actual vanishing point of the road in front of the vehicle based on the lane line position information, and determine the predicted road slope value based on the actual vanishing point. On one hand, the road line detection model can perform detection based on captured multi-scale contextual information, enhancing the global perception capability of deep features and effectively compensating for the insufficient spatial resolution of high-level features. This improves the accuracy of road line detection and simultaneously enhances the accuracy of subsequent road slope prediction based on the road line detection results. On the other hand, the road line detection model can also perform detection based on captured long-distance spatial correlation information of lane lines, enhancing contextual interaction between features and improving the perception capability of the global structure of lane lines in complex scenes. This further improves the accuracy of road line detection in complex environments and simultaneously enhances the accuracy of subsequent road slope prediction based on the road line detection results. In addition, the above method is based on the vanishing point for road slope prediction, which can improve the accuracy of obtaining the predicted road slope value. Compared with the complex method of using vehicle dynamics model for slope prediction in related technologies, the above method is simple and easy to implement, and can also improve the efficiency of slope prediction. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a diagram illustrating the application environment of the road slope prediction method in the embodiments of this application.

[0049] Figure 2 This is one of the flowcharts illustrating the road slope prediction method in the embodiments of this application;

[0050] Figure 3 This is the second flowchart illustrating the road slope prediction method in the embodiments of this application;

[0051] Figure 3A This is one of the structural schematic diagrams of the road line detection model in the embodiments of this application;

[0052] Figure 4 This is the third flowchart illustrating the road slope prediction method in the embodiments of this application;

[0053] Figure 4AThis is the second schematic diagram of the road line detection model in the embodiments of this application;

[0054] Figure 5 This is the fourth flowchart illustrating the road slope prediction method in the embodiments of this application;

[0055] Figure 5A This is the third schematic diagram of the road line detection model in the embodiments of this application;

[0056] Figure 5B This is the fourth schematic diagram of the road line detection model in the embodiments of this application;

[0057] Figure 6 This is the fifth flowchart illustrating the road slope prediction method in the embodiments of this application;

[0058] Figure 7 This is the sixth flowchart illustrating the road slope prediction method in the embodiments of this application;

[0059] Figure 7A This is a schematic diagram of lane lines in an embodiment of this application;

[0060] Figure 8 This is the seventh flowchart illustrating the road slope prediction method in the embodiments of this application;

[0061] Figure 8A This is a schematic diagram showing the positions of the actual and theoretical cancellation points in the embodiments of this application;

[0062] Figure 9 This is the eighth flowchart illustrating the road slope prediction method in the embodiments of this application;

[0063] Figure 10 This is the ninth flowchart illustrating the road slope prediction method in the embodiments of this application;

[0064] Figure 11 This is a schematic diagram of the road slope prediction device in the embodiments of this application;

[0065] Figure 12 This is a diagram showing the internal structure of the vehicle-mounted device in an embodiment of this application. Detailed Implementation

[0066] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of this application.

[0067] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, this does not indicate any order, quantity, or importance, but is merely used to distinguish different components. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. Words such as “comprising” or “including” mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, without excluding other elements or objects.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0069] The road slope prediction method provided in this application can be applied to, for example... Figure 1 In the application environment shown, the pre-aiming device 102 is installed on the vehicle 104. The pre-aiming device 102 can be linked with the sensor system and electronic control system on the vehicle 104, and can perceive and analyze the condition of the road or environment where the vehicle 104 is located based on various perception data or other data. In this embodiment, the pre-aiming device 102 can pre-aime the slope of the road where the vehicle 104 is traveling, so as to determine the pre-aimed road slope value in real time. The pre-aiming device 102 can be an in-vehicle device on the vehicle 104, and can also be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0070] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the pre-aiming device to which the present application is applied. A specific pre-aiming device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0071] In some exemplary embodiments, such as Figure 2 As shown, a road slope prediction method is provided, which can be applied to... Figure 1 Taking the pre-aiming device 102 as an example, the explanation includes the following steps:

[0072] S201, input the image data of the road in front of the vehicle into the road line detection model to detect the road line and obtain the lane line position information.

[0073] The road line detection model can be a neural network or machine learning model, which can be pre-trained based on road image sample data under various environments. The road line detection model is used to identify or detect lane lines on the road. In this embodiment, the road line detection model can capture multi-scale contextual information and / or long-distance spatial association information of lane lines, and perform detection based on the captured multi-scale contextual information and / or long-distance spatial association information of lane lines. Lane line location information includes multiple lane line spatial coordinate sequences, each representing a corresponding lane line. For example, each lane line spatial coordinate sequence is: {(x1,y1),(x2,y2),...,(x...} n ,y n )}, where each coordinate in the sequence represents a lane line point.

[0074] In this embodiment, the pre-aiming device can acquire in real time image data of the road in front of the vehicle collected by an image acquisition device (e.g., monocular camera, front-facing camera, dashcam, etc.) during vehicle operation. When the vehicle is driving uphill or downhill, the road slope pre-aiming function can be activated. Based on this, the pre-aiming device can acquire a trained road line detection model and the image data of the road in front of the vehicle, and input the image data of the road in front of the vehicle into the road line detection model. In the first scenario, the road line detection model can capture multi-scale contextual information from the image data during the detection process, and perform road line detection based on the multi-scale contextual information to obtain lane line position information. The multi-scale contextual information is used to enhance the joint perception capability of local details and global semantics of lane lines through feature fusion of different spatial granularities, thereby coping with scale changes, occlusion, and lighting interference in complex road scenes. Optionally, the multi-scale contextual information includes small-scale features (fine-grained local information), medium-scale features, and large-scale features (global semantic information). Small-scale features include lane line edge information, color, and texture information, such as the gaps between dashed line segments, the transition information between solid and dashed lines, yellow solid lines, white dashed lines, etc. Medium-scale features may include local curvature changes of lane lines and the positional relationship between lane lines and surrounding road elements, such as the curvature of turning lane lines and the relative position of lane lines and medians. Large-scale features may include the shape (direction) of lane lines or the topology of the road, such as straight lane lines, S-shaped curves, one-way lanes, lanes at intersections, etc.

[0075] The second scenario involves the road line detection model capturing long-distance spatial association information of lane lines during the detection process. Based on this information, road line detection is performed to obtain lane line location information. This long-distance spatial association information represents the logical relationship between lane line regions that are far apart, enhancing contextual interaction between features and improving the ability to perceive the global structure of lane lines in complex scenarios. This information can include key information crucial to ensuring the spatial, semantic, and topological continuity of lane lines. Key information ensuring spatial continuity includes spatial continuity features and geometric feature transformation information, such as lane line curvature gradient information, lane line slope continuity information, and lane line spacing information. Key information ensuring semantic continuity includes the type of the same lane line and the semantic relationship between the lane line and its surrounding environment (such as shoulders, medians, and traffic signs), such as the type of the near-end lane line and the type of the far-end lane line. Key information ensuring the topological continuity of lane lines includes lane line parallelism features and lane line orientation features, such as the parallel relationships between multiple lane lines, lane line bifurcation information, lane line merging information, and lane line turning information. In the third scenario, the road line detection model can simultaneously capture multi-scale contextual information and long-distance spatial association information of lane lines during the detection process, and perform road line detection based on this multi-scale contextual information and long-distance spatial association information to obtain lane line location information.

[0076] S202 determines the actual vanishing point of the road ahead based on the lane line location information, and determines the predicted road slope value based on the actual vanishing point.

[0077] The actual vanishing point is the point where parallel lines of the road where the vehicle is located converge at a distance, also known as the disappearance point. It is usually the point where the lane lines on both sides of the vehicle converge at a distance. The projected road slope value is a key parameter in intelligent vehicle control systems used to perceive the slope of the road ahead in advance, directly affecting the vehicle's power distribution, energy consumption optimization, and driving safety.

[0078] In this embodiment, when the pre-aiming device obtains the lane line position information of the road in front of the vehicle based on the aforementioned steps, since the lane line position information includes multiple lane line spatial coordinate sequences, the pre-aiming device can first filter out the lane lines on both sides of the vehicle from the multiple lane line spatial coordinate sequences, and determine the actual vanishing point of the road in front of the vehicle based on the lane lines on both sides of the vehicle. Since the actual vanishing point is the point where the lane lines on both sides of the vehicle converge at a distance, the road slope can be analyzed by analyzing this distant convergence point. Therefore, when calculating the actual vanishing point, the pre-aiming device can combine the road horizon and the position of the actual vanishing point to predict the pre-aiming road slope value of the road in front of the vehicle. For example, it can calculate the offset of the actual vanishing point relative to the horizon, and then determine the pre-aiming road slope value based on the offset.

[0079] The road slope prediction method described in the above embodiments inputs image data of the road in front of the vehicle into a road line detection model for road line detection, obtains lane line position information, determines the actual vanishing point of the road in front of the vehicle based on the lane line position information, and determines the predicted road slope value based on the actual vanishing point. On one hand, the road line detection model can perform detection based on captured multi-scale contextual information, enhancing the global perception capability of deep features and effectively compensating for the insufficient spatial resolution of high-level features. This improves the accuracy of road line detection and simultaneously enhances the accuracy of subsequent road slope prediction based on the road line detection results. On the other hand, the road line detection model can also perform detection based on the captured long-distance spatial association information of lane lines, enhancing the contextual interaction between features and improving the perception capability of the global structure of lane lines in complex scenes. This further improves the accuracy of road line detection in complex environments and simultaneously enhances the accuracy of subsequent road slope prediction based on the road line detection results. In addition, the above method is based on the vanishing point for road slope prediction, which can improve the accuracy of obtaining the predicted road slope value. Compared with the complex method of using vehicle dynamics model for slope prediction in related technologies, the above method is simple and easy to implement, and can improve the efficiency of slope prediction.

[0080] In some exemplary embodiments, a road line detection model of a certain type is provided, and a method for lane line detection based on this type of road line detection model is provided, such as... Figure 3 As shown, the method includes:

[0081] S301, input the image data of the road in front of the vehicle into the backbone network for multi-scale feature extraction, and obtain the first features at multiple levels.

[0082] The first feature across multiple levels includes second features across multiple intermediate levels and a third feature at the final level. The backbone network is a pre-trained network responsible for extracting features from images or other data; it is typically a Convolutional Neural Network (CNN) or a Residual Neural Network (ResNet). In this embodiment, the backbone network is the core feature extraction module in the road line detection model, responsible for capturing multi-level, multi-scale visual features from the input image, providing rich semantic and geometric information for subsequent lane line localization and refinement. This embodiment uses the ResNet34, a lighter-weight deep residual neural network than ResNet50, as the backbone network for image feature extraction, which improves feature extraction efficiency. Correspondingly, other types of deep residual neural networks, such as ResNet18, HRNetW18, and HRNetW32, can also be used as the backbone network in this embodiment. The first feature across multiple levels can include multi-stage features. For example, the first feature across multiple levels includes multiple low-level features and multiple high-level features. The low-level and high-level features represent semantic information expressed by the outputs of different levels of the neural network. In this embodiment, the low-level and high-level features are features from the outputs of different feature extraction convolutional layers in the backbone network 10. For example, see... Figure 3A As shown, the backbone network is constructed from multiple cascaded feature extraction layers. The backbone network 10 includes four cascaded feature extraction convolutional layers (C3, C4, C5, and C6 from bottom to top). Among them, the C3 and C4 feature extraction convolutional layers belong to linear networks, so they output low-level features. The C5 and C6 feature extraction convolutional layers belong to deep networks, so they output high-level features.

[0083] In this embodiment, when the pre-aiming device acquires image data of the road in front of the vehicle, it can input the image data into the backbone network for multi-scale feature extraction. Each convolutional layer in the backbone network outputs features corresponding to its level, resulting in multiple levels of first features, including multiple intermediate level second features and the last level of third features; for example, see... Figure 3A As shown, the backbone network 10 in the lane detection model 1 includes four cascaded feature extraction convolutional layers: layer1 (C3), layer2 (C4), layer3 (C5), and layer4 (C6). The multiple first features output by this backbone network include the first features output by layer1, layer2, layer3, and layer4. The first features output by layer1, layer2, and layer3 are the second features of the intermediate layers, and the first feature output by layer4 is the third feature of the last layer.

[0084] S302, the third feature is input into the feature enhancement network for feature enhancement to obtain the enhanced feature.

[0085] The enhanced features include multi-scale contextual features and / or long-range spatial association features of lane lines. The feature enhancement network is a pre-trained network used to capture multi-scale contextual information and / or long-range spatial association information of lane lines. When used to capture multi-scale contextual information, it can be a pyramid pooling sub-network; when used to capture long-range spatial association information of lane lines, it can be an attention mechanism sub-network.

[0086] In this embodiment, when the pre-aiming device acquires first features at multiple levels, it can input the third feature at the last level into the enhancement network for feature enhancement. On one hand, since the enhancement network can capture multi-scale contextual information, multi-scale contextual features can be obtained after feature enhancement. On the other hand, since the enhancement network can also capture long-range spatial association information of lane lines, long-range spatial association features of lane lines can be obtained after feature enhancement. For example, see... Figure 3A As shown, the first feature output by layer 4 (C6) in the backbone network 10 of the lane line detection model 1 is input into the feature enhancement network 20 to obtain the enhanced feature Z.

[0087] S303 inputs enhanced features and secondary features from multiple intermediate layers into the detection network for feature fusion and detection to obtain lane line location information.

[0088] The detection network can include a feature fusion structure and a detection module. The feature fusion structure can be a cross-layer refinement network (CLRNet) or an optimized CLRNet, namely FASP-CLRNet. This optimized CLRNet optimizes the FPN structure in CLRNet into a PAPPN feature fusion structure. The PAPPN feature fusion structure is a bidirectional connection structure with both bottom-up and top-down connections, which can improve feature transfer efficiency by 40% and reduce information loss by 34% compared to the traditional FPN structure in CLRNet. The feature fusion structure is used to effectively fuse high-level semantics with low-level details to output multiple fused features. The detection module is used to detect lane lines based on the multiple fused features output by the feature fusion structure to obtain lane line location information.

[0089] In this embodiment of the application, when the pre-aiming device obtains enhanced features and multiple intermediate-level second features based on the aforementioned steps, it can input the enhanced features and multiple intermediate-level second features into the feature fusion structure in the detection network for feature fusion to obtain multiple fused features. These multiple fused features are then input into the detection module for lane line detection to obtain lane line location information. For example, see... Figure 3A As shown, the enhanced feature Z output by the feature enhancement network 20 and the multiple second features (second features output by C3, C4 and C5) output by the backbone network 10 are input into the detection network 30 for feature fusion and detection to obtain lane line location information.

[0090] The method described in the above embodiments can capture multi-scale contextual information and / or long-distance spatial association information of lane lines through the feature enhancement network. This enables the detection network to perform feature fusion and detection based on multi-scale contextual information and / or long-distance spatial association information of lane lines. This can enhance the global perception capability of deep features and the contextual interaction between features, thereby improving the detection accuracy of multi-scale lane lines, especially lane lines in small, curved or occluded parts. Therefore, it improves the accuracy of lane line detection in complex environments.

[0091] In some exemplary embodiments, a feature enhancement network is provided, which includes a pyramid pooling subnetwork and an attention mechanism subnetwork. The above-described S302 "inputs a third feature into the feature enhancement network for feature enhancement, obtaining enhanced features," as shown below. Figure 4 As shown, it includes:

[0092] S401, the third feature is input into the pyramid pooling subnetwork to capture multi-scale contextual information and obtain multi-scale contextual features.

[0093] The pyramid pooling subnetwork is used to perform spatial pyramid pooling on the input features, which captures multi-scale contextual information, enhances the global perception capability of deep features, and effectively compensates for the insufficient spatial resolution of high-level features. In practical applications, the pyramid pooling subnetwork can be a spatial pyramid pooling network (SPP); the attention mechanism subnetwork can be a full attention mechanism network.

[0094] In this embodiment, when the pre-aiming device acquires first features at multiple levels, the third feature at the last level can be input into the pyramid pooling subnetwork in the enhancement network for spatial pyramid pooling processing. Since this pyramid pooling subnetwork can capture multi-scale contextual information, multi-scale contextual features can be obtained after feature enhancement through this subnetwork. For example, see... Figure 4AAs shown, the third feature output by C6 in the backbone network 10 of the lane line detection model 1 is input into the pyramid pooling subnetwork 21 in the feature enhancement network 20 to obtain the multi-scale context feature Z1.

[0095] S402, the third feature is input into the attention mechanism subnetwork to capture the long-distance spatial association information of the lane lines, and the long-distance spatial association features of the lane lines are obtained.

[0096] The attention mechanism subnetwork is used to enhance the input features, which can effectively capture long-distance spatial correlation information of lane lines (such as curvature and occlusion patterns), enhance the contextual interaction between features, and improve the perception of the global structure of lane lines in complex scenes.

[0097] In this embodiment, when the pre-aiming device acquires the first features of multiple levels output by the backbone network, it can input the third feature of the last level into the attention mechanism subnetwork in the enhancement network for enhancement processing. Since this attention mechanism subnetwork can capture long-range spatial correlation information of lane lines, after feature enhancement by this subnetwork, long-range spatial correlation features of lane lines can be obtained. Simultaneously, background noise interference can be suppressed, enhancing the model's ability to focus on subtle or blurred lane lines, improving the model's global structural perception of lane lines, and further improving detection accuracy and robustness. For example, see... Figure 4A As shown, the third feature output by layer 4 (C6) in the backbone network 10 of the lane line detection model 1 is input into the attention mechanism subnetwork 22 in the feature enhancement network 20 to obtain the long-distance spatial association feature Z2 of the lane line.

[0098] The method described in the above embodiments, by setting up a pyramid pooling subnetwork and an attention mechanism subnetwork, can, on the one hand, perform capture detection based on multi-scale contextual information, enhance the global perception capability of deep features, and effectively compensate for the problem of insufficient spatial resolution of high-level features; on the other hand, it can also perform capture detection based on long-distance spatial correlation information, improve the perception capability of the global structure of lane lines in complex scenes; at the same time, it suppresses background noise interference, strengthens the model's ability to focus on subtle or blurry lane lines, improves the model's perception capability of the global structure of lane lines, and further improves detection accuracy and robustness.

[0099] In some exemplary embodiments, a detection network is provided, which includes a first feature pyramid, a second feature pyramid, and a detection module. Based on this, the above-mentioned S303 "inputs enhanced features and multiple intermediate-level second features into the detection network for feature fusion and detection to obtain lane line location information," such as... Figure 5 As shown, it includes:

[0100] S501, multi-scale contextual features and multiple second features are input into the first feature pyramid for top-down layer-by-layer feature fusion to obtain multiple first fused features.

[0101] The first feature pyramid, also known as the Feature Pyramid Network (FPN), is the core module for multi-scale feature fusion. Its core function is to address scale variation issues in lane detection (such as the difference between wide lane lines in the foreground and narrow lane lines in the background), while enhancing the model's ability to perceive slender structures and small objects. The first feature pyramid is used to fuse multi-scale features through top-down and lateral connections, generating high-resolution and semantically rich feature maps. The first feature pyramid is constructed from multiple cascaded convolutional fusion layers and is laterally connected to some feature extraction convolutional layers in the backbone network. For example, see [link to related documentation]. Figure 5A As shown, the first feature pyramid 31 includes four cascaded convolutional fusion layers.

[0102] In this embodiment, when the pre-aiming device acquires multiple intermediate-level second features output from the backbone network and multi-scale contextual features output from the pyramid pooling sub-network, the pre-aiming device can input the multi-scale contextual features and multiple second features into the first feature pyramid for top-down layer-by-layer feature fusion, obtaining the first fused feature output from each convolutional fusion layer in the first feature pyramid. For example, see... Figure 5A As shown, the backbone network 10 includes four feature extraction convolutional layers (C3, C4, C5, C6 from bottom to top), and the first feature pyramid 31 includes four cascaded convolutional fusion layers (P5, P4, P3, P2 from top to bottom). The aiming device can input the multi-scale contextual features Z1 output by the pyramid pooling sub-network 21 into the convolutional fusion layer P5 in the first feature pyramid 31, and input the fused features output by the convolutional fusion layer P5 and the second features output by the feature extraction convolutional layer C5 in the backbone network 10 into the convolutional fusion layer P4 in the first feature pyramid 31 for feature convolutional fusion. The fused feature output by P4 and the second feature output by the feature extraction convolutional layer C4 in the backbone network 10 are input into the convolutional fusion layer P3 in the first feature pyramid 31 for feature convolutional fusion. Then, the fused feature output by the convolutional fusion layer P3 and the second feature output by the feature extraction convolutional layer C3 in the backbone network 10 are input into the convolutional fusion layer P2 in the first feature pyramid 31 for feature convolutional fusion. During this process, each convolutional fusion layer in the first feature pyramid 31 outputs the first fused feature, thus obtaining the first fused features output by the convolutional fusion layers P5, P4, P3, and P2.

[0103] S502, input multiple first fusion features and long-distance spatial correlation features of lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features.

[0104] The second feature pyramid, also known as the cross-layer refinement module, is the core module for multi-scale feature fusion. Its core function is to pass strong localization features from lower layers, i.e., to solve the localization accuracy problem in lane detection (especially for curved, occluded, or small-scale lanes) through progressive layer-by-layer optimization. The second feature pyramid is used for bottom-up feature fusion, and together with the first feature pyramid, they form the Path Aggregation Feature Pyramid Network (PAFPN). This not only retains the semantic enhancement capabilities of the FPN network but also supplements the transmission of localization information, making the entire network more comprehensive and accurate in processing multi-scale targets. The second feature pyramid is constructed from multiple cascaded convolutional fusion layers and is laterally connected to the convolutional fusion layers in the first feature pyramid. For example, see [link to related documentation]. Figure 5A As shown, the second feature pyramid 32 includes four cascaded convolutional fusion layers.

[0105] In this embodiment, when the pre-aiming device acquires multiple first fused features output from the first feature pyramid and long-range spatial correlation features output from the attention mechanism sub-network, the pre-aiming device can input the long-range spatial correlation features and multiple first fused features into the second feature pyramid for bottom-up, layer-by-layer feature fusion, to obtain the second fused features output by each convolutional fusion layer in the second feature pyramid. For example, see... Figure 5AAs shown, the first feature pyramid 31 includes four cascaded convolutional fusion layers (P5, P4, P3, P2 from top to bottom). The aiming device can input the first fused feature output from the convolutional fusion layer P2 in the first feature pyramid 31 to the convolutional fusion layer N2 in the second feature pyramid 32 for feature fusion. Then, the fused feature output from the convolutional fusion layer N2 and the first fused feature output from the convolutional fusion layer P3 in the first feature pyramid 31 are input to the convolutional fusion layer N3 in the second feature pyramid 32 for feature fusion. Finally, the fused feature output from the convolutional fusion layer N3 and the first fused feature output from the convolutional fusion layer P4 in the first feature pyramid 31 are input to the convolutional fusion layer N3 in the second feature pyramid 32 for feature fusion. The features are input into the convolutional fusion layer N4 in the second feature pyramid 32 for feature fusion. Then, the fused features output from convolutional fusion layer N4 and the first fused features output from convolutional fusion layer P5 in the first feature pyramid 31 are input into the convolutional fusion layer N5 in the second feature pyramid 32. Simultaneously, the long-range spatial correlation features Z2 output from the attention mechanism subnetwork 22 are input into the convolutional fusion layer N5 for feature fusion. During this process, each convolutional fusion layer in the second feature pyramid 32 outputs a second fused feature, thus obtaining the second fused features output by convolutional fusion layers N5, N4, N3, and N2. It should be noted that the feature fusion process in each convolutional fusion layer of the first feature pyramid 31 can also be understood as an upsampling process. The feature fusion process in each convolutional fusion layer of the second feature pyramid 32 can also be understood as a downsampling process.

[0106] S503, input multiple second fusion features into the detection module to detect lane lines and obtain lane line location information.

[0107] The detection module is used to decode the second fused features output by each feature fusion layer in the second feature pyramid into specific lane line parameters and classification results.

[0108] In this embodiment of the application, when the pre-aiming device obtains multiple second fusion features output by the second feature pyramid, the multiple second fusion features can be input into the detection module for classification and regression processing to obtain lane line location information.

[0109] The detection network described in the above embodiments provides a bidirectional connection structure of "bottom-up + top-down", which can improve feature transfer efficiency (e.g., by about 40%) and reduce information loss by 34% compared to FPN. By enhancing the path aggregation capability between different feature levels, it effectively integrates high-level semantics and low-level details, improving the detection accuracy of multi-scale lane lines (especially small, curved, or occluded parts). At the same time, it optimizes feature transfer efficiency and reduces information loss, making it more suitable for lane line recognition tasks in complex scenarios. The structure of the detection network, which combines a pyramid pooling sub-network, further strengthens the complementarity of features at different levels, improving the model's detection accuracy and robustness for multi-scale lane lines in complex scenarios.

[0110] In some exemplary embodiments, a detection network is also provided, which further includes an information collection module based on the network structure described in the foregoing embodiments. Based on this type of detection network, the aforementioned... Figure 5 The method described in the embodiment further includes: inputting multiple second fusion features into an information collection module for feature aggregation to obtain aggregated features; correspondingly, the pre-aiming device performs the above... Figure 5 In step S503 of the method described in the embodiment, the following steps can be specifically executed: inputting the aggregated features into the detection module to detect lane lines and obtain lane line location information.

[0111] The information gathering module is an attention mechanism that enhances the global contextual information of lane line features. It can efficiently extract and aggregate features from local lane line regions in the feature map to improve the localization accuracy and semantic consistency of lane line detection. This information gathering module is also known as the RoIGather module.

[0112] In this embodiment, when the pre-aiming device acquires multiple second fused features output from the second feature pyramid, it can input these multiple second fused features into the information collection module for feature aggregation to obtain aggregated features; then, the aggregated features are input into the detection module for lane line detection to obtain lane line location information. For example, participating in... Figure 5B As shown, the information collection module 50 aggregates multiple second fusion features output from the second feature pyramid, and the aggregated features are input to the detection module 40 for detection to obtain lane line location information. This embodiment of the application, by adding an information collection module for feature aggregation and further performing lane line detection based on the aggregated features, can greatly improve the accuracy of lane line detection. Especially in complex lane environments, it can solve the problem of poor detection performance when dealing with curved, occluded, or blurred lane lines.

[0113] In summary, the above Figures 2-5The road line detection model involved in this embodiment, firstly, through a joint network architecture design of a lightweight backbone network, a pyramid pooling subnetwork, an attention mechanism subnetwork, and a detection network including a first feature pyramid, a second feature pyramid, and a detection module, significantly improves the capabilities of the road line detection model in terms of inference speed, global structure perception, feature transfer, lane line focusing, and background noise suppression. This effectively solves the problem of poor detection performance when encountering dense, fine, curved, blurred, or partially occluded lane lines. Secondly, by setting up a feature enhancement network including a pyramid pooling subnetwork and an attention mechanism subnetwork, the model can learn cross-domain feature invariance and local missing feature completion capabilities. Ultimately, in multi-scenario testing, this improves detection accuracy in extreme weather scenarios, increases recall in occluded scenarios, and reduces generalization variance across datasets, comprehensively improving the model's detection accuracy and robustness, ensuring high stability and accuracy in complex environments.

[0114] In some exemplary embodiments, a method for training a road line detection model is also provided, such as... Figure 6 As shown, the method includes:

[0115] S601, acquire sample image data.

[0116] The sample image data includes road line images from various scenarios, as well as labeled image data. These scenarios can include general and complex scenarios, with complex scenarios including occlusion, blurring, and curves. The inclusion of image data from multiple scenarios in the sample image data described in this application enriches the sample data type and improves the detection accuracy of lane line detection models trained based on this type of sample data.

[0117] In this embodiment, the pre-aiming device can pre-train a road line detection model. Before this, it can acquire initial image data and process it. For example, the pre-aiming device can introduce random perturbations, such as noise, blurring, rotation, and scaling, and other data augmentation techniques to process the initial image data, obtaining sample image data containing complex scenes. The pre-aiming device can also use a CycleGAN network for style transfer (e.g., converting sunny images to rainy or dark scenes) to generate highly realistic sample image data of complex scenes. In addition, a CutOut technique is used to randomly occlude the data samples, simulating the scenario where lane lines are occluded. In summary, this improves the generalization and robustness of the model, enabling it to adapt to lane line detection problems in different scenarios and improving the real-time performance and accuracy of lane line detection. Furthermore, the pre-aiming device can also annotate part of the initial image data, that is, split the initial image data into frames, filter them, open the image using a corresponding graphical interface, select annotation tools (such as polyline tools), draw lane lines on the image, and add labels to them. After completion, the annotation results will be saved as a JSON file, containing image information and annotation data. Finally, this annotated data will be organized to form a structured Tusimple format dataset, which will ultimately be used as sample image data. It should be noted that when the aiming device acquires the initial image data, appropriate software can be used for simulation modeling, and various lane simulation scenarios with different slopes can be built. Finally, the image data collected by the image acquisition device during the vehicle's movement in the lane simulation scenario will be output as sample image data.

[0118] S602, Train the initial road line detection model based on the sample image data to obtain the road line detection model.

[0119] The initial road line detection model includes an initial backbone network, an initial feature enhancement network, and an initial detection network; alternatively, the initial road line detection model may also include an initial backbone network, an initial feature enhancement network, an initial information collection network, and an initial detection network.

[0120] In this embodiment, when the aiming device acquires sample image data based on the aforementioned steps, it can input the sample image data into the initial road line detection model. During this process, the target loss is determined based on the output of the initial road line detection model and a preset training loss method. The parameters of the initial road line detection model are then adjusted based on the target loss until training is completed, resulting in a trained road line detection model. The target loss may include classification loss and regression loss, and the type of loss is not limited.

[0121] The method described in the above embodiments relates to a training method for a road line detection model and provides sample image data in various scenarios, including sample image data for complex scenarios. Based on this, the road line detection model can be trained to improve the detection accuracy of the trained road line detection model, especially the detection accuracy in complex scenarios.

[0122] In some exemplary embodiments, a method for determining the vanishing point is provided, such as... Figure 7 As shown, the method includes:

[0123] S701 determines the target lane line of the road in front of the vehicle based on the lane line location information.

[0124] The target lane line is the lane line within the outer area of ​​the vehicle, that is, the target lane line is the lane line at both ends of the vehicle.

[0125] In this embodiment, when the pre-aiming device acquires lane line location information, it can filter the lane line positioning information to select target lane lines, facilitating subsequent slope pre-aiming based on the target lane lines. Specifically, when filtering lane line location information, it can select the two lane lines containing the most points, namely the lane lines at the left and right ends in front of the vehicle. For example, see... Figure 7A As shown, L1 and L2 are the lane lines at both ends of vehicle A, and these two lane lines contain the most points. For example, lane line point information includes multiple lane line spatial coordinate sequences representing lane lines, and each lane line spatial coordinate sequence includes multiple points. The lane line corresponding to the lane line spatial coordinate sequence containing the most points is taken as the target lane line.

[0126] S702, the target lane line is fitted, and the actual vanishing point is determined based on the fitted lane line.

[0127] Among them, the fitting process can be least squares fitting.

[0128] In this embodiment, the aiming device can perform least-squares fitting on the target lane lines to determine the slopes of the two target lane lines at the left and right ends of the vehicle. Finally, based on the slopes and intersection of the two target lane lines, the position of the actual vanishing point in the image data of the road in front of the vehicle is determined, providing necessary parameters for calculating the road slope. The method described in the above embodiment provides a method for determining the actual vanishing point based on the lane lines at both ends of the vehicle. This method is simple and easy to implement, and can improve the efficiency and accuracy of vanishing point determination.

[0129] In some exemplary embodiments, a method for determining the vanishing point is also provided, the method being... Figure 7The embodiment also includes the step of: performing point interception processing on the target lane line to obtain the intercepted target lane line. Correspondingly, when the pre-aiming device performs the above-mentioned step S702, it specifically performs: performing fitting processing on the intercepted target lane line, and determining the actual vanishing point based on the lane line generated after fitting.

[0130] In this embodiment, when the pre-aiming device acquires lane line location information, it can filter the lane line positioning information to select target lane lines, facilitating subsequent slope pre-aiming based on the target lane lines. Specifically, when filtering lane line location information, the two lane lines containing the most points can be selected, namely the lane lines at the left and right ends in front of the vehicle. Then, points at a preset distance are extracted from the selected two lane lines. Because lane lines far from the vehicle are not very meaningful for calculating the vanishing point, it is necessary to extract points at a preset distance from the two lane lines before calculating the vanishing point. For example, only the lane line at the closest 1 / 3 of the vehicle end can be selected, or the lane line 20-30 meters away from the vehicle end can be selected. See [reference needed]. Figure 7A As shown, L11 and L21 are the cropped target lane lines. The aiming device can perform least-squares fitting on the cropped target lane lines to determine the slopes of the two cropped target lane lines at the left and right ends of the vehicle. Finally, based on the slopes and intersection of the two cropped target lane lines, the position of the actual vanishing point in the image data of the road in front of the vehicle is determined, providing necessary parameters for calculating the road slope. The method described in the above embodiment provides a method for determining the actual vanishing point based on the cropped target lane lines. This method is simple and easy to implement, and can improve the efficiency and accuracy of vanishing point determination.

[0131] In some exemplary embodiments, a method for determining the target road slope value based on the actual vanishing point is also provided, such as... Figure 8 As shown, the method includes:

[0132] S801 analyzes the image data to determine the theoretical vanishing point in the image data.

[0133] The theoretical vanishing point is the center point of the horizon line in the image data, and also the theoretical vanishing point when the road has no slope. See also Figure 8A As shown, L0 is the horizon line, P0 is the theoretical point of shadow disappearance, and P1 is the actual point of shadow disappearance.

[0134] In this embodiment of the application, when the pre-aiming device acquires image data, since the image data includes image data of the road where the vehicle is located, the pre-aiming device can first identify the horizon line of the road in the image data when locating the theoretical cancellation point, then determine the center point of the horizon line, and finally use the center point as the theoretical cancellation point in the image data.

[0135] S802 performs a camera coordinate system transformation on the theoretical and actual vanishing points to obtain the transformed theoretical and actual vanishing points.

[0136] In this embodiment of the application, when the aiming device obtains the theoretical and actual cancellation points, since these two cancellation points are points in the pixel coordinate system, it is necessary to combine the parameters of the image acquisition device (e.g., the parameters of the monocular camera on the vehicle) to convert the theoretical and actual cancellation points into points in the image coordinate system, and then convert the theoretical and actual cancellation points in the image coordinate system into points in the camera coordinate system.

[0137] S803, based on the pinhole imaging principle, determines the predicted road slope value in front of the vehicle by combining the converted theoretical and actual vanishing points.

[0138] In this embodiment of the application, when the pre-aiming device obtains the theoretical and actual vanishing points after coordinate transformation, the pre-aiming road slope value can be calculated based on the pinhole imaging principle, i.e., the following relationship (1):

[0139]

[0140] Where y0 represents the actual vanishing point; y1 represents the theoretical vanishing point; f represents the camera's focal length; and α represents the pre-aimed road slope value. The method described in the above embodiment utilizes the pinhole imaging principle to calculate the pre-aimed road slope value based on the vanishing point. It is simple and easy to implement, and can improve the efficiency of vehicle slope pre-aiming.

[0141] In some exemplary embodiments, a method for determining the target road slope value is also provided, such as... Figure 9 As shown, the above-mentioned S803 "based on the pinhole imaging principle, determining the pre-aimed road slope value of the road in front of the vehicle according to the converted theoretical and actual vanishing points" includes:

[0142] S901, based on the pinhole imaging principle, determines the initial preview road slope value of the road in front of the vehicle according to the converted theoretical vanishing point and the actual vanishing point.

[0143] The method described in this embodiment is basically the same as the method described in step S803 above. For details, please refer to the foregoing description, which will not be repeated here. The method described in this embodiment obtains the initial pre-planned road slope value.

[0144] S902, obtain the vehicle's current pitch angle information, and compensate the initial target road slope value based on the pitch angle information to obtain the target road slope value.

[0145] In this embodiment, when the aiming device obtains the initial aiming road slope value based on the aforementioned steps, it can further obtain the vehicle's current pitch angle information (e.g., the vehicle's pitch angle at time K), and then perform timestamp alignment, i.e., obtain the aforementioned initial aiming road slope value and pitch angle information at the same time. Finally, the vehicle's current pitch angle information is added to the initial aiming road slope value as information compensation to obtain the final aiming road slope value. The method described in the above embodiment takes into account the influence of the vehicle's attitude on slope estimation. Therefore, obtaining the real-time initial aiming road slope value and pitch angle information, and using the pitch angle information for slope estimation compensation, can improve the accuracy of the aiming road slope value estimation.

[0146] In conclusion, Figures 7-9 The road slope prediction method described in the embodiment proposes a method for slope prediction using vanishing points, which has the advantages of low latency and high accuracy, and can realize real-time slope prediction. Moreover, it also incorporates a vehicle dynamic attitude compensation strategy, which, based on timestamp alignment, compensates the vehicle pitch angle information to the slope data in real time, accurately calculates the predicted slope value, and optimizes the slope prediction accuracy.

[0147] Based on the methods described in all the above embodiments, a road slope prediction method is provided, such as... Figure 10 As shown, the method includes:

[0148] S1001, input the image data of the road in front of the vehicle into the backbone network for multi-scale feature extraction, and obtain the first features at multiple levels.

[0149] The first feature of multiple levels includes the second feature of multiple intermediate levels and the third feature of the last level.

[0150] S1002, the third feature is input into the pyramid pooling subnetwork to capture multi-scale contextual information and obtain multi-scale contextual features.

[0151] S1003, input the multi-scale context features and multiple second features into the first feature pyramid for top-down layer-by-layer feature fusion to obtain multiple first fused features.

[0152] S1004, the third feature is input into the attention mechanism subnetwork to capture the long-distance spatial association information of the lane lines, and the long-distance spatial association features of the lane lines are obtained.

[0153] S1005, input multiple first fusion features and long-distance spatial correlation features of lane lines into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features.

[0154] S1006, input multiple second fusion features into the information collection module for feature aggregation to obtain aggregated features.

[0155] S1007, input the aggregated features into the detection module to detect lane lines and obtain lane line location information.

[0156] S1008, determine the target lane line of the road in front of the vehicle based on the lane line location information.

[0157] The target lane line refers to the lane line within the outer area of ​​the vehicle.

[0158] S1009, perform point interception processing on the target lane line to obtain the intercepted target lane line.

[0159] S1010: Fit the extracted target lane line and determine the actual vanishing point based on the lane line generated after fitting.

[0160] S1011, Analyze the image data to determine the theoretical vanishing point in the image data.

[0161] S1012 performs a camera coordinate system transformation on the theoretical and actual vanishing points to obtain the transformed theoretical and actual vanishing points.

[0162] S1013, based on the pinhole imaging principle, determines the initial preview road slope value of the road in front of the vehicle according to the converted theoretical vanishing point and the actual vanishing point.

[0163] S1014: Obtain the vehicle's current pitch angle information and compensate the initial target road slope value based on the pitch angle information to obtain the target road slope value.

[0164] The methods described in the above embodiments have all been described in the foregoing. For details, please refer to the foregoing description, which will not be repeated here.

[0165] The method described in this application aims to solve the road slope prediction problem and improves the accuracy and real-time performance of road slope prediction, thereby enhancing the applicability and safety of on-vehicle prediction devices in complex road conditions. Furthermore, the lane line detection model proposed in this application improves inference speed by setting a lightweight backbone network and enhances features through a feature enhancement network, thereby improving the global structure perception capability of lane lines and increasing the detection accuracy of lane lines on complex roads such as intersecting lane lines, curves, blurred areas, and occlusions, thus also improving the detection recall rate. Moreover, the lane line detection model also sets up an efficient path aggregation detection network (such as including a first feature pyramid and a second feature pyramid with lateral connections), greatly reducing information loss. In addition, by setting up an information collection module in the lane detection model, it can focus on the lane line region, suppressing non-lane line texture interference in blurred scenes, achieving background noise suppression, and greatly reducing the false detection rate. In summary, the method described in this application achieves a triple breakthrough in accuracy, speed, and robustness in lane detection tasks. It effectively solves the problem of poor detection performance when encountering dense, fine, curved, blurred, or partially obscured lane lines, providing more accurate lane line identification and accurate basic data for subsequent slope calculation. This allows for effective and accurate prediction of the road slope ahead by combining the vanishing points of lane lines, thus compensating for the shortcomings of slope prediction in related technologies. Finally, the method described in this application, by estimating the predicted road slope value, allows the vehicle's prediction device to pre-adjust the vehicle's suspension damping and body height, reducing the impact and bumps caused by slope changes and enhancing the applicability of the prediction device to different road conditions. Moreover, the above method achieves slope prediction based on visual information, reducing reliance on other external sensors (such as GPS), lowering system cost and complexity. It also demonstrates that the prediction device of this embodiment can be integrated into existing intelligent transportation systems and autonomous vehicles, exhibiting good scalability and compatibility.

[0166] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0167] Based on the same inventive concept, this application also provides a road slope prediction device for implementing the road slope prediction method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more road slope prediction device embodiments provided below can be found in the limitations of the road slope prediction method described above, and will not be repeated here.

[0168] In some exemplary embodiments, such as Figure 11 As shown, a road slope prediction device is provided, comprising:

[0169] The detection module 110 is used to input image data of the road in front of the vehicle into the road line detection model to detect the road lines and obtain lane line position information; the road line detection model is used to perform detection based on multi-scale contextual information and / or long-distance spatial correlation information of lane lines captured from the image data;

[0170] The determination module 111 is used to determine the actual vanishing point of the road in front of the vehicle based on the lane line position information, and to determine the predicted road slope value of the road in front of the vehicle based on the actual vanishing point.

[0171] In some exemplary embodiments, the detection module 110 described above includes:

[0172] The extraction unit is used to input the image data of the road in front of the vehicle into the backbone network for multi-scale feature extraction to obtain multiple levels of first features; the multiple levels of first features include multiple intermediate levels of second features and the last level of third features;

[0173] An enhancement unit is used to input the third feature into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial association features of lane lines;

[0174] The detection unit is used to input the enhanced features and the second features of the multiple intermediate layers into the detection network for feature fusion and detection to obtain the lane line location information.

[0175] In some exemplary embodiments, the above-described enhancement unit includes:

[0176] The first capture subunit is used to input the third feature into the pyramid pooling subnetwork to capture multi-scale context information and obtain the multi-scale context feature.

[0177] The second capture subunit is used to input the third feature into the attention mechanism subnetwork to capture the long-distance spatial association information of the lane line, thereby obtaining the long-distance spatial association feature of the lane line.

[0178] In some exemplary embodiments, the detection unit includes:

[0179] The first input subunit is used to input the multi-scale context features and the multiple second features into the first feature pyramid for top-to-bottom layer-by-layer feature fusion to obtain multiple first fused features;

[0180] The second input subunit is used to input the multiple first fusion features and the long-distance spatial correlation features of the lane lines into the second feature pyramid for sequential layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features;

[0181] The third input subunit is used to input the multiple second fusion features into the detection module to detect lane lines and obtain the lane line location information.

[0182] In some exemplary embodiments, the detection unit further includes:

[0183] The aggregation subunit is used to input the multiple second fusion features into the information collection module for feature aggregation to obtain aggregated features;

[0184] Correspondingly, the third input subunit is specifically used to input the aggregated features into the detection module to detect lane lines and obtain the lane line location information.

[0185] In some exemplary embodiments, the road slope prediction device described above further includes:

[0186] The acquisition module is used to acquire sample image data; the sample image data includes image data of road lines in various scenarios;

[0187] The training module is used to train the initial road line detection model based on the sample image data to obtain the road line detection model.

[0188] In some exemplary embodiments, the above-described determining module includes:

[0189] The first determining unit is used to determine the target lane line of the road in front of the vehicle based on the lane line location information; the target lane line is the lane line in the outer area of ​​the vehicle.

[0190] The second determining unit is used to perform fitting processing on the target lane line and determine the actual vanishing point based on the fitted lane line.

[0191] In some exemplary embodiments, the determining module further includes:

[0192] The interception unit is used to perform point interception processing on the target lane line to obtain the intercepted target lane line.

[0193] Correspondingly, the second determining unit is specifically used to perform fitting processing on the intercepted target lane line, and determine the actual vanishing point based on the lane line generated after fitting.

[0194] In some exemplary embodiments, the determining module further includes:

[0195] The third determining unit is used to analyze the image data and determine the theoretical cancellation point in the image data;

[0196] The transformation unit is used to perform camera coordinate system transformation on the theoretical vanishing point and the actual vanishing point to obtain the transformed theoretical vanishing point and the actual vanishing point.

[0197] The fourth determining unit is used to determine the pre-aimed road slope value of the road in front of the vehicle based on the pinhole imaging principle, according to the converted theoretical vanishing point and the actual vanishing point.

[0198] In some exemplary embodiments, the fourth determining unit described above includes:

[0199] A subunit is defined to determine the initial pre-aimed road slope value of the road in front of the vehicle based on the pinhole imaging principle and the converted theoretical and actual vanishing points.

[0200] The compensation subunit is used to acquire the current pitch angle information of the vehicle and compensate the initial preview road slope value according to the pitch angle information to obtain the preview road slope value.

[0201] Each module in the aforementioned road slope prediction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0202] In some exemplary embodiments, an in-vehicle device is provided, which may be a terminal, and its internal structure diagram may be as follows. Figure 12As shown, the vehicle-mounted device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a road slope prediction method. The display unit of the vehicle-mounted device is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the vehicle-mounted equipment can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the housing of the vehicle-mounted equipment, or external keyboards, touchpads, or mice, etc.

[0203] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0204] In some exemplary embodiments, an in-vehicle device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0205] Image data of the road in front of the vehicle is input into a road line detection model to detect road lines and obtain lane line location information; the road line detection model is used to perform detection based on multi-scale contextual information and / or long-distance spatial correlation information of lane lines captured from the image data;

[0206] The actual vanishing point of the road in front of the vehicle is determined based on the lane line location information, and the predicted road slope value of the road in front of the vehicle is determined based on the actual vanishing point.

[0207] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0208] Image data of the road in front of the vehicle is input into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the last level;

[0209] The third feature is input into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial association features of lane lines.

[0210] The enhanced features and the second features of the multiple intermediate layers are input into the detection network for feature fusion and detection to obtain the lane line location information.

[0211] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0212] The third feature is input into the pyramid pooling subnetwork to capture multi-scale contextual information, thereby obtaining the multi-scale contextual feature.

[0213] The third feature is input into the attention mechanism subnetwork to capture long-distance spatial association information of lane lines, thereby obtaining the long-distance spatial association features of lane lines.

[0214] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0215] The multi-scale context features and the multiple second features are input into the first feature pyramid for top-to-bottom layer-by-layer feature fusion to obtain multiple first fused features;

[0216] The multiple first fusion features and the long-distance spatial correlation features of the lane lines are input into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features;

[0217] The multiple second fusion features are input into the detection module to detect lane lines and obtain the lane line location information.

[0218] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0219] The multiple second fusion features are input into the information collection module for feature aggregation to obtain aggregated features;

[0220] The step of inputting the plurality of second fused features into the detection module for lane line detection to obtain the lane line location information includes:

[0221] The aggregated features are input into the detection module to detect lane lines and obtain the lane line location information.

[0222] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0223] Acquire sample image data; the sample image data includes road line image data in various scenarios;

[0224] The initial road line detection model is trained based on the sample image data to obtain the road line detection model.

[0225] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0226] The target lane line of the road in front of the vehicle is determined based on the lane line location information; the target lane line is the lane line within the outer area of ​​the vehicle.

[0227] The target lane line is fitted, and the actual vanishing point is determined based on the fitted lane line.

[0228] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0229] The target lane line is subjected to point interception processing to obtain the intercepted target lane line;

[0230] The process of fitting the target lane line and determining the actual vanishing point based on the fitted lane line includes:

[0231] The extracted target lane line is fitted, and the actual vanishing point is determined based on the lane line generated after fitting.

[0232] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0233] The image data is analyzed to determine the theoretical cancellation points in the image data;

[0234] The theoretical and actual cancellation points are transformed using camera coordinate systems to obtain the transformed theoretical and actual cancellation points.

[0235] Based on the pinhole imaging principle, the predicted road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0236] In some embodiments, when the processor executes a computer program, it further performs the following steps:

[0237] Based on the pinhole imaging principle, the initial pre-aimed road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0238] The vehicle's current pitch angle information is obtained, and the initial preview road slope value is compensated based on the pitch angle information to obtain the preview road slope value.

[0239] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0240] Image data of the road in front of the vehicle is input into a road line detection model to detect road lines and obtain lane line location information; the road line detection model is used to perform detection based on multi-scale contextual information and / or long-distance spatial correlation information of lane lines captured from the image data;

[0241] The actual vanishing point of the road in front of the vehicle is determined based on the lane line location information, and the predicted road slope value of the road in front of the vehicle is determined based on the actual vanishing point.

[0242] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0243] Image data of the road in front of the vehicle is input into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the last level;

[0244] The third feature is input into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial association features of lane lines.

[0245] The enhanced features and the second features of the multiple intermediate layers are input into the detection network for feature fusion and detection to obtain the lane line location information.

[0246] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0247] The third feature is input into the pyramid pooling subnetwork to capture multi-scale contextual information, thereby obtaining the multi-scale contextual feature.

[0248] The third feature is input into the attention mechanism subnetwork to capture long-distance spatial association information of lane lines, thereby obtaining the long-distance spatial association features of lane lines.

[0249] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0250] The multi-scale context features and the multiple second features are input into the first feature pyramid for top-to-bottom layer-by-layer feature fusion to obtain multiple first fused features;

[0251] The multiple first fusion features and the long-distance spatial correlation features of the lane lines are input into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features;

[0252] The multiple second fusion features are input into the detection module to detect lane lines and obtain the lane line location information.

[0253] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0254] The multiple second fusion features are input into the information collection module for feature aggregation to obtain aggregated features;

[0255] The step of inputting the plurality of second fused features into the detection module for lane line detection to obtain the lane line location information includes:

[0256] The aggregated features are input into the detection module to detect lane lines and obtain the lane line location information.

[0257] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0258] Acquire sample image data; the sample image data includes road line image data in various scenarios;

[0259] The initial road line detection model is trained based on the sample image data to obtain the road line detection model.

[0260] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0261] The target lane line of the road in front of the vehicle is determined based on the lane line location information; the target lane line is the lane line within the outer area of ​​the vehicle.

[0262] The target lane line is fitted, and the actual vanishing point is determined based on the fitted lane line.

[0263] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0264] The target lane line is subjected to point interception processing to obtain the intercepted target lane line;

[0265] The process of fitting the target lane line and determining the actual vanishing point based on the fitted lane line includes:

[0266] The extracted target lane line is fitted, and the actual vanishing point is determined based on the lane line generated after fitting.

[0267] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0268] The image data is analyzed to determine the theoretical cancellation points in the image data;

[0269] The theoretical and actual cancellation points are transformed using camera coordinate systems to obtain the transformed theoretical and actual cancellation points.

[0270] Based on the pinhole imaging principle, the predicted road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0271] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0272] Based on the pinhole imaging principle, the initial pre-aimed road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0273] The vehicle's current pitch angle information is obtained, and the initial preview road slope value is compensated based on the pitch angle information to obtain the preview road slope value.

[0274] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0275] Image data of the road in front of the vehicle is input into a road line detection model to detect road lines and obtain lane line location information; the road line detection model is used to perform detection based on multi-scale contextual information and / or long-distance spatial correlation information of lane lines captured from the image data;

[0276] The actual vanishing point of the road in front of the vehicle is determined based on the lane line location information, and the predicted road slope value of the road in front of the vehicle is determined based on the actual vanishing point.

[0277] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0278] Image data of the road in front of the vehicle is input into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the last level;

[0279] The third feature is input into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial association features of lane lines.

[0280] The enhanced features and the second features of the multiple intermediate layers are input into the detection network for feature fusion and detection to obtain the lane line location information.

[0281] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0282] The third feature is input into the pyramid pooling subnetwork to capture multi-scale contextual information, thereby obtaining the multi-scale contextual feature.

[0283] The third feature is input into the attention mechanism subnetwork to capture long-distance spatial association information of lane lines, thereby obtaining the long-distance spatial association features of lane lines.

[0284] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0285] The multi-scale context features and the multiple second features are input into the first feature pyramid for top-to-bottom layer-by-layer feature fusion to obtain multiple first fused features;

[0286] The multiple first fusion features and the long-distance spatial correlation features of the lane lines are input into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features;

[0287] The multiple second fusion features are input into the detection module to detect lane lines and obtain the lane line location information.

[0288] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0289] The multiple second fusion features are input into the information collection module for feature aggregation to obtain aggregated features;

[0290] The step of inputting the plurality of second fused features into the detection module for lane line detection to obtain the lane line location information includes:

[0291] The aggregated features are input into the detection module to detect lane lines and obtain the lane line location information.

[0292] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0293] Acquire sample image data; the sample image data includes road line image data in various scenarios;

[0294] The initial road line detection model is trained based on the sample image data to obtain the road line detection model.

[0295] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0296] The target lane line of the road in front of the vehicle is determined based on the lane line location information; the target lane line is the lane line within the outer area of ​​the vehicle.

[0297] The target lane line is fitted, and the actual vanishing point is determined based on the fitted lane line.

[0298] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0299] The target lane line is subjected to point interception processing to obtain the intercepted target lane line;

[0300] The process of fitting the target lane line and determining the actual vanishing point based on the fitted lane line includes:

[0301] The extracted target lane line is fitted, and the actual vanishing point is determined based on the lane line generated after fitting.

[0302] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0303] The image data is analyzed to determine the theoretical cancellation points in the image data;

[0304] The theoretical and actual cancellation points are transformed using camera coordinate systems to obtain the transformed theoretical and actual cancellation points.

[0305] Based on the pinhole imaging principle, the predicted road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0306] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0307] Based on the pinhole imaging principle, the initial pre-aimed road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

[0308] The vehicle's current pitch angle information is obtained, and the initial preview road slope value is compensated based on the pitch angle information to obtain the preview road slope value.

[0309] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0310] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0311] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for predicting road slope, characterized in that, The method includes: Image data of the road in front of the vehicle is input into a road line detection model to detect road lines and obtain lane line location information; the road line detection model is used to perform detection based on multi-scale contextual information and / or long-distance spatial correlation information of lane lines captured from the image data; The actual vanishing point of the road in front of the vehicle is determined based on the lane line location information, and the predicted road slope value of the road in front of the vehicle is determined based on the actual vanishing point. The road line detection model includes a backbone network, a feature enhancement network, and a detection network; the step of inputting image data of the road in front of the vehicle into the road line detection model for road line detection to obtain lane line location information includes: Image data of the road in front of the vehicle is input into the backbone network for multi-scale feature extraction to obtain first features at multiple levels; the first features at multiple levels include second features at multiple intermediate levels and third features at the last level; The third feature is input into the feature enhancement network for feature enhancement to obtain enhanced features; the enhanced features include multi-scale context features and / or long-distance spatial association features of lane lines. The enhanced features and the second features of the multiple intermediate layers are input into the detection network for feature fusion and detection to obtain the lane line location information.

2. The method according to claim 1, characterized in that, The feature enhancement network includes a pyramid pooling subnetwork and an attention mechanism subnetwork. The step of inputting the third feature into the feature enhancement network for feature enhancement to obtain enhanced features includes: The third feature is input into the pyramid pooling subnetwork to capture multi-scale contextual information, thereby obtaining the multi-scale contextual feature. The third feature is input into the attention mechanism subnetwork to capture long-distance spatial association information of lane lines, thereby obtaining the long-distance spatial association features of lane lines.

3. The method according to claim 1, characterized in that, The detection network includes a first feature pyramid, a second feature pyramid, and a detection module. The enhanced features and the multiple intermediate-level second features are input into the detection network for feature fusion and detection to obtain the lane line location information, including: The multi-scale context features and the multiple second features are input into the first feature pyramid for top-to-bottom layer-by-layer feature fusion to obtain multiple first fused features; The multiple first fusion features and the long-distance spatial correlation features of the lane lines are input into the second feature pyramid for layer-by-layer feature fusion from bottom to top to obtain multiple second fusion features; The multiple second fusion features are input into the detection module to detect lane lines and obtain the lane line location information.

4. The method according to claim 3, characterized in that, The detection network further includes an information collection module, and the method further includes: The multiple second fusion features are input into the information collection module for feature aggregation to obtain aggregated features; The step of inputting the plurality of second fused features into the detection module for lane line detection to obtain the lane line location information includes: The aggregated features are input into the detection module to detect lane lines and obtain the lane line location information.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Acquire sample image data; the sample image data includes road line image data in various scenarios; The initial road line detection model is trained based on the sample image data to obtain the road line detection model.

6. The method according to any one of claims 1-4, characterized in that, Determining the actual disappearance point of the road in front of the vehicle based on the lane line location information includes: The target lane line of the road in front of the vehicle is determined based on the lane line location information; the target lane line is the lane line within the outer area of ​​the vehicle. The target lane line is fitted, and the actual vanishing point is determined based on the fitted lane line.

7. The method according to claim 6, characterized in that, The method further includes: The target lane line is subjected to point interception processing to obtain the intercepted target lane line; The process of fitting the target lane line and determining the actual vanishing point based on the fitted lane line includes: The extracted target lane line is fitted, and the actual vanishing point is determined based on the lane line generated after fitting.

8. The method according to any one of claims 1-4, characterized in that, The step of determining the target road slope value based on the actual vanishing point includes: The image data is analyzed to determine the theoretical cancellation points in the image data; The theoretical and actual cancellation points are transformed using camera coordinate systems to obtain the transformed theoretical and actual cancellation points. Based on the pinhole imaging principle, the predicted road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point.

9. The method according to claim 8, characterized in that, The method of determining the pre-aimed road slope value of the road in front of the vehicle based on the pinhole imaging principle, according to the converted theoretical and actual vanishing points, includes: Based on the pinhole imaging principle, the initial pre-aimed road slope value of the road in front of the vehicle is determined according to the converted theoretical vanishing point and the actual vanishing point. The vehicle's current pitch angle information is obtained, and the initial preview road slope value is compensated based on the pitch angle information to obtain the preview road slope value.

10. A road slope prediction device, characterized in that, The device includes: The detection module is used to input image data of the road in front of the vehicle into the road line detection model to detect road lines and obtain lane line position information. The road line detection model is used to perform detection based on captured multi-scale context information and / or long-distance spatial correlation information of lane lines. The road line detection model includes a backbone network, a feature enhancement network and a detection network. The detection module includes an extraction unit, an enhancement unit, and a detection unit. The extraction unit inputs image data of the road in front of the vehicle into the backbone network for multi-scale feature extraction, obtaining multiple levels of first features. These multiple levels of first features include multiple intermediate-level second features and a final-level third feature. The enhancement unit inputs the third feature into the feature enhancement network for feature enhancement, obtaining enhanced features. These enhanced features include multi-scale contextual features and / or long-distance spatial association features of lane lines. The detection unit inputs the enhanced features and the multiple intermediate-level second features into the detection network for feature fusion and detection, obtaining the lane line location information. The determination module is used to determine the actual vanishing point of the road in front of the vehicle based on the lane line location information, and to determine the predicted road slope value of the road in front of the vehicle based on the actual vanishing point.

11. An in-vehicle device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images