Intelligent driving recognition system and method based on deep learning
The intelligent driving recognition system based on deep learning solves the problems of unstable image segmentation and insufficient obstacle reconstruction in autonomous driving under extreme conditions, and achieves the stability and seamless integration of functions of the 3D rendering interface, thereby improving driving safety and user experience.
Patent Information
- Application Number
- CN202511830031.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-03-31
AI Technical Summary
Existing autonomous driving perception and visualization technologies suffer from decreased image segmentation accuracy and unstable recognition boundaries under extreme weather or low lighting conditions, making it difficult to accurately reconstruct non-standard obstacles. Furthermore, the functions of the in-vehicle system are fragmented and the operation is complex, affecting driving safety and user experience.
The intelligent driving recognition system, based on deep learning, integrates modules for data acquisition, neural network model, data post-processing, graphics rendering and control, and display. It uses a hybrid neural network for semantic segmentation and obstacle 3D reconstruction, combined with multimodal data fusion and temporal filtering to generate a stable 3D rendering interface, and integrates navigation and parking information in a single interface.
It achieves precise segmentation of drivable areas and road boundaries in harsh environments, realistically reproduces the three-dimensional shape of obstacles, reduces operational complexity, improves driving safety and convenience, and provides integrated driving environment perception feedback.
Smart Images

Figure CN121767957A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of autonomous driving and intelligent cockpit display technology, and in particular to an intelligent driving recognition system and method based on deep learning. Background Technology
[0002] With the rapid popularization and development of autonomous driving technology, intelligent driving assistance systems have become an important feature of modern automobiles. To enhance driver trust and safety, vehicles typically display environmental perception results (SR, Simulated Reality) to the driver through the instrument panel or central control screen. This involves reconstructing the road environment, traffic participants, and driving trajectory around the vehicle in real time through a 3D rendering interface. However, as users' demands for intelligent driving experiences continue to increase, existing environmental perception and visualization technologies still have many shortcomings in complex scenarios.
[0003] In terms of environmental perception and road model construction, existing visual perception solutions are limited by algorithm architecture and post-processing mechanisms. When faced with extreme weather conditions (such as heavy rain or fog) or insufficient lighting, the accuracy of image segmentation decreases. More critically, existing technologies lack effective temporal stability processing, causing the identified road boundaries and drivable areas in each frame to be easily affected by environmental noise, resulting in abrupt positional changes. This instability in the recognition results is directly reflected in the 3D rendering interface, manifesting as frequent jittering or abrupt changes in the edges of the road surface model. This not only reduces the fidelity of the virtual environment to the real world but also easily leads to visual fatigue and distrust among drivers.
[0004] In terms of obstacle recognition and rendering, most current intelligent driving systems rely on standardized geometric models (such as cube boxes) to represent common targets such as vehicles and pedestrians. However, for non-standard, irregular, and irregularly shaped obstacles on the road (i.e., low-feature obstacles, such as fallen rocks, piles of dirt, and scattered goods), existing technologies often struggle to perform accurate 3D reconstruction. While pure vision solutions can identify obstacle areas through semantic segmentation, they lack depth information and cannot construct a 3D shape; while pure radar solutions, although possessing depth information, lack semantic contours. This results in such obstacles either being completely absent from the 3D rendering interface or being displayed only as generic icons without any morphological features, failing to accurately reflect the obstacle's volume and shape, thus increasing driving safety hazards.
[0005] Furthermore, in terms of human-computer interaction and information presentation architecture, current in-vehicle systems generally suffer from a fragmented functional hierarchy. Intelligent driving, automatic parking, and navigation maps typically exist as independent applications or functional modules, lacking deep information integration between them. When drivers need to view navigation directions or find parking spaces while driving, they often have to manually switch or jump between different interfaces. This cumbersome operational hierarchy not only causes delays in obtaining key information (such as navigation TBT commands and parking space locations) but also forces drivers to be distracted by operating the screen while driving, increasing cognitive load and operational risks, making it difficult to meet users' needs for a seamless integrated experience of driving, parking, and navigation. Summary of the Invention
[0006] The purpose of this invention is to provide a deep learning-based intelligent driving recognition system and method, which at least solves one of the technical problems in the prior art.
[0007] To address the above problems, the first aspect of this invention provides a deep learning-based intelligent driving recognition system. This system aims to solve the technical problems of inaccurate fusion of two-dimensional image semantic information and three-dimensional spatial data, jittery recognition boundaries, and distorted 3D reconstruction morphology in existing autonomous driving perception and visualization processes. The system integrates a data acquisition and provision module, a neural network model module, a data post-processing and fusion module, a graphics rendering and control module, and a display module.
[0008] The data acquisition and provision module is responsible for acquiring RGB image data and 3D point cloud data from hardware sensors and constructing a multimodal data input stream. This module transmits the RGB image data to the neural network model module, and simultaneously performs coordinate system registration on the 3D point cloud data to ensure the consistency of the point cloud data and image data in terms of spatial reference. The registered data is then sent to the data post-processing and fusion module.
[0009] The neural network model module is the perception core of the system, internally deploying a hybrid neural network. This module receives RGB image data and uses the hybrid neural network to perform pixel-level semantic segmentation of the image. This processing is not limited to simple object detection, but generates refined segmentation results data containing information on drivable areas, road boundaries, and common obstacle areas, providing semantic masks for subsequent 3D reconstruction.
[0010] To improve segmentation accuracy and restore spatial details, the hybrid neural network employs an encoder-decoder architecture. The encoder path consists of multiple downsampling stages, extracting deep features using residual blocks and employing max-pooling layers to reduce the spatial dimensionality of the feature maps to expand the receptive field. The decoder path consists of multiple upsampling stages, progressively restoring the spatial resolution of the feature maps through deconvolutional layers. Crucially, the feature maps output from each stage in the encoder path are concatenated to the corresponding input in the decoder path via skip connections. This connection method integrates shallow, low-level texture features with deep, high-level semantic features, improving the accuracy of segmentation edges. Furthermore, during network training, the system uses a combined loss function consisting of a weighted sum of cross-entropy loss and Dice loss to optimize the model from two dimensions: the difference in pixel classification probability distribution and the region overlap, thus addressing the problem of class imbalance in autonomous driving scenarios.
[0011] The data post-processing and fusion module is responsible for data cleaning and cross-modal fusion. Addressing the common boundary abruptness problem in visual recognition, this module performs temporal filtering on road boundary information. Specifically, this mechanism determines boundary points in the current frame's road boundary information and searches for the nearest spatial point in the filtered information from the previous frame. The Euclidean distance is obtained by calculating the square root of the sum of the squares of the differences between the two points' coordinates, and this Euclidean distance is compared to a preset threshold. If the distance is within the threshold, it is considered normal fluctuation, and the current point is directly adopted; if the distance exceeds the threshold, it is considered an abnormal abrupt change, and a weighted average of the nearest point and the current point is calculated. This logic effectively suppresses recognition noise and generates stable road boundary information.
[0012] In obstacle 3D reconstruction, the data post-processing and fusion module performs joint geometric and semantic verification. The system traverses the 3D point cloud data, using sensor extrinsic and camera intrinsic matrices to transform radar points from the radar coordinate system through the vehicle and camera coordinate systems and project them onto the image pixel coordinate system. Subsequently, the system queries the semantic values of the projected coordinates in the general obstacle area information. Only when the semantic value indicates that the location belongs to an obstacle area is the system retained and aggregated to form a 3D point cloud cluster. This method eliminates background noise points and generates obstacle 3D morphological data with accurate semantic attributes.
[0013] The graphics rendering and control module is responsible for transforming perceived abstract data into visualized geometric models. This module receives stabilized road boundary information and obstacle 3D morphology data, generates road surface geometry models, lane line geometry models, and general obstacle 3D models, and combines them to create a 3D rendering interface. Furthermore, this module supports deep integration of navigation and parking assistance functions. When receiving navigation command data containing TBT steering information and navigation thumbnails, the module generates corresponding visual elements in specific areas of the screen. In parking scenarios, the system combines the position information output by the parking space detection module and uses a projection matrix to draw selectable parking space elements in 3D space; when a selection operation is detected, the system highlights the selected area and calculates a guide trajectory line based on the vehicle's posture, achieving a closed loop from perception to interaction.
[0014] The display module ultimately receives and presents the aforementioned 3D rendered interface containing rich semantic and geometric information, providing users with intuitive feedback on their driving environment.
[0015] A second aspect of the present invention provides a deep learning-based intelligent driving recognition method, applied to a deep learning-based intelligent driving recognition system, the method comprising the following steps:
[0016] First, the data acquisition and provision module acquires RGB image data and 3D point cloud data for environmental perception, and performs multi-stage coordinate system registration on the 3D point cloud data to achieve data alignment in spatial dimensions.
[0017] Secondly, the RGB image data is processed using a hybrid neural network through a neural network model module. This step uses deep learning algorithms to extract image features and perform upsampling restoration, outputting pixel-level semantic segmentation results to generate segmentation data that includes drivable areas, road boundaries, and obstacle areas.
[0018] Next, the perceived data is optimized through a data post-processing and fusion module. This step involves two aspects: firstly, using a temporal filtering algorithm based on Euclidean distance to smooth the road boundaries and eliminate inter-frame jitter; secondly, using image semantic masks to filter and fuse the 3D point cloud to construct an accurate 3D point cloud cluster of obstacles.
[0019] Subsequently, the graphics rendering and control module constructs the geometric and three-dimensional models of each element based on the processed stable boundary data and obstacle 3D morphology data, and combines them to generate a complete 3D rendering interface.
[0020] Finally, the 3D rendering interface is displayed on the physical screen through the display module, showing the road geometry, lane lines and three-dimensional obstacle information around the vehicle in real time.
[0021] The above solution achieves the following beneficial technical effects:
[0022] This application deploys a hybrid neural network that combines an encoder-decoder architecture (U-Net structure) with a residual module, enabling in-depth extraction of image features and achieving pixel-level accurate segmentation of drivable areas and road boundaries even in low light or inclement weather. Combined with the temporal filtering algorithm in the data post-processing module, Euclidean distance is used to smoothly correct inter-frame boundary points, solving the problem of frequent boundary jumps and jitter in existing technologies and ensuring the continuity and stability of the road surface geometry model in the 3D rendering interface.
[0023] This application employs multimodal data fusion technology, which combines the general obstacle area information generated by visual semantic segmentation with the coordinate-registered radar 3D point cloud data. For low-feature obstacles such as mounds and falling rocks that are difficult to represent using standard geometric models, the system uses semantic masks to filter and aggregate the corresponding radar point cloud clusters for filling and display, thereby realistically restoring the 3D shape of the obstacle in the rendering interface, making up for the blind spots of pure vision solutions, and improving the driver's ability to perceive potential dangers.
[0024] This application constructs an integrated 3D rendering interface through a graphics rendering and control module, breaking down the barriers between the independent functions of traditional intelligent driving systems (driving, parking, and navigation). It directly integrates and displays navigation TBT steering information and thumbnails in the main driving interface, and seamlessly reveals parking space detection results and recommended trajectories in low-speed scenarios. This allows users to obtain core driving information without the need for cumbersome switching of application levels, reducing operational complexity and improving safety and convenience during driving. Attached Figure Description
[0025] Figure 1 A structural block diagram of a deep learning-based intelligent driving recognition system provided in an embodiment of the present invention;
[0026] Figure 2 A flowchart illustrating an intelligent driving recognition method based on deep learning, provided in an embodiment of the present invention.
[0027] Figure 3 This is a schematic diagram illustrating the principle of multimodal data coordinate system registration in an embodiment of the present invention;
[0028] Figure 4 This is a schematic diagram of the structure of the hybrid neural network model in an embodiment of the present invention;
[0029] Figure 5 This is a flowchart of the road boundary temporal filtering process in an embodiment of the present invention. Detailed Implementation
[0030] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] See attached document Figure 1 The intelligent driving recognition system based on deep learning provided by the present invention may include: a data acquisition and provision module, a neural network model module, a data post-processing and fusion module, a graphics rendering and control module, and a display module.
[0032] Specifically, the data acquisition and provision module includes one or more image sensors, one or more radar sensors, and a vehicle state perception unit. Its function is to acquire RGB image data, 3D point cloud data, and the vehicle's own attitude and position data in real time from the vehicle's surrounding environment. The data acquisition and provision module organizes the acquired data through the data preparation interface SetInfoData() and sends the RGB image data to the neural network model module through the first data sending interface SendInfoData().
[0033] The neural network model module, whose input is connected to the data acquisition and provision module, receives RGB image data sent by the first data transmission interface. Internally, it deploys a pre-trained hybrid neural network, whose structure includes an encoder path and a decoder path. The encoder path consists of multiple downsampling stages, each including at least one residual block for feature extraction and a max-pooling layer for reducing the spatial dimension of the feature map. The decoder path consists of multiple upsampling stages, each including a deconvolution layer for restoring the spatial dimension of the feature map. The output feature map of each stage in the encoder path is concatenated to the input of the corresponding stage in the decoder path via skip connections. The neural network model module performs pixel-level semantic segmentation processing on the received RGB image data to generate segmentation result data containing drivable area information, road boundary information, and general obstacle area information. This segmentation result data is then sent to the data post-processing and fusion module via the first data return interface SendResultData().
[0034] The data post-processing and fusion module, whose input is connected to the neural network model module, performs optimization processing on the segmented data and fuses multimodal data. The data post-processing and fusion module executes the filtering function `onFiltering()` to implement temporal filtering of road boundaries. This processing targets road boundary information in consecutive time frames, allowing... Indicates the timestamp The set of road boundary points identified in the frame image, let Indicates the timestamp The set of road boundary points identified in the frame image. For Each boundary point in ,exist Determine its corresponding nearest point Calculate the Euclidean distance between two points. :
[0035] ;
[0036] in, Point Two-dimensional coordinates, Indicates the nearest point Two-dimensional coordinates. The calculated distance. With a preset distance threshold Compare to generate corrected boundary points. The revised rules are as follows:
[0037] ;
[0038] in, A smooth weighting coefficient between 0 and 1. (Processed) The set constitutes stable road boundary information.
[0039] Indicates when the condition When established, it indicates the current boundary point. If the positional change is within an acceptable range, then the corrected boundary points will be... Assign the value to the current boundary point .
[0040] when conditions When not true (i.e.) ), indicating the current boundary point A jump exceeding the threshold occurred; at this point, the corrected boundary point will be... The value is assigned to the closest point in history. With the current boundary point weighted average .
[0041] The post-processing and fusion module also performs 3D morphological data fusion of general obstacles. It receives the general obstacle region mask from the segmentation result data, as well as the radar point cloud data after coordinate system registration. By traversing all radar points, it determines whether the projection position of each radar point in the image coordinate system falls within the general obstacle region mask. All radar points whose projection positions fall within the mask are filtered and aggregated to form a 3D point cloud cluster, which is the 3D morphological data of the obstacle.
[0042] The graphics rendering and control module, whose input is connected to the data post-processing and fusion module, receives stabilized road boundary information and 3D obstacle morphology data through the second data sending interface `setData()`. The graphics rendering and control module executes the 3D scene generation function `3DMaker()`, generating geometric models of the road surface and lane lines based on the stabilized road boundary information, and generating 3D models of general obstacles based on the obstacle morphology data. The graphics rendering and control module combines the generated geometric and 3D models to form a complete 3D rendering interface.
[0043] The display module, connected to the graphics rendering and control module, receives the 3D rendering interface via the 3DShow() display interface and displays it on the physical screen. The modules interact with each other through interfaces, forming a complete processing chain from data input to final screen display.
[0044] See attached document Figure 2 This invention provides a deep learning-based intelligent driving recognition method, whose overall process begins with data input and ends with the final output rendering interface, specifically including the following processing steps:
[0045] First, the system performs multimodal data acquisition and preprocessing. This process acquires continuous RGB image frame sequences from one or more image sensors and 3D point cloud data from one or more radar sensors. To ensure spatial consistency between data from different sources, the system transforms the radar point cloud data from its own radar coordinate system to the pixel coordinate system of the image sensors using a pre-calibrated intrinsic and extrinsic parameter matrix, completing spatial registration. This preprocessing step provides the data foundation for the subsequent fusion of semantic and 3D information.
[0046] Next, the system performs scene semantic segmentation based on a hybrid neural network. This process inputs pre-processed RGB image data into a pre-trained hybrid neural network model that combines U-Net and residual network structures. This hybrid neural network model performs forward inference calculations on each input frame and outputs multi-channel segmentation result data of the same size as the input image. Specifically, this segmentation result data is a set of pixel-level masks, including a drivable area mask to identify drivable areas, a road boundary mask to identify road edges, and a general obstacle area mask to identify unconventional objects.
[0047] Subsequently, the system performs post-processing and multimodal fusion on the segmentation results output from the previous process. This process includes two parallel processing branches. First, temporal filtering is performed on the road boundary mask. By comparing the boundary position information of the current frame with that of historical frames and based on a preset distance threshold, the coordinates of boundary points that have undergone positional jumps are smoothly corrected, thereby generating stable road boundary information. Second, the general obstacle area mask is spatially correlated with the radar point cloud data registered in the coordinate system. Radar points whose projection positions fall within the mask area are selected, and these points are aggregated to form one or more three-dimensional point cloud clusters, which serve as the three-dimensional morphological data of the obstacle.
[0048] Finally, the system performs integrated UI generation and display. This process inputs stabilized road boundary information, 3D morphological data of obstacles, and other contextual information (such as navigation instructions or vehicle status) into the graphics rendering and control module. Based on the input data, this module generates an integrated 3D rendered interface that includes the road surface, lane lines, 3D obstacle models, and dynamic UI elements (such as TBT steering information or selectable parking spaces). This integrated 3D rendered interface is then transmitted to the display module and presented to the user on the physical screen.
[0049] See attached document Figure 3 The first step of the deep learning-based intelligent driving recognition method provided by this invention is multimodal data acquisition and preprocessing. This step acquires raw data from multiple onboard sensors and performs spatial and temporal alignment processing on the data.
[0050] Specifically, the system acquires RGB image frame sequences in real time through one or more image sensors and simultaneously acquires 3D point cloud data of the environment through one or more radar sensors. To achieve subsequent fusion processing, the data from different sensors must be unified to the same coordinate system. In this embodiment, the core of this preprocessing step is to transform the radar point cloud data into its coordinate system, projecting it onto the pixel coordinate system of the image sensors.
[0051] This coordinate system transformation involves conversions between the radar coordinate system, vehicle coordinate system, camera coordinate system, and image pixel coordinate system. A three-dimensional point in the radar coordinate system... Its homogeneous coordinates are represented as The projection of this point onto the image pixel coordinate system. Its homogeneous coordinates are represented as The transformation relationship between the two is defined by the following formula:
[0052] ;
[0053] in, As a scaling factor; This is a 4x4 sensor extrinsic parameter matrix used to transform points from the radar coordinate system to the vehicle coordinate system. This sensor extrinsic parameter matrix consists of a rotation matrix. Translation vector The composition refers to the external parameters of the sensor. This is a 4x4 sensor extrinsic parameter matrix used to transform points from the vehicle coordinate system to the camera coordinate system. This sensor extrinsic parameter matrix consists of a rotation matrix. Translation vector The composition also belongs to the external parameters of the sensor; Let be a 3x3 camera intrinsic parameter matrix, in the following form:
[0054] ;
[0055] in, These are the focal lengths of the camera along the x-axis and y-axis of the image plane, respectively; The coordinates of the principal point of the camera's optical center in the image pixel coordinate system.
[0056] Camera intrinsic matrix and sensor extrinsic matrix All parameters were obtained through a pre-executed sensor joint calibration process and used as fixed parameters during system operation.
[0057] By performing the coordinate transformation described above, each radar point is assigned a corresponding image pixel coordinate. After completing this preprocessing step, the system obtains spatially aligned multimodal data, providing input for scene semantic segmentation and multimodal fusion processing in subsequent steps.
[0058] See attached document Figure 4The second step of the deep learning-based intelligent driving recognition method of the present invention is scene semantic segmentation based on a hybrid neural network. This step receives the preprocessed RGB image data from step one and inputs it into a pre-trained hybrid neural network model to perform forward inference computation. The purpose of the scene semantic segmentation step based on the hybrid neural network is to perform pixel-level classification on the input RGB image data to generate segmentation result data containing scene semantic information.
[0059] During execution, the hybrid neural network model processes each frame of RGB image data as input, outputting a multi-channel probability map with the same spatial dimensions as the input image. Each channel of this multi-channel probability map corresponds to a predefined semantic category, such as a drivable area, road boundary, or general obstacle. In this probability map, the value of each pixel on a specific channel represents the probability that the pixel belongs to the semantic category corresponding to that channel.
[0060] To enable the hybrid neural network model to accurately segment different semantic categories, this hybrid neural network model uses a combined loss function during the training phase. Optimization is performed. The combined loss function consists of cross-entropy loss. and Dice loss The weighted summation is defined as follows:
[0061] ;
[0062] in, and These are the weighting coefficients used to adjust the contributions of the two loss components, respectively.
[0063] Cross-entropy loss This is used to measure the difference between the pixel class probability distribution predicted by the model and the true label. For a single pixel in an image, the cross-entropy loss is calculated using the following formula:
[0064] ;
[0065] in, This represents the total number of predefined semantic categories; Represents a number from 1 to... The category index to be iterated; Represents pixels The true label is a component of a one-hot encoded vector, when the pixel... The true category is When the time condition is met, its value is 1; otherwise, it is 0. Represents the pixel predicted by the neural network model. Category The probability value; Represents the natural logarithm function; This represents the summation operator, which sums the expression... For all categories From 1 to The results are then summed.
[0066] Dice loss The Dice loss is used to measure the overlap between the segmented regions predicted by the model and the ground truth label regions. For a specific semantic category, the Dice loss is calculated as follows:
[0067] ;
[0068] in, This represents the total number of pixels in an image; This represents the summation of all pixels in the entire image after multiplying the model's predicted probability map for category k with the true label map pixel by pixel. This result is related to the area of intersection between the two regions. This represents the summation of the squared pixel values in the model's predicted probability map for category k over all pixels in the entire image. This represents the summation of the squared pixel values of each pixel in the true label image of category k over all pixels in the entire image.
[0069] Through the aforementioned forward inference calculation, the scene semantic segmentation step based on the hybrid neural network finally generates segmentation result data containing drivable area masks, road boundary masks, and general obstacle area masks, and transmits it to subsequent steps for processing.
[0070] In the intelligent driving recognition method based on deep learning provided by this invention, a specific implementation of step three involves performing temporal filtering on road boundary information. This process receives segmentation result data representing road boundaries generated by the neural network model in step two, and performs temporal smoothing processing on it to generate positionally stable road boundary information for subsequent rendering.
[0071] Specifically, the process begins with the current frame (timestamp: Extract the set of road boundary points from the segmentation results data. And get the previous frame (timestamp is...) The set of road boundary points after filtering For the current set of frames Each boundary point in In the set of the previous frame In this process, the unique corresponding point is determined by calculating the minimum distance. .
[0072] After determining the correspondence, calculate the boundary points of the current frame. Its corresponding point in the previous frame Euclidean distance between .
[0073] Then, the calculated Euclidean distance Distance threshold Compare the points and adjust the current boundary points accordingly. The coordinates are corrected to generate the final, stable boundary points for the current frame. .
[0074] By performing the above processing on all road boundary points in the current frame, the system generates a complete and stable set of road boundary points. The set of road boundary points is then transferred to the graphics rendering and control module to generate a road and lane line model without obvious abrupt changes.
[0075] In the deep learning-based intelligent driving recognition method provided by this invention, another specific implementation of step three is to perform fusion processing of general obstacle 3D morphology data. The purpose of this processing is to combine semantic information at the 2D image level with point cloud data in 3D space to generate obstacle data with 3D morphology.
[0076] The process receives two main inputs: the first is the universal obstacle region mask generated in step two; the second is the radar point cloud data after coordinate system registration in step one. The universal obstacle region mask is a two-dimensional data matrix, where each element identifies whether the corresponding pixel in the image belongs to the universal obstacle region.
[0077] This fusion process involves traversing every three-dimensional radar point in the radar point cloud data. To achieve this. Each 3D radar point Each has a three-dimensional spatial coordinate system, and corresponds to a projected pixel coordinate on the image plane calculated in step one. .
[0078] The system uses the projected pixel coordinates Query the general obstacle region mask in image pixel coordinates The value at the specified location. A filtering condition is applied: the general obstacle region mask is at the specified pixel coordinates in the image. The value at that location indicates when the pixel belongs to an obstacle region, which is the value of that 3D radar point. Radar points identified as valid obstacles are retained. All radar points whose projected locations do not fall within the general obstacle area mask are discarded.
[0079] All 3D radar points identified as valid obstacles are aggregated to form a new 3D point cloud. This new point cloud constitutes the obstacle's 3D morphological data. This data not only includes the obstacle's 3D spatial location but also reflects its outline and shape through the distribution of the point cloud. This 3D morphological data is then transmitted to the graphics rendering and control module to generate a 3D model of the obstacle.
[0080] See attached document Figure 5 The fourth step of the method in this embodiment of the invention is integrated UI generation and display. In a specific implementation scenario, this step achieves the fusion display of driving scene and navigation information.
[0081] This step begins with the graphics rendering and control module generating a main 3D rendering interface based on the stabilized road boundary information and obstacle morphology data generated in step three. This main 3D rendering interface displays a real-time 3D model of the road, lane lines, and obstacles in front of the vehicle.
[0082] Simultaneously, the system acquires the current vehicle operating status and determines whether a navigation status flag is active. When the navigation status flag is active, it indicates that the vehicle is currently navigating a route. At this time, the graphics rendering and control module receives navigation command data from a navigation module.
[0083] The navigation instruction data specifically includes: a TBT (Turn-by-Turn) information data packet and a navigation thumbnail data packet. The TBT information data packet contains a turn type identifier, a remaining distance value to the turn point, and a road name string. The navigation thumbnail data packet contains two-dimensional graphic data representing the vehicle's current position and the surrounding road network.
[0084] The graphics rendering and control module generates a TBT steering information element within a preset first screen space area of the main 3D rendering interface, based on the TBT steering information data package. This module selects and draws the corresponding steering arrow icon based on the steering type identifier, and then displays the remaining distance value and road name string next to the icon.
[0085] Simultaneously, the graphics rendering and control module generates a navigation thumbnail element within a preset second screen space area of the main 3D rendering interface, based on the navigation thumbnail data package. This module maps the 2D graphics data as a texture onto a 2D rectangular primitive for rendering.
[0086] The generation process of the aforementioned TBT steering information elements and navigation thumbnail elements is dynamically repeated as the navigation command data is updated in real time. Finally, the graphics rendering and control module combines the main 3D rendering interface, TBT steering information elements, and navigation thumbnail elements into a final display frame, and outputs it to the display module through the display interface, realizing the integrated display of driving environment perception and navigation information within a single interface.
[0087] The fourth step of the method in this embodiment of the invention is integrated UI generation and display. In another specific implementation scenario, this step achieves the integrated display of driving scenes and parking assistance functions.
[0088] In this scenario, while maintaining the real-time display of the main 3D rendering interface, the system analyzes environmental data through the parking space detection module to identify available parking spaces. When one or more available parking spaces are detected, the parking space detection module outputs a parking space dataset containing the location, orientation, and type information of each available parking space.
[0089] The graphics rendering and control module receives a set of parking space data and generates a corresponding selectable parking space element for each parking space in the set. To determine the drawing position of this element on the screen, the center point of the parking space in the world coordinate system or vehicle coordinate system needs to be determined. Project onto the screen coordinate system. This projection transformation is defined by the following formula:
[0090] ;
[0091] in, Represents a scale factor; This represents the homogeneous coordinates of the center point of the parking space in the screen coordinate system. This represents a 3x3 camera intrinsic parameter matrix; This represents a 4x4 sensor extrinsic matrix used to transform points from the world coordinate system to the camera coordinate system. This sensor extrinsic matrix is a pre-calibrated camera extrinsic matrix. Indicates the center point of the parking space Homogeneous coordinates in the world coordinate system.
[0092] The graphics rendering and control module calculates the image pixel coordinates. Draw optional parking space elements overlay at this location.
[0093] The system continuously monitors user input events. When a user selects an available parking space via a touchscreen or other input device, the system records the unique identifier of the selected parking space. Upon receiving confirmation of the selection, the graphics rendering and control module switches the interface state. This module will highlight the 3D space area corresponding to the selected parking space identifier in the main 3D rendering interface, and calculate and draw a recommended trajectory line to guide the vehicle into the parking space based on the vehicle's current posture and the target parking space location.
[0094] Finally, the integrated display frame, consisting of the main 3D rendering interface, highlighted parking spaces, and recommended trajectory lines, is transmitted to the display module for presentation, providing users with intuitive parking guidance.
[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1.A deep learning-based intelligent driving recognition system, characterized in that, The system comprises a data acquisition and provision module, a neural network model module, a data post-processing and fusion module, a graphics rendering and control module, and a display module. The data acquisition and provision module is configured to acquire RGB image data and three-dimensional point cloud data, and send the RGB image data to the neural network model module, and send the three-dimensional point cloud data to the data post-processing and fusion module after coordinate system registration. The neural network model module is configured to receive the RGB image data, perform pixel-level semantic segmentation on the RGB image data using a hybrid neural network, generate segmentation result data containing drivable area information, road boundary information, and general obstacle area information, and send the segmentation result data to the data post-processing and fusion module. The data post-processing and fusion module is configured to receive the segmentation result data and the three-dimensional point cloud data after coordinate system registration, perform time-series filtering on the road boundary information to generate stabilized road boundary information, fuse the general obstacle area information with the three-dimensional point cloud data to generate three-dimensional shape data of obstacles, and send the stabilized road boundary information and the three-dimensional shape data of obstacles to the graphics rendering and control module. The graphics rendering and control module is configured to receive the stabilized road boundary information and the three-dimensional shape data of obstacles, generate a road surface geometric model, a lane line geometric model, and a general obstacle three-dimensional model, combine to generate a three-dimensional rendering interface, and send the three-dimensional rendering interface to the display module. The display module is configured to receive and present the three-dimensional rendering interface. 2.The intelligent driving recognition system based on deep learning of claim 1, wherein, The hybrid neural network comprises an encoder path and a decoder path. The encoder path comprises a plurality of down-sampling stages, each of which comprises one or more residual blocks for feature extraction and a max-pooling layer for reducing the spatial dimension of a feature map. The decoder path comprises a plurality of up-sampling stages, each of which comprises a deconvolution layer for restoring the spatial dimension of a feature map. The feature map output by each down-sampling stage in the encoder path is spliced to the input end of the corresponding up-sampling stage in the decoder path through a skip connection. 3.The intelligent driving recognition system based on deep learning of claim 1, wherein, The hybrid neural network is optimized using a combined loss function in the training stage, which is composed of a cross-entropy loss and a Dice loss. The cross-entropy loss is used to measure the difference between the pixel class probability distribution predicted by the model and the true label. The Dice loss is used to measure the degree of overlap between the segmentation region predicted by the model and the true label region. 4.The intelligent driving recognition system based on deep learning of claim 1, wherein, When the data acquisition and provision module or the data post-processing and fusion module performs coordinate system registration, the three-dimensional point cloud data is transformed from a radar coordinate system to a vehicle coordinate system through a sensor extrinsic parameter matrix, then from the vehicle coordinate system to a camera coordinate system, and finally projected to an image pixel coordinate system through a camera intrinsic parameter matrix, achieving the alignment of the three-dimensional point cloud data and the RGB image data in space. 5.The intelligent driving recognition system based on deep learning of claim 1, wherein, The data post-processing and fusion module, when performing timing filtering processing, first determines a boundary point in the road boundary information of a current frame, and determines a corresponding nearest point in the road boundary information of a previous frame after filtering processing; The Euclidean distance between the boundary point and the nearest point is calculated by performing square root operation on the sum of squares of the coordinate values of the boundary point and the nearest point, and the Euclidean distance is compared with a preset distance threshold value; When the Euclidean distance is less than or equal to the distance threshold value, the boundary point is determined as a corrected boundary point; When the Euclidean distance is greater than the distance threshold value, a weighted average value of the nearest point and the boundary point is calculated as the corrected boundary point. 6.The intelligent driving recognition system based on deep learning of claim 1, wherein, When the data post-processing and fusion module generates the three-dimensional shape data of the obstacle, each radar point in the three-dimensional point cloud data is traversed, the radar point is transformed from a radar coordinate system to a vehicle coordinate system and a camera coordinate system by using a sensor extrinsic matrix, and the transformed radar point is projected to an image pixel coordinate system by using a camera intrinsic matrix to calculate the projection pixel coordinates of each radar point on an image plane; The value corresponding to the projection pixel coordinates in the general obstacle region information is queried; When the value indicates that the corresponding pixel belongs to an obstacle region, the radar point is retained; All retained radar points are aggregated to form a three-dimensional point cloud cluster as the three-dimensional shape data of the obstacle. 7.The intelligent driving recognition system based on deep learning of claim 1, wherein, The graphic rendering and control module is further configured to receive navigation instruction data, the navigation instruction data including a TBT steering information data packet and a navigation thumbnail data packet; The graphic rendering and control module generates a TBT steering information element in a first screen space region in the three-dimensional rendering interface according to the TBT steering information data packet, and generates a navigation thumbnail element in a second screen space region in the three-dimensional rendering interface according to the navigation thumbnail data packet. 8.The intelligent driving recognition system based on deep learning of claim 1, wherein, The intelligent driving recognition system further includes a parking space detection module configured to output a parking space data set containing parking space position information; the graphic rendering and control module receives the parking space data set, projects the parking space position information to a screen coordinate system by using a camera intrinsic matrix and a sensor extrinsic matrix, and draws a selectable parking space element in the three-dimensional rendering interface. 9.The intelligent driving recognition system based on deep learning of claim 8, wherein, The system is further configured with an input device configured to monitor a selection operation on the selectable parking space element; When the selection operation is received, the graphic rendering and control module highlights a three-dimensional space region corresponding to the selected parking space, and calculates and draws a recommended trajectory line for guiding the vehicle to enter the selected parking space according to the current attitude of the vehicle and the position of the selected parking space. 10.A method for intelligent driving recognition based on deep learning, characterized in that, The intelligent driving recognition system based on deep learning is applied to any one of claims 1-9, comprising the following steps: S1, acquiring RGB image data and three-dimensional point cloud data by a data acquisition and providing module, and performing coordinate system registration on the three-dimensional point cloud data; S2, pixel-level semantic segmentation of the RGB image data is performed by a hybrid neural network through a neural network model module to generate segmentation result data containing drivable area information, road boundary information, and general obstacle region information; S3, time-series filtering processing is performed on the road boundary information through a data post-processing and fusion module to generate stabilized road boundary information, and the general obstacle region information is fused with the registered three-dimensional point cloud data to generate three-dimensional shape data of obstacles; S4, geometric models and three-dimensional models are generated based on the stabilized road boundary information and the three-dimensional shape data of obstacles through a graphics rendering and control module, and a three-dimensional rendering interface is generated by combination; S5, the three-dimensional rendering interface is presented on a physical screen through a display module.
Citation Information
Cited By
Self-adaptive parking method and system based on hierarchical reinforcement learning and kinematics constraint
CN122058902A
