Complex road condition navigation method and system, electronic equipment and computer readable medium

Abstract

The invention provides a complex road condition navigation method and system, electronic equipment and a readable medium, and belongs to the field of intelligent driving. Video image data and laser point cloud data in front of a driving vehicle are collected; forming a color feature matrix and a position coordinate feature matrix according to the video image data; forming a distance characteristic matrix and a physical property characteristic matrix according to the laser point cloud data; performing fusion processing on the color feature matrix and the position feature matrix to determine an object region position coordinate set; performing fusion processing on the distance feature matrix and the physical property feature matrix to extract an object contour position coordinate matrix; forming a target contour matrix according to the object region position coordinate set and the object contour position coordinate matrix; according to the method, two kinds of image data are collected for fusion calculation, the calculated driving obstacle contour serves as a display object and is presented to a user, and accurate and feasible driving route guiding service can be provided for a driver in a complex road condition environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a navigation method, system, electronic device, and computer-readable medium for complex road conditions. Background Technology

[0002] Currently, navigation systems primarily rely on big data analysis to identify and predict road congestion when providing route planning and navigation functions. This helps drivers choose the best routes and avoid traffic jams, which is very helpful for daily driving.

[0003] Because navigation systems are built on big data analytics, primarily using vehicle driving data and road network data, they can analyze traffic congestion in real time and provide route suggestions to drivers. However, relying solely on big data analytics is insufficient to handle all road conditions. On non-main roads in cities, pedestrians, non-motorized vehicles, and mobile vendors occupy the roads, leading to complex and unpredictable traffic conditions, frequent unexpected traffic accidents, and easy congestion. Furthermore, these road-occupying groups cannot upload real-time road condition data, meaning the navigation system cannot accurately reflect the real-time conditions of these roads. This can cause the navigation system to overlook these congested areas when planning routes, resulting in unreasonable route planning and leading drivers onto congested roads.

[0004] This demonstrates that big data analysis in navigation systems greatly facilitates route planning and navigation. However, in complex urban environments, drivers still need to remain vigilant and combine their driving experience and judgment to deal with unforeseen emergencies. Therefore, how to effectively plan routes to avoid traffic congestion on complex non-main roads in cities has become an urgent problem to be solved. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a navigation method, system, electronic device and computer-readable medium for complex road conditions.

[0006] In a first aspect, embodiments of the present invention provide a navigation method for complex road conditions, comprising the following steps:

[0007] Collect video image data and laser point cloud data in front of the vehicle;

[0008] A color feature matrix and a position coordinate feature matrix are formed based on the video image data;

[0009] A distance feature matrix and a physical property feature matrix are formed based on the laser point cloud data;

[0010] The color feature matrix and the position feature matrix are fused together to determine the set of position coordinates of the object region.

[0011] The distance feature matrix and the physical property feature matrix are fused together to extract the object contour position coordinate matrix;

[0012] A target contour matrix is ​​formed based on the set of object region position coordinates and the object contour position coordinate matrix;

[0013] The target contour matrix is ​​displayed in the environment ahead of the driving path.

[0014] In some embodiments, displaying the target contour matrix in the environment ahead of the driving path includes: classifying the target contour matrix and selectively displaying it according to the classification.

[0015] In some embodiments, the classification includes objects higher than the vehicle body belonging to the safety category, which are not displayed; obstacles outside the vehicle body edge extension line and within the vehicle body height range belonging to the threat category, which are included in the display candidate range; and obstacles within the vehicle body edge extension line and within the vehicle body height range belonging to the danger category, which are displayed.

[0016] In some embodiments, including the threat object in the display candidate range includes: including the outline of the object on the side of the vehicle body in the display candidate range, and using the angle between the outline and the side of the vehicle body as the judgment criterion, selecting a preset angle as the threshold value, and displaying the outline portion with an angle greater than or equal to the preset angle with the side of the vehicle body.

[0017] In some embodiments, obstacle contour features are also included to display the area below the driving plane.

[0018] In some embodiments, the step of displaying the obstacle contour features of the area below the driving plane includes:

[0019] Extract geometric feature parameters from the target contour moments;

[0020] Construct a passability judgment condition matrix based on the vehicle body geometric dimension model parameters;

[0021] Import the geometric feature parameters of objects below the driving plane into the passability judgment condition matrix;

[0022] Based on the passability assessment, determine whether the area below this driving plane will cause the vehicle to scrape or get stuck;

[0023] If so, the user experience will be presented through methods such as driving route avoidance and displaying the outline of roadblocks; otherwise, it cannot be passed.

[0024] In some embodiments, the accessibility determination conditions include:

[0025] When the span of the depression is no greater than the threshold value of 1, the vehicle can pass through the depression normally;

[0026] When the span is greater than threshold 1 but not greater than threshold 2, the tires lose traction, the vehicle body sinks into the recessed area, and driving is obstructed.

[0027] When the span is greater than threshold 2, the vehicle body tilts, and when the downhill angle of the recessed area is less than threshold 3, the tires can smoothly reach the bottom of the recessed area, the front bumper of the vehicle will not scrape, and the vehicle can pass normally; otherwise, the front bumper of the vehicle will scrape against the bottom of the recessed area, and driving will be obstructed.

[0028] Secondly, embodiments of the present invention provide a system for the complex road condition navigation method described above, comprising:

[0029] The acquisition unit is used to acquire video image data and laser point cloud data in front of the vehicle.

[0030] The first matrix unit is used to form a color feature matrix and a position coordinate feature matrix based on the video image data;

[0031] The second matrix unit is used to form a distance feature matrix and a physical property feature matrix based on the laser point cloud data;

[0032] A fusion unit is used to fuse the color feature matrix and the position coordinate feature matrix to determine the object region position coordinate set.

[0033] The extraction unit is used to fuse the distance feature matrix and the physical property feature matrix to extract the object contour position coordinate matrix;

[0034] The target unit is used to form a target contour matrix based on the set of position coordinates of the object region and the object contour position coordinate matrix;

[0035] The display unit is used to display the target contour matrix in the environment ahead of the driving path.

[0036] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0037] One or more processors;

[0038] Memory, used to store one or more programs;

[0039] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods.

[0040] Fourthly, embodiments of the present invention also provide a computer-readable medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described.

[0041] The complex road condition navigation method provided by this invention collects video image data and laser point cloud data ahead of the vehicle; forms a color feature matrix and a position coordinate feature matrix based on the video image data; forms a distance feature matrix and a physical property feature matrix based on the laser point cloud data; fuses the color feature matrix and the position coordinate matrix to determine the object region position coordinate set; fuses the distance feature matrix and the physical property feature matrix to extract the object contour position coordinate matrix; forms a target contour matrix based on the object region position coordinate set and the object contour position coordinate matrix; and displays the target contour matrix in the environment ahead of the driving path. This invention, by collecting two types of image data and performing fusion calculations, presents the calculated driving obstacle contours as the display objects to the user, enabling accurate and feasible driving route guidance services for drivers in complex road conditions. It effectively avoids problems such as low functional recognition and poor functional usability caused by information redundancy and interference. Attached Figure Description

[0042] Figure 1 This is a flowchart of one embodiment of the complex road condition navigation method of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the principle of one embodiment of the complex road condition navigation method of the present invention;

[0044] Figure 3 This is a schematic diagram of an embodiment of the present invention for classifying the outlines of target objects in the environment;

[0045] Figure 4 This is a flowchart illustrating the steps of an embodiment of the present invention for classifying and displaying three major categories of objects;

[0046] Figure 5 This is a flowchart illustrating the steps of an embodiment of the passability determination and inspection process navigation method of the present invention;

[0047] Figure 6 This is a schematic diagram of a structure of an embodiment of the passability judgment check provided by the present invention;

[0048] Figure 7 This is a schematic diagram of the structure of an embodiment of the complex road condition navigation system provided by the present invention;

[0049] Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0051] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0052] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0053] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0054] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0055] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0056] In related technologies, navigation systems are built upon big data analytics, primarily using vehicle driving data and road network data. This allows for real-time analysis of traffic congestion and provides route suggestions to drivers. However, relying solely on big data analytics is insufficient to handle all road conditions. On non-main roads in cities, pedestrians, non-motorized vehicles, and mobile vendors occupy the roads, leading to complex and unpredictable traffic conditions, frequent unexpected traffic accidents, and a high risk of congestion. Furthermore, these road-occupying groups do not upload real-time road condition data, preventing navigation systems from accurately reflecting the real-time situation. This can cause navigation systems to overlook these congested areas when planning routes, resulting in unreasonable route planning and leading drivers onto congested roads.

[0057] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a navigation method for complex road conditions. Figure 1 and Figure 2 The present invention provides a schematic diagram and principle diagram of a complex road condition navigation method, which includes the following steps S10 to S70. The implementation of each step is described in detail below.

[0058] Step S10: Collect video image data and laser point cloud data in front of the vehicle.

[0059] In this embodiment, machine vision methods such as cameras and LiDAR scanning can capture the outlines of objects related to the driving route in the driving environment, replacing street scene video, reducing information redundancy and interference, and focusing on displaying the target. In complex road conditions, such as forests, busy urban areas, and complex underground parking garages, it provides drivers with accurate and feasible driving route guidance services. Cameras, as a fundamental capability of machine vision, can extract information such as color and brightness from images of the driving environment. LiDAR can measure the distance between different objects and between objects and the vehicle itself by measuring the time difference between emitted and reflected light. Furthermore, the material properties of various objects can be distinguished by the differences in reflectivity of their surfaces. By using a LiDAR array for scanning, the scanning range is expanded, scanning efficiency is improved, and real-time, large-area environmental data acquisition of light time difference and reflectivity is achieved.

[0060] Step S20: Form a color feature matrix and a position coordinate feature matrix based on the video image data.

[0061] It is understandable that video image data includes information such as color and brightness of the driving environment. The image data processing unit organizes the color and brightness data in the image according to the matrix data format. By extracting the color region distribution information in the color matrix and classifying the same or similar colors, various objects in the environment can be identified, and their regional location coordinates can be determined, such as the sky, ground, pedestrians, vehicles, obstacles, etc., thereby obtaining the color feature matrix and the location coordinate feature matrix.

[0062] Step S30: Form a distance feature matrix and a physical property feature matrix based on the laser point cloud data.

[0063] LiDAR (Light Detection and Ranging) can measure the distance between different objects and between an object and the surrounding workshop by analyzing the time difference between emitted and reflected light. Furthermore, it can distinguish the physical properties of various objects by observing the differences in reflectivity of their surfaces. By using a LiDAR array for scanning, the scanning range is expanded, scanning efficiency is improved, and real-time, large-area environmental data acquisition of light time difference and reflectivity is achieved. The LiDAR array sends the acquired data to a LiDAR data processing unit. The LiDAR data processing unit integrates and calculates the data to obtain the distance and physical properties of all objects, thereby acquiring the distance feature matrix and the physical property feature matrix.

[0064] Step S40: The color feature matrix and the position coordinate feature matrix are fused to determine the object region position coordinate set.

[0065] Specifically, color information (such as RGB, HSV, Lab values) of each pixel / region in the image is extracted from the color feature matrix to form matrix C, with dimensions (H, W, D_c) (H = height, W = width, D_c = number of color channels). Texture, edge, shape, or material features (such as LBP, HOG, SIFT, or CNN features) are extracted from the material property feature matrix to form matrix P, with dimensions (H, W, D_p) (D_p = material property feature dimension).

[0066] Specifically, (1) Early fusion: directly splice color and physical property features to generate fusion matrix F; (2) Mid-term fusion (feature-level fusion): perform feature transformation on C and P respectively (such as PCA dimensionality reduction or CNN encoding), and then fuse; (3) Late-term fusion (decision-level fusion): independently process the two feature matrices, generate their respective object region proposals, and then fuse the results.

[0067] It is understandable that by fusing the color matrix and the position coordinate feature matrix, different colors can be incorporated into the same object, thus accurately extracting the color of the entire area of ​​the object, thereby determining the size and position coordinate range of the object in the image.

[0068] Step S50: The distance feature matrix and the material property feature matrix are fused to extract the object contour position coordinate matrix.

[0069] It can be understood that the distance feature matrix has the shape `[N, H, W, C_d]` or `[N, P, C_d]`, where: `N` is the batch size; `H, W` are the spatial dimensions (if it is an image or a grid); `P` is the number of points (if it is a point cloud); and `C_d` is the number of feature channels, which includes coordinate information (such as x, y, z), distance information (such as the distance to a reference point), etc.

[0070] The physical property feature matrix has the shape `[N, H, W, C_m]` or `[N, P, C_m]`, where `C_m` is the number of channels for physical properties such as density, elastic modulus, color, and texture.

[0071] The distance feature matrix and the physical property feature matrix are fused. Since the two are spatially aligned (i.e., the distance feature and physical property feature at each location correspond to the same spatial location), they can be concatenated and fused along the feature channel dimension to form a fused feature matrix, thereby obtaining the object contour position coordinate matrix.

[0072] It's understandable that the distance feature matrix provides the initial position of an object (such as a bounding box), while the distance matrix provides the spatial relationships between objects (such as adjacency, overlap, etc.). Using the distance matrix to group objects (clustering) merges multiple objects belonging to the same group, combining the bounding boxes of all objects within the group to obtain a new contour (usually a larger or more complex contour). After clustering and merging, K groups are obtained (each group corresponds to a merged contour). The contour position coordinate matrix is ​​then the set of coordinates for these K contours.

[0073] Step S60: Form a target contour matrix based on the object region position coordinate set and the object contour position coordinate matrix.

[0074] Specifically, the set of coordinates of the object region is usually a set of bounding boxes, a list in the format of [x_min, y_min, x_max, y_max], with each element corresponding to an object.

[0075] Specifically, the object contour position coordinate matrix is ​​a three-dimensional matrix (number of objects × number of contour points × 2) for each extracted object contour point, or a list where each element is a two-dimensional array (number of contour points × 2).

[0076] The target contour matrix is ​​formed based on the existing set of object region location coordinates (bounding box) and the object contour location coordinate matrix (each object consists of multiple contour points), specifically including:

[0077] 1. Data preparation: Ensure that the two input data are consistent in the number of objects and correspond in order (i.e., the i-th bounding box corresponds to the i-th contour).

[0078] 2. Contour point correction: Since contour points may exceed the bounding box or be inaccurate, we can use the bounding box to constrain and correct the contour points.

[0079] 3. Outline point sorting: Ensure that the outline points are ordered (clockwise or counterclockwise) to facilitate subsequent processing (such as drawing, calculating area, etc.).

[0080] 4. Contour point simplification (optional): If there are too many contour points, simplification can be performed to reduce the amount of calculation.

[0081] 5. Form the target contour matrix: Store the contour points of each object as a two-dimensional array, and all objects form a three-dimensional matrix.

[0082] It is understood that this invention aims to integrate two machine vision technologies to accurately depict the outer contours of objects associated with a driving route. Then, through machine computation, it accurately identifies the available driving path ranges between various objects and provides driving guidance services to the driver via guide arrows superimposed on the contour layer. Compared to guidance services superimposed on AR navigation, guidance services superimposed on the contour layer have higher recognition accuracy and a stronger perceptual experience. This advantage is particularly pronounced in complex road conditions where the road ahead is unknown and obstacles are numerous.

[0083] Step S70: Display the target contour matrix in the environment ahead of the driving path.

[0084] It is understood that this invention uses an in-vehicle forward-facing camera in conjunction with an array-scanning LiDAR mounted on the roof to collect video image data of the road ahead and laser reflection data from surfaces of objects along the driving route. The data is then sent to an image data processing unit and a radar data processing unit. The image data processing unit integrates the color and position coordinate data of the image pixel matrix. The radar data processing unit integrates the distance and physical property data of the objects. These feature parameters are stored in matrix form, forming a color feature matrix, a position coordinate feature matrix, a distance feature matrix, and a physical property feature matrix. The color matrix and position coordinate feature matrix are fused, combining different colors into the same object to accurately extract the color of the entire object area. This determines the size and position coordinate range of the object in the image. The distance feature and physical property feature matrix are fused to extract the object contour position coordinate matrix and the target contour matrix. Finally, the target contour matrix data is displayed visually through the instrument panel and head-up display, clearly showing the contours of objects that pose a threat or obstacle to the driving path and their distances relative to the vehicle body. This invention can not only effectively identify conventional objects such as pedestrians, buildings, and the sky in front of a vehicle, but also accurately identify unconventional and irregular structural objects such as fire hydrants, ground piles, wells, and stairs.

[0085] In this embodiment, the following steps are also included: classifying the target contour matrix and selectively displaying it according to the classification.

[0086] It is understood that this embodiment uses object outlines instead of real-world street video to reduce information redundancy and interference, focus on displaying the target, and provide a simple and intuitive human-computer interaction presentation that goes beyond the vehicle's visual range. In complex road conditions, such as forests, busy urban areas, and complex underground parking garages, it provides drivers with accurate and feasible route guidance services.

[0087] Please see Figure 3 and Figure 4To control the amount of displayed information and avoid negative impacts on the functional presentation due to information redundancy, the outlines of target objects in the environment are classified into three categories: safe, threat, and danger. Objects higher than the vehicle body belong to the safe category, such as the sky. Obstacles within the vehicle body's height but beyond the vehicle body's edge are considered threat objects. Obstacles within the vehicle body's height but within the vehicle body's edge are considered danger objects. To avoid information interference caused by display redundancy, which weakens the presentation of the main target object and prevents users from capturing the target object in the displayed information within the necessary time, thus affecting the functional perception experience, the following strategy is proposed to classify the above three categories of objects for display presentation: Safe objects do not pose a threat or obstruction to driving and will not be displayed. Threat objects pose a potential threat to both the front and sides of the vehicle body and are included in the display candidate range. To control display information redundancy and highlight key information, the display strategy only includes the outline of the object closest to the vehicle body in the display candidate range, rather than the complete outline of the object. Outlines facing away from the vehicle body are not driving safety factors and are not included in the strategy data range. For object outlines near the vehicle body, the display strategy is further optimized and emphasized. Using the angle between the outline and the vehicle side as the criterion, a preset angle is selected as the threshold value, for example, 5 degrees. Outlines with an angle greater than or equal to 5 degrees to the vehicle side are sent to the instrument cluster and HUD display for presentation. In this way, the displayed information will only show the parts of the object outline most relevant to driving safety, highlighting key points and clearly indicating the intended message. (e.g.) Figure 4 Since hazardous objects are located within the vehicle's coordinate range and constitute direct obstacles on the driving route, posing a direct danger to driving safety, their complete outlines will be sent to the instrument cluster and HUD display for presentation. Objects along the driving route will be categorized and displayed according to their relevance to route accessibility, effectively controlling the amount of information displayed, highlighting key elements, enhancing functional recognition, and providing users with an intuitive and efficient functional experience.

[0088] Furthermore, it also includes the following steps: further displaying the obstacle contour features in the area below the driving plane, specifically including:

[0089] Extract geometric feature parameters from the target contour moments.

[0090] A passability judgment condition matrix is ​​constructed based on the vehicle body geometric dimension model parameters.

[0091] Import the geometric feature parameters of objects below the driving plane into the passability judgment condition matrix.

[0092] Based on the passability assessment, determine whether the area below the driving plane will cause the vehicle to scrape or get stuck.

[0093] If so, the user experience will be presented through methods such as driving route avoidance and displaying the outline of roadblocks; otherwise, it cannot be passed.

[0094] Please see Figure 5 To replace visual observation in complex terrain environments and formulate accurate and reliable driving route guidance strategies, in addition to identifying and displaying the outlines of dangerous obstacles above the vehicle's driving plane, it is also necessary to identify and display the outline features of obstacles below the driving plane. Examples include steep slopes, wells, deep pits, ditches, and ravines. Unlike above-level obstacles, which focus on the potential for scrapes or collisions to the front and sides of the vehicle, below-level obstacles focus on the damage or accessibility barriers caused by dented objects to the vehicle's undercarriage or wheels. Distance feature parameters of the area below the driving plane are collected using LiDAR, and position coordinate feature parameters are collected using a camera. These two are fused to obtain an object outline coordinate matrix. Geometric feature parameters such as length, width, and depth of the object are extracted from the outline coordinate matrix. A accessibility judgment condition matrix is ​​constructed using the vehicle's geometric dimensions. The geometric feature parameters of objects below the driving plane are imported into the accessibility judgment condition matrix to perform accessibility checks, determining whether the area below the driving plane will cause scrapes or entrapment of the vehicle. The user experience is then presented through driving route avoidance and obstacle outline display.

[0095] Please see Figure 6 The passability assessment conditions are detailed below. When the span of the depression is less than threshold 1, the tires retain traction, and the vehicle can pass through the depression normally. When the span is greater than threshold 1 but less than threshold 2, the tires lose traction, the vehicle sinks into the depression, and driving is obstructed. When the span is greater than threshold 2 but less than infinity, the vehicle tilts. In this scenario, the influence of the downhill angle parameter of the depression must be considered. When the downhill angle of the depression is less than threshold 3, the tires can smoothly reach the bottom of the depression, the front bumper will not scrape, and the vehicle can pass normally. Otherwise, the front bumper will scrape against the bottom of the depression, and driving will be obstructed.

[0096] This embodiment improves the accuracy and reliability of route guidance in complex road conditions by highlighting the outlines of objects related to the driving route in a real-world driving environment, ignoring irrelevant information such as the object's color, material, and expression, and shielding unrelated objects. This invention effectively avoids problems such as low functional recognition and poor functional usability caused by information redundancy and interference.

[0097] The complex road condition navigation method provided by this invention collects two types of image data and performs fusion calculations. The calculated outline of driving obstacles is then presented to the user. This method can provide drivers with accurate and feasible driving route guidance services in complex road conditions, effectively avoiding problems such as low functional recognition and poor functional usability caused by information redundancy and interference.

[0098] This invention also provides a navigation system for complex road conditions. Figure 7 The diagram shows the structure of a system provided in an embodiment of the present invention, which is applied to the complex road condition navigation method provided in the above embodiment. Specifically, it includes: a data acquisition unit, a first matrix unit, a second matrix unit, a fusion unit, an extraction unit, a target unit, and a display unit.

[0099] The acquisition unit is used to acquire video image data and laser point cloud data in front of the vehicle.

[0100] Specifically, machine vision methods such as cameras and LiDAR scanning can capture the outlines of objects related to the driving route in the driving environment, replacing street scene video, reducing information redundancy and interference, and focusing on displaying targets. In complex road conditions, such as forests, busy urban areas, and complex underground parking garages, accurate and feasible route guidance services can be provided to drivers. Cameras, as a fundamental capability of machine vision, can extract information such as color and brightness from images of the driving environment. LiDAR can measure the distance between different objects and between objects and the vehicle itself by measuring the time difference between emitted and reflected light. Furthermore, the material properties of various objects can be distinguished by the differences in reflectivity of their surfaces. By using LiDAR array scanning, the scanning range is expanded, scanning efficiency is improved, and real-time, large-area environmental data acquisition of light time difference and reflectivity is achieved.

[0101] The first matrix unit is used to form a color feature matrix and a position coordinate feature matrix based on the video image data.

[0102] It is understandable that video image data includes information such as color and brightness of the driving environment. The image data processing unit organizes the color and brightness data in the image according to the matrix data format. By extracting the color region distribution information in the color matrix and classifying the same or similar colors, various objects in the environment can be identified, and their regional location coordinates can be determined, such as the sky, ground, pedestrians, vehicles, obstacles, etc., thereby obtaining the color feature matrix and the location coordinate feature matrix.

[0103] The second matrix unit is used to form a distance feature matrix and a physical property feature matrix based on the laser point cloud data.

[0104] LiDAR (Light Detection and Ranging) can measure the distance between different objects and between an object and the surrounding workshop by analyzing the time difference between emitted and reflected light. Furthermore, it can distinguish the physical properties of various objects by observing the differences in reflectivity of their surfaces. By using a LiDAR array for scanning, the scanning range is expanded, scanning efficiency is improved, and real-time, large-area environmental data acquisition of light time difference and reflectivity is achieved. The LiDAR array sends the acquired data to a LiDAR data processing unit. The LiDAR data processing unit integrates and calculates the data to obtain the distance and physical properties of all objects, thereby acquiring the distance feature matrix and the physical property feature matrix.

[0105] The fusion unit is used to fuse the color feature matrix and the position coordinate feature matrix to determine the position coordinate set of the object region.

[0106] Specifically, color information (such as RGB, HSV, Lab values) of each pixel / region in the image is extracted from the color feature matrix to form matrix C, with dimensions (H, W, D_c) (H = height, W = width, D_c = number of color channels). Texture, edge, shape, or material features (such as LBP, HOG, SIFT, or CNN features) are extracted from the material property feature matrix to form matrix P, with dimensions (H, W, D_p) (D_p = material property feature dimension).

[0107] Specifically, (1) Early fusion: directly splice color and physical property features to generate fusion matrix F; (2) Mid-term fusion (feature-level fusion): perform feature transformation on C and P respectively (such as PCA dimensionality reduction or CNN encoding), and then fuse; (3) Late-term fusion (decision-level fusion): independently process the two feature matrices, generate their respective object region proposals, and then fuse the results.

[0108] It is understandable that by fusing the color matrix and position coordinate features, different colors can be combined into the same object, and the color of the entire area of ​​the object can be accurately extracted, thereby determining the size and position coordinate range of the object in the image.

[0109] The extraction unit is used to fuse the distance feature matrix and the physical property feature matrix to extract the object contour position coordinate matrix.

[0110] It can be understood that the distance feature matrix has the shape `[N, H, W, C_d]` or `[N, P, C_d]`, where: `N` is the batch size; `H, W` are the spatial dimensions (if it is an image or a grid); `P` is the number of points (if it is a point cloud); and `C_d` is the number of feature channels, which includes coordinate information (such as x, y, z), distance information (such as the distance to a reference point), etc.

[0111] The physical property feature matrix has the shape `[N, H, W, C_m]` or `[N, P, C_m]`, where `C_m` is the number of channels for physical properties such as density, elastic modulus, color, and texture.

[0112] It's understandable that the location distance feature matrix provides the initial position of an object (e.g., a bounding box), while the distance matrix provides the spatial relationships between objects (e.g., adjacency, overlap, etc.). Using the distance matrix to group objects (clustering) merges multiple objects belonging to the same group, combining the bounding boxes of all objects within the group to obtain a new contour (usually a larger or more complex contour). After clustering and merging, K groups are obtained (each group corresponds to a merged contour). The contour position coordinate matrix is ​​then the set of coordinates for these K contours.

[0113] The target unit is used to form a target contour matrix based on the set of position coordinates of the object region and the position coordinate matrix of the object contour.

[0114] Specifically, the set of coordinates of the object region is usually a set of bounding boxes, a list in the format of [x_min, y_min, x_max, y_max], with each element corresponding to an object.

[0115] Specifically, the object contour position coordinate matrix is ​​a three-dimensional matrix (number of objects × number of contour points × 2) for each extracted object contour point, or a list where each element is a two-dimensional array (number of contour points × 2).

[0116] The target contour matrix is ​​formed based on the existing set of object region location coordinates (bounding box) and the object contour location coordinate matrix (each object consists of multiple contour points), specifically including:

[0117] 1. Data preparation: Ensure that the two input data are consistent in the number of objects and correspond in order (i.e., the i-th bounding box corresponds to the i-th contour).

[0118] 2. Contour point correction: Since contour points may exceed the bounding box or be inaccurate, we can use the bounding box to constrain and correct the contour points.

[0119] 3. Outline point sorting: Ensure that the outline points are ordered (clockwise or counterclockwise) to facilitate subsequent processing (such as drawing, calculating area, etc.).

[0120] 4. Contour point simplification (optional): If there are too many contour points, simplification can be performed to reduce the amount of calculation.

[0121] 5. Form the target contour matrix: Store the contour points of each object as a two-dimensional array, and all objects form a three-dimensional matrix.

[0122] The display unit is used to display the target contour matrix in the environment ahead of the driving path.

[0123] It is understood that this invention uses an in-vehicle forward-facing camera in conjunction with an array-scanning LiDAR mounted on the roof to collect video image data of the road ahead and laser reflection data from surfaces of objects along the driving route. The data is then sent to an image data processing unit and a radar data processing unit. The image data processing unit integrates the color and position coordinate data of the image pixel matrix. The radar data processing unit integrates the distance and physical property data of the objects. These feature parameters are stored in matrix form, forming a color feature matrix, a position coordinate feature matrix, a distance feature matrix, and a physical property feature matrix. The color matrix and position coordinate feature matrix are fused, combining different colors into the same object to accurately extract the color of the entire object area. This determines the size and position coordinate range of the object in the image. The distance feature matrix and the physical property feature matrix are fused to extract the object contour position coordinate matrix and the target contour matrix. Finally, the target contour matrix data is displayed visually through the instrument panel and head-up display, clearly showing the contours of objects that pose a threat or obstacle to the driving path and their distances relative to the vehicle body. This invention can not only effectively identify conventional objects such as pedestrians, buildings, and the sky in front of a vehicle, but also accurately identify unconventional and irregular structural objects such as fire hydrants, ground piles, wells, and stairs.

[0124] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 8 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the complex road condition navigation methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0125] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0126] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0127] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0128] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the complex road condition navigation methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0129] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described intelligent driving assistance method.

[0130] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0131] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0132] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0133] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0134] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0135] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0137] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0139] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A navigation method for complex road conditions, characterized in that, Includes the following steps: Collect video image data and laser point cloud data in front of the vehicle; A color feature matrix and a position coordinate feature matrix are formed based on the video image data; A distance feature matrix and a physical property feature matrix are formed based on the laser point cloud data; The color feature matrix and the position coordinate matrix are fused together to determine the set of position coordinates of the object region. The distance feature matrix and the physical property feature matrix are fused together to extract the object contour position coordinate matrix; A target contour matrix is ​​formed based on the set of object region position coordinates and the object contour position coordinate matrix; The target contour matrix is ​​displayed in the environment ahead of the driving path.

2. The complex road condition navigation method according to claim 1, characterized in that, Displaying the target contour matrix in the environment ahead of the driving path includes: classifying the target contour matrix and selectively displaying it according to the classification.

3. The complex road condition navigation method according to claim 2, characterized in that, The classification includes: objects higher than the vehicle body belong to the safety category, and these safety objects are not displayed. Obstacles outside the extended edge line of the vehicle body and within the vehicle body height range are considered threat objects, and these threat objects are included in the display candidate range; Obstacles within the extended edge of the vehicle body and within the vehicle body height range are classified as hazardous objects, and these hazardous objects are displayed.

4. The complex road condition navigation method according to claim 3, characterized in that, The threat-type objects included in the display candidate scope are: The outline of the object near the side of the vehicle body is included in the display candidate range. The angle between the outline and the side of the vehicle body is used as the judgment criterion. A preset angle is selected as the threshold value, and the outline part with an angle greater than or equal to the preset angle with the side of the vehicle body is displayed.

5. The complex road condition navigation method according to claim 1, characterized in that, Also includes: Displays the outline features of obstacles in the area below the driving plane.

6. The complex road condition navigation method according to claim 5, characterized in that, The obstacle contour features displayed below the driving plane include: Extract geometric feature parameters from the target contour moments; Construct a passability judgment condition matrix based on the vehicle body geometric dimension model parameters; Import the geometric feature parameters of objects below the driving plane into the passability judgment condition matrix; Based on the passability assessment, determine whether the area below this driving plane will cause the vehicle to scrape or get stuck; If so, the user experience will be presented through methods such as driving route avoidance and displaying the outline of roadblocks; otherwise, it cannot be passed.

7. The complex road condition navigation method according to claim 6, characterized in that, The passability determination conditions include: When the span of the depression is no greater than the threshold value of 1, the vehicle can pass through the depression normally; When the span is greater than threshold 1 but not greater than threshold 2, the tires lose grip, the vehicle body sinks into the recessed area, and driving is obstructed. When the span is greater than threshold 2, the vehicle body tilts, and when the downhill angle of the recessed area is not greater than threshold 3, the tires can smoothly reach the bottom of the recessed area, the front bumper of the vehicle will not scrape, and the vehicle can pass normally; otherwise, the front bumper of the vehicle will scrape the bottom of the recessed area, and driving will be obstructed.

8. A system applying the complex road condition navigation method according to any one of claims 1-7, characterized in that, include: The acquisition unit is used to acquire video image data and laser point cloud data in front of the vehicle. The first matrix unit is used to form a color feature matrix and a position coordinate feature matrix based on the video image data; The second matrix unit is used to form a distance feature matrix and a physical property feature matrix based on the laser point cloud data; A fusion unit is used to fuse the color feature matrix and the position feature matrix to determine the object region position coordinate set; The extraction unit is used to fuse the distance feature matrix and the physical property feature matrix to extract the object contour position coordinate matrix; The target unit is used to form a target contour matrix based on the set of position coordinates of the object region and the object contour position coordinate matrix; The display unit is used to display the target contour matrix in the environment ahead of the driving path.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images