Vehicle navigation method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a vehicle navigation method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: acquiring a road image acquired by an image acquisition device in a vehicle; recognizing a building in the road image, and recognizing the road type of the road where the vehicle is located according to the building; under the condition that the road type is an urban road, identifying a target object in the road image by using a first-stage target detection algorithm; under the condition that the road type is a rural road, identifying a target object in the road image by using a two-stage target detection algorithm; and generating virtual navigation information based on the target object, and displaying the virtual navigation information at a preset position. The technical problem that a related augmented reality-based navigation technology cannot select recognition algorithms adapted to different roads according to the environment where the vehicle is located so as to recognize important objects around the vehicle is solved.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and more specifically, to a vehicle navigation method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] In the field of in-vehicle navigation, AR real-scene navigation captures the real road scene through the camera, and combines vehicle positioning, map data and AI scene recognition technology to superimpose virtual navigation instructions in the driver's field of vision. This navigation method not only reduces the difficulty for users to interpret road conditions, but also provides a navigation screen that is closer to the actual driving field of vision. At the same time, AR navigation can also intelligently identify the surrounding environment, such as vehicles, pedestrians, lane lines, traffic lights, etc., and provide drivers with a series of safety assistance functions, such as vehicle distance monitoring, collision warning, pedestrian warning, etc., thereby improving driving safety.

[0003] However, the existing technology is unable to select recognition algorithms adapted to different roads according to the environment in which the vehicle is located, such as urban roads and rural roads, so as to identify important objects (key information) around the vehicle.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a vehicle navigation method and device, a non-volatile storage medium, and an electronic device to at least solve the technical problem that the relevant augmented reality-based navigation technology cannot select a recognition algorithm adapted to different roads according to the vehicle's environment to identify important objects around the vehicle.

[0006] According to one aspect of an embodiment of the present application, a vehicle navigation method is provided, including: acquiring a road image captured by an image acquisition device in a vehicle; identifying buildings in the road image, and based on the buildings, identifying the road type of the road on which the vehicle is located; when the road type is an urban road, using a one-stage target detection algorithm to identify the target object in the road image; when the road type is a rural road, using a two-stage target detection algorithm to identify the target object in the road image; generating virtual navigation information based on the target object, and displaying the virtual navigation information at a preset position.

[0007] Optionally, when the road type is an urban road, a one-stage target detection algorithm is used to identify target objects in road images, including: performing convolution, pooling and full connection processing on the road image to obtain a first feature image; dividing the first feature image into n*n grid units, wherein n is a positive integer greater than 1, and each grid unit is used to predict a bounding box corresponding to the target object, and the target object includes at least: a turning point, an exit and a traffic sign; determining the center coordinates, confidence and category probability of the bounding box in each grid unit, wherein the confidence is used to characterize the probability that the bounding box contains the target object, and the category probability is used to characterize the probability that the object in the bounding box is the target object; using non-maximum suppression, determine a target bounding box whose confidence is greater than a first preset threshold and whose category probability is greater than a second preset threshold in all bounding boxes, and output the center coordinates corresponding to the target bounding box.

[0008] Optionally, when the road type is a rural road, a two-stage target detection algorithm is used to identify target objects in the road image, including: performing convolution, pooling and full connection processing on the road image to obtain a second feature image; generating multiple anchor points in the second feature image, wherein the anchor points include bounding boxes of different scales and aspect ratios, and the target objects include at least: turning points and traffic signs; judging whether each anchor point is a target object and predicting the position offset of the target object relative to each anchor point to obtain classification and regression results; generating multiple candidate areas based on the classification and regression results; mapping the multiple candidate areas to corresponding positions in the second feature image to obtain multiple candidate features; converting each candidate feature into a feature vector of a fixed size to obtain multiple feature vectors; inputting the multiple feature vectors into a fully connected layer, classifying the objects in the candidate areas, and adjusting the position of the bounding box based on the initial position and feature vector in the candidate areas to obtain recognition results of the target objects.

[0009] Optionally, virtual navigation information based on the target object is generated, including: when the distance between the vehicle and the turning point in the target object is less than a preset distance, generating arrow indication information for indicating the direction of vehicle travel; when it is identified that the traffic sign in the target object includes a preset field, generating reminder information corresponding to the preset field, wherein the preset fields include: school area, speed limit area, and pedestrian-dense area.

[0010] Optionally, when the distance between the vehicle and the turning point in the target object is less than a preset distance, arrow indication information is generated to indicate the direction of vehicle travel, including: when the distance between the vehicle and the turning point is less than a first threshold and greater than a second threshold, green arrow indication information is generated; when the distance between the vehicle and the turning point is less than the second threshold, red arrow indication information is generated.

[0011] Optionally, the virtual navigation information is displayed at a preset position, including: using simultaneous positioning and map building technology to build a three-dimensional model of the vehicle's environment in real time, superimposing the virtual navigation information on the three-dimensional model in the form of a virtual image, and rendering the superimposed three-dimensional model to obtain a target three-dimensional model; displaying the target three-dimensional model at a preset position in the vehicle.

[0012] Optionally, before identifying the buildings in the road image, the method further includes: performing denoising and illumination correction processing on the road image.

[0013] According to another aspect of the embodiment of the present application, a vehicle navigation device is also provided, including: an acquisition module, used to acquire a road image captured by an image acquisition device in a vehicle; a first recognition module, used to identify buildings in the road image, and identify the road type of the road where the vehicle is located based on the buildings; a second recognition module, used to identify the target object in the road image using a one-stage target detection algorithm when the road type is an urban road; a third recognition module, used to identify the target object in the road image using a two-stage target detection algorithm when the road type is a rural road; and a generation module, used to generate virtual navigation information based on the target object, and display the virtual navigation information at a preset position.

[0014] According to another aspect of the embodiment of the present application, a non-volatile storage medium is further provided, the storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above vehicle navigation method.

[0015] According to another aspect of the embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above vehicle navigation method is executed when the program is run.

[0016] According to yet another aspect of the embodiments of the present application, a computer program is further provided, wherein the above vehicle navigation method is implemented when the computer program is executed by a processor.

[0017] According to another aspect of the embodiment of the present application, a computer program product is also provided, the computer program product comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above vehicle navigation method is implemented.

[0018] In an embodiment of the present application, a road image captured by an image acquisition device in a vehicle is acquired; buildings in the road image are identified, and based on the buildings, the road type of the road on which the vehicle is located is identified; when the road type is an urban road, a one-stage target detection algorithm is used to identify the target object in the road image; when the road type is a rural road, a two-stage target detection algorithm is used to identify the target object in the road image; virtual navigation information based on the target object is generated, and the virtual navigation information is displayed at a preset position. This achieves the purpose of selecting a recognition algorithm adapted to different roads according to the environment in which the vehicle is located to identify important objects around the vehicle, thereby achieving the technical effect of precise navigation under different road conditions, and further solves the technical problem that the related augmented reality-based navigation technology cannot select a recognition algorithm adapted to different roads according to the environment in which the vehicle is located to identify important objects around the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flow chart of a vehicle navigation method according to an embodiment of the present application;

[0021] Figure 2 is a structural diagram of a vehicle navigation device according to an embodiment of the present application;

[0022] Figure 3 It is a hardware structure block diagram of a computer terminal of a vehicle navigation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an embodiment of the present application, a method embodiment of a vehicle navigation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] Figure 1 is a flow chart of a vehicle navigation method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0027] Step S101, acquiring a road image captured by an image capture device in a vehicle.

[0028] Install a high-resolution camera (such as 4K or higher) on the front of the vehicle to ensure that it can fully capture the image of the road ahead. At night or in low-light conditions, it can be equipped with an infrared or night vision camera.

[0029] Step S102, identifying buildings in the road image, and identifying the road type of the road where the vehicle is located based on the buildings.

[0030] Use a deep learning model to perform semantic segmentation on the image and identify building areas. Object-based detection models can also be used, but semantic segmentation models can provide more detailed building information. By analyzing the identified building features (such as density, type, height, etc.), combined with information such as road width and the presence of roadside green belts, the road type can be determined. For example, a high density of multi-story buildings and narrow roads may indicate that the vehicle is on an urban road, while a low density of single-story or bungalows and wide roads may indicate that the vehicle is on a rural road.

[0031] Step S103: When the road type is an urban road, a first-stage target detection algorithm is used to identify the target object in the road image.

[0032] Select YOLO series algorithms (such as YOLOv4, YOLOv5, etc.), which are known for their fast and real-time performance and are suitable for situations where target objects frequently change on urban roads. Pre-train the YOLO model using large datasets (such as COCO, Cityscapes, etc.) to recognize common road targets such as pedestrians, cars, bicycles, traffic signs, etc. Then, fine-tune according to specific needs (such as recognizing specific types of billboards, specific buildings, etc.).

[0033] Step S104: when the road type is a rural road, a two-stage target detection algorithm is used to identify the target object in the road image.

[0034] Choose Faster R-CNN or similar algorithms, which are better in object detection accuracy, although they are slower in processing speed. Again, use a large dataset for pre-training, and then fine-tune for special objects on rural roads (such as animals, specific road signs, farmland boundaries, etc.).

[0035] Step S105: Generate virtual navigation information based on the target object, and display the virtual navigation information at a preset position.

[0036] Based on the results of target detection, combined with map data and the vehicle's current position, generate virtual navigation information such as arrow instructions, distance annotations, and road condition information. Use augmented reality (AR) technology, such as ARKit or ARCore, to overlay virtual navigation information on the image captured by the camera in real time. This requires precise image registration technology to ensure that the virtual information is accurately aligned with the actual scene. Develop a user interface to display the overlaid image. The virtual information should be dynamically updated according to the movement of the vehicle to ensure the real-time and accuracy of the information.

[0037] According to the above steps, a road image captured by an image acquisition device in a vehicle is acquired; buildings in the road image are identified, and based on the buildings, the road type of the vehicle is identified; when the road type is an urban road, a one-stage target detection algorithm is used to identify the target object in the road image; when the road type is a rural road, a two-stage target detection algorithm is used to identify the target object in the road image; virtual navigation information based on the target object is generated, and the virtual navigation information is displayed at a preset position. This achieves the purpose of selecting a recognition algorithm adapted to different roads according to the environment in which the vehicle is located to identify important objects around the vehicle, thereby achieving the technical effect of accurate navigation under different road conditions.

[0038] The following Figure 1 The steps shown are exemplary and explanatory.

[0039] According to some optional embodiments of the present application, when the road type is an urban road, identifying the target object in the road image using a one-stage target detection algorithm can be achieved by the following method:

[0040] The road image is subjected to convolution, pooling and full connection processing to obtain a first feature image; the first feature image is divided into n*n grid units, wherein n is a positive integer greater than 1, and each grid unit is used to predict a bounding box corresponding to a target object, and the target object includes at least: a turning point, an exit and a traffic sign; the center coordinates, confidence and category probability of the bounding box in each grid unit are determined, wherein the confidence is used to characterize the probability that the bounding box contains the target object, and the category probability is used to characterize the probability that the object in the bounding box is the target object; using non-maximum suppression, a target bounding box whose confidence is greater than a first preset threshold and whose category probability is greater than a second preset threshold is determined in all bounding boxes, and the center coordinates corresponding to the target bounding box are output.

[0041] First, the road image is input into a pre-trained convolutional neural network (such as Darknet, ResNet, etc.), which extracts the features of the image through multiple convolutional layers and pooling layers. The convolutional layer can recognize specific patterns in the image, such as edges, textures, etc., while the pooling layer is used to reduce the size of the feature map while maintaining the information of the most important features. Finally, the local features are combined into global features through the fully connected layer to obtain the first feature image. This process helps the algorithm understand the complex structure in the image.

[0042] Secondly, the first feature image is divided into multiple grid cells, each of which is responsible for predicting the target within its coverage. The size of the grid cell (n*n) can be adjusted according to the resolution of the image and the needs of target detection. Larger grid cells (such as larger n values) can handle larger targets, while smaller grid cells are more suitable for detecting small objects. Each grid cell predicts multiple bounding boxes (can be 2 or 3), which are used to surround the target object. The prediction of the bounding box includes the center coordinates, width, height, and a confidence score associated with the target object. The YOLO algorithm predicts the center coordinates, width, height, confidence and category probability for each bounding box. The confidence reflects the probability of the authenticity of the target contained in the bounding box, while the category probability reflects the probability that the object in the bounding box belongs to a specific category.

[0043] Non-maximum suppression is a post-processing step that selects the most likely bounding box from all predicted bounding boxes while removing those that overlap or have low confidence. Specifically, NMS first sorts all bounding boxes by confidence, then selects a bounding box and removes all bounding boxes that overlap with it (the degree of overlap is usually measured by the intersection over union (IoU)). This process is repeated until all bounding boxes have been processed.

[0044] Finally, two thresholds are set, one for filtering bounding boxes with low confidence and the other for filtering bounding boxes with low class probability. Only when a bounding box satisfies both threshold conditions will it be considered a valid detection result. The center coordinates of these bounding boxes are output for subsequent AR navigation information generation.

[0045] Through the above steps, the YOLO algorithm can quickly and accurately detect target objects in road images, such as turning points, exits, and traffic signs, providing real-time visual information for the AR navigation system to help drivers better understand and respond to the driving environment. The efficiency of this process makes YOLO an ideal choice for high-speed target detection in urban road environments.

[0046] According to some other optional embodiments of the present application, when the road type is a rural road, the target object in the road image is identified by using a two-stage target detection algorithm, which can be achieved by the following method: convolution, pooling and full connection processing are performed on the road image to obtain a second feature image; multiple anchor points are generated in the second feature image, wherein the anchor points include bounding boxes of different scales and aspect ratios, and the target objects include at least: turning points and traffic signs; whether each anchor point is a target object and the position offset of the target object relative to each anchor point is determined to obtain classification and regression results; based on the classification and regression results, multiple candidate areas are generated; the multiple candidate areas are mapped to corresponding positions in the second feature image to obtain multiple candidate features; each candidate feature is converted into a feature vector of a fixed size to obtain multiple feature vectors; the multiple feature vectors are input into the fully connected layer, the objects in the candidate areas are classified, and the position of the bounding box is adjusted according to the initial position and feature vector in the candidate area to obtain the recognition result of the target object.

[0047] A pre-trained convolutional neural network (such as VGG, ResNet, etc.) is used as the basic feature extractor. Multiple convolutional layers are used to extract local features of the image. The pooling layer is used to reduce the size of the feature map, and the fully connected layer is used to generate global features. This series of operations converts the original image into a feature map, the second feature image, which contains the key visual information in the image.

[0048] A set of predefined anchors are generated at each position of the feature map, corresponding to a series of bounding boxes with different sizes and aspect ratios, which are used to cover objects of different sizes and shapes. The size and aspect ratio of these anchors are usually determined based on the distribution characteristics of the objects in the training dataset.

[0049] The second feature image is processed using a Region Proposal Network (RPN), which predicts the confidence of the target and non-target and the position offset of the bounding box for each anchor point. The position offset is used to adjust the bounding box of the anchor point so that it more accurately surrounds the target object.

[0050] Based on the output of RPN, the bounding boxes adjusted by anchors with higher confidence are used as candidate regions, which are used for further classification and position regression in the next step. The feature map is processed using the RoI (Region of Interest) Pooling layer, each candidate region is mapped to the corresponding position of the feature map, and then a fixed-size feature vector, i.e., the candidate feature, is extracted for subsequent classification and regression tasks. The RoIPooling layer or its improved version (such as RoIAlign) generates a fixed-size feature vector. This process ensures that even if the candidate regions are of different sizes, they can be input into the subsequent fully connected layer for processing.

[0051] The candidate features are input into the classifier and regressor. The classifier is responsible for determining what kind of target object the candidate region belongs to (such as turning points, traffic signs, etc.), while the regressor is used to further fine-tune the position of the bounding box so that it surrounds the target more accurately. Finally, the recognition result of the target object is generated based on the classification and regression results.

[0052] Through the above steps, the Faster R-CNN algorithm can accurately identify target objects such as turning points and traffic signs from road images, providing high-quality visual information for the AR navigation system, helping drivers drive safely in complex environments such as rural roads.

[0053] In some optional embodiments of the present application, generating virtual navigation information based on the target object can be achieved by the following method: when the distance between the vehicle and the turning point in the target object is less than a preset distance, generating arrow indication information for indicating the direction of vehicle travel; when it is identified that the traffic sign in the target object includes a preset field, generating reminder information corresponding to the preset field, wherein the preset fields include: school area, speed limit area, and pedestrian-dense area.

[0054] When a target object (such as a turning point or traffic sign) is identified, the distance between the vehicle and the target object is calculated using the vehicle's GPS position, IMU data, and the target object's positioning information. If this distance is less than a preset distance (for example, within 50 meters of the next turning point or a specific traffic sign), the next step is to generate specific navigation information.

[0055] If the target object is a turning point and the distance between the vehicle and the turning point is less than the preset distance, the system will generate an arrow indication information. This information should be intuitively displayed in the direction of the vehicle's travel, prompting the driver to the upcoming turn position and direction. The arrow should be adjusted according to the direction of the turn (left turn, right turn), and its size and color can change dynamically based on the distance from the turning point to enhance the visual effect. For example, when the distance is close, the arrow can be larger and more colorful to attract the driver's attention. Use SLAM technology to superimpose the arrow indication information on the appropriate position of the road image to ensure that the arrow is visually aligned with the actual turning point, providing accurate navigation guidance.

[0056] If the recognized traffic signs include preset fields (for example, school areas, speed limit areas, and pedestrian-dense areas), the system will generate relevant reminder information. This information is crucial for drivers to drive safely in specific areas. Information content: The reminder information should clearly state the nature of the target area, such as "Entering the school area ahead, please slow down" or "Speed ​​limit 30 km / h". Display form: In addition to text reminders, icons or animations can also be used to enhance the communication effect of the information. For example, reminders in school areas can use a flashing school icon, and speed limit areas can be a combination of numbers and speed limit signs. In addition to visual display, reminder information can also be converted into voice prompts through speech synthesis technology to ensure that drivers can receive important information while concentrating on driving.

[0057] Monitor the vehicle's position and the distance to the target object in real time. Once the distance between the vehicle and the turning point or specific traffic sign changes, immediately adjust the display of arrow instructions and reminder information to keep the information real-time and accurate.

[0058] Through the above steps, the AR navigation system can generate intuitive and timely navigation information based on the real-time relationship between the vehicle and the target object, which not only improves the driver's perception of the route, but also enhances the awareness of safe driving in special areas (such as school areas, speed limit areas, and pedestrian-dense areas).

[0059] As some optional embodiments of the present application, when the distance between the vehicle and a turning point in a target object is less than a preset distance, arrow indication information for indicating the direction of vehicle travel is generated, which can be achieved by the following method: when the distance between the vehicle and the turning point is less than a first threshold and greater than a second threshold, green arrow indication information is generated; when the distance between the vehicle and the turning point is less than the second threshold, red arrow indication information is generated.

[0060] Using the vehicle's GPS positioning information and the pre-stored turning point coordinates, the straight-line distance between the vehicle and the next turning point is calculated through a geographic information system (GIS) or a direct mathematical formula (such as Euclidean distance). This step is the basis for dynamically adjusting the arrow color. The setting of the first threshold and the second threshold needs to be optimized according to the actual usage scenario and driving habits. For example, the first threshold can be set to 100 meters, which means that the vehicle is about to approach the turning point; the second threshold can be set to 50 meters, indicating that the vehicle is very close to the turning point and needs to prepare to turn immediately. The setting of these thresholds should take into account the driver's reaction time and the vehicle's driving speed to ensure the timeliness and effectiveness of navigation prompts.

[0061] The color of the arrow indication information is dynamically adjusted according to the distance between the vehicle and the turning point. When the distance is between the first threshold and the second threshold, a green arrow indication information is generated. Green is usually regarded as a passable and safe color, which conveys a message to the driver that "prepare to turn, but there is still time". When the distance is less than the second threshold, a red arrow indication information is generated. Red is a warning and urgent color, prompting the driver to "prepare to turn immediately, time is running out". The changes in the vehicle position and the turning point distance are monitored in real time. Once the distance crosses the threshold, the color of the arrow is updated immediately. The generated arrow indication information is superimposed on the vehicle's front-view camera image in real time and displayed to the driver through the on-board display. At the same time, through speech synthesis technology, the distance information and the change of the arrow color are converted into voice prompts, further enhancing the driver's perception and reaction speed.

[0062] In some optional embodiments of the present application, displaying virtual navigation information at a preset position can be achieved by the following method: utilizing simultaneous positioning and map building technology to construct a three-dimensional model of the vehicle's environment in real time, superimposing the virtual navigation information on the three-dimensional model in the form of a virtual image, and rendering the superimposed three-dimensional model to obtain a target three-dimensional model; and displaying the target three-dimensional model at a preset position in the vehicle.

[0063] First, SLAM technology is used to estimate the position and posture of the vehicle in real time, and to build a three-dimensional model of the vehicle's surroundings. This step requires the combination of multi-sensor data, such as on-board cameras, LiDAR, IMU (inertial measurement unit), etc. The SLAM algorithm determines the relative position of the vehicle to the surrounding environment through image feature matching, laser scanning point cloud matching, etc., and estimates the environmental structure to generate a three-dimensional model.

[0064] Once the 3D model is built, the next step is to superimpose virtual navigation information (such as arrow instructions, distance annotations, road condition information, etc.) into the 3D model in the form of a virtual image. This requires precise image registration and rendering technology to ensure the consistency of virtual information with the real environment in position, direction and size. Use a 3D rendering engine (such as Unity, UnrealEngine, etc.) combined with the 3D model coordinate system generated by SLAM to place the virtual navigation information in the appropriate position. For example, if you need to display an arrow pointing to the left turn, the algorithm will place it in front of the left of the turning point in the 3D model to ensure that the driver can intuitively see the navigation instructions when observing the AR view.

[0065] The 3D model with virtual navigation information superimposed is rendered in real time to generate an AR view. This step requires high-performance graphics processing capabilities, because real-time rendering means that the entire scene needs to be quickly rebuilt and rendered in each frame. Finally, the target 3D model obtained by real-time rendering is displayed at a preset position on the in-vehicle display screen to ensure that the driver can view the AR navigation information without obstruction or distraction while driving.

[0066] Through the above steps, the AR navigation system can build a three-dimensional model of the vehicle's surrounding environment in real time, and accurately superimpose virtual navigation information in the model to provide the driver with intuitive and real-time visual navigation assistance. Especially in complex environments such as rural roads, this AR navigation can significantly improve driving safety and efficiency.

[0067] As some other optional embodiments of the present application, before identifying the buildings in the road image, the following steps may be performed: performing denoising and illumination correction processing on the road image.

[0068] The purpose of denoising is to reduce random noise in the image, improve image quality, and make subsequent feature extraction and target recognition more accurate. Denoising methods include: Mean filtering: By calculating the average value of pixels in the local area and replacing the value of the central pixel, the image is smoothed and noise is removed. Median filtering: Similar to mean filtering, but using the median of the pixel values ​​in the local area, this method can effectively remove salt and pepper noise while maintaining the clarity of the image edge. Double Gaussian filtering: Combining the smoothing effect and edge retention characteristics of the Gaussian filter, through two Gaussian filtering operations, it can reduce image noise while better retaining edge information. Adaptive threshold denoising: According to the characteristics of the local area of ​​the image, the threshold is dynamically adjusted to remove noise. This method can meet the denoising needs of images under different lighting conditions.

[0069] Lighting conditions have an important impact on the recognition of buildings. Differences in light intensity and direction at different times and in different weather conditions may cause significant changes in the appearance of buildings, affecting the recognition effect. The purpose of lighting correction is to keep the image consistent in brightness and contrast under different lighting conditions. Lighting correction methods include: Histogram equalization: By adjusting the brightness distribution of the image to make the distribution of the histogram more uniform, the image contrast is enhanced and the image quality under low or high light conditions is improved. Local contrast enhancement: For each local area in the image, the contrast is adjusted to enhance the details. This method is particularly effective for environments with uneven lighting. Normalization: Adjust the brightness and color range of the image to a standard range to reduce the impact of lighting on the visual effects of the image. Lighting invariant feature extraction: Use specialized image processing algorithms (such as SIFT, SURF, etc.), which are robust to lighting changes and can extract stable and recognizable features even when lighting conditions change.

[0070] Figure 2 is a structural diagram of a vehicle navigation device according to an embodiment of the present application, such as Figure 2 As shown, the device comprises:

[0071] The acquisition module 21 is used to acquire a road image acquired by an image acquisition device in the vehicle.

[0072] The first recognition module 22 is used to recognize buildings in the road image and, based on the buildings, recognize the road type of the road where the vehicle is located.

[0073] The second recognition module 23 is used to recognize the target object in the road image by using a one-stage target detection algorithm when the road type is an urban road.

[0074] The third recognition module 24 is used to recognize the target object in the road image by using a two-stage target detection algorithm when the road type is a rural road.

[0075] The generating module 25 is used to generate virtual navigation information based on the target object and display the virtual navigation information at a preset position.

[0076] Optionally, when the road type is an urban road, a one-stage target detection algorithm is used to identify target objects in road images, including: performing convolution, pooling and full connection processing on the road image to obtain a first feature image; dividing the first feature image into n*n grid units, wherein n is a positive integer greater than 1, and each grid unit is used to predict a bounding box corresponding to the target object, and the target object includes at least: a turning point, an exit and a traffic sign; determining the center coordinates, confidence and category probability of the bounding box in each grid unit, wherein the confidence is used to characterize the probability that the bounding box contains the target object, and the category probability is used to characterize the probability that the object in the bounding box is the target object; using non-maximum suppression, determine a target bounding box whose confidence is greater than a first preset threshold and whose category probability is greater than a second preset threshold in all bounding boxes, and output the center coordinates corresponding to the target bounding box.

[0077] Optionally, when the road type is a rural road, a two-stage target detection algorithm is used to identify target objects in the road image, including: performing convolution, pooling and full connection processing on the road image to obtain a second feature image; generating multiple anchor points in the second feature image, wherein the anchor points include bounding boxes of different scales and aspect ratios, and the target objects include at least: turning points and traffic signs; judging whether each anchor point is a target object and predicting the position offset of the target object relative to each anchor point to obtain classification and regression results; generating multiple candidate areas based on the classification and regression results; mapping the multiple candidate areas to corresponding positions in the second feature image to obtain multiple candidate features; converting each candidate feature into a feature vector of a fixed size to obtain multiple feature vectors; inputting the multiple feature vectors into a fully connected layer, classifying the objects in the candidate areas, and adjusting the position of the bounding box based on the initial position and feature vector in the candidate areas to obtain recognition results of the target objects.

[0078] Optionally, virtual navigation information based on the target object is generated, including: when the distance between the vehicle and the turning point in the target object is less than a preset distance, generating arrow indication information for indicating the direction of vehicle travel; when it is identified that the traffic sign in the target object includes a preset field, generating reminder information corresponding to the preset field, wherein the preset fields include: school area, speed limit area, and pedestrian-dense area.

[0079] Optionally, when the distance between the vehicle and the turning point in the target object is less than a preset distance, arrow indication information is generated to indicate the direction of vehicle travel, including: when the distance between the vehicle and the turning point is less than a first threshold and greater than a second threshold, green arrow indication information is generated; when the distance between the vehicle and the turning point is less than the second threshold, red arrow indication information is generated.

[0080] Optionally, the virtual navigation information is displayed at a preset position, including: using simultaneous positioning and map building technology to build a three-dimensional model of the vehicle's environment in real time, superimposing the virtual navigation information on the three-dimensional model in the form of a virtual image, and rendering the superimposed three-dimensional model to obtain a target three-dimensional model; displaying the target three-dimensional model at a preset position in the vehicle.

[0081] Optionally, before identifying the buildings in the road image, the method further includes: performing denoising and illumination correction processing on the road image.

[0082] It should be noted that the above Figure 2 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0083] It should be noted that Figure 2 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0084] Figure 3 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a vehicle navigation method. Figure 3 As shown, the computer terminal 30 may include one or more (302a, 302b, ..., 302n are used to illustrate) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission module 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown.

[0085] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 30. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0086] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the vehicle navigation method in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, realizing the above-mentioned vehicle navigation method. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the computer terminal 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0087] The transmission module 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 30. In one example, the transmission module 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0088] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 30 .

[0089] It should be noted that, in some optional embodiments, the above Figure 3 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 3This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0090] It should be noted that Figure 3 The computer terminal shown is used to execute Figure 1 The vehicle navigation method shown, therefore the relevant explanations and instructions in the execution method of the above-mentioned commands are also applicable to the electronic device and will not be repeated here.

[0091] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above vehicle navigation method.

[0092] A program for a non-volatile storage medium to perform the following functions: acquiring a road image acquired by an image acquisition device in a vehicle; identifying buildings in the road image, and based on the buildings, identifying the road type of the vehicle; in the case where the road type is an urban road, using a one-stage target detection algorithm to identify the target object in the road image; in the case where the road type is a rural road, using a two-stage target detection algorithm to identify the target object in the road image; generating virtual navigation information based on the target object, and displaying the virtual navigation information at a preset position.

[0093] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above vehicle navigation method is executed when the program is run.

[0094] The processor is used to run a program that performs the following functions: acquiring a road image acquired by an image acquisition device in a vehicle; identifying buildings in the road image, and based on the buildings, identifying the road type of the road on which the vehicle is located; when the road type is an urban road, using a one-stage target detection algorithm to identify the target object in the road image; when the road type is a rural road, using a two-stage target detection algorithm to identify the target object in the road image; generating virtual navigation information based on the target object, and displaying the virtual navigation information at a preset position.

[0095] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0096] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0097] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0102] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A vehicle navigation method, characterized in that: include: Acquiring a road image acquired by an image acquisition device in a vehicle; Identifying buildings in the road image, and identifying the road type of the road where the vehicle is located based on the buildings; In the case where the road type is an urban road, using a one-stage target detection algorithm to identify a target object in the road image; In the case where the road type is a rural road, a two-stage target detection algorithm is used to identify a target object in the road image; Generate virtual navigation information based on the target object, and display the virtual navigation information at a preset position.

2. The method according to claim 1, characterized in that In the case where the road type is an urban road, identifying a target object in the road image using a one-stage target detection algorithm includes: Performing convolution, pooling and full connection processing on the road image to obtain a first feature image; Divide the first feature image into n*n grid units, where n is a positive integer greater than 1, and each grid unit is used to predict a bounding box corresponding to the target object, and the target object includes at least: a turning point, an exit, and a traffic sign; Determine the center coordinates, confidence, and category probability of a bounding box in each grid unit, wherein the confidence is used to characterize the probability that the bounding box contains the target object, and the category probability is used to characterize the probability that the object in the bounding box is the target object; By utilizing non-maximum suppression, a target bounding box whose confidence is greater than a first preset threshold and whose category probability is greater than a second preset threshold is determined from among all the bounding boxes, and the center coordinates corresponding to the target bounding box are output.

3. The method according to claim 1, characterized in that In the case where the road type is a rural road, a two-stage target detection algorithm is used to identify a target object in the road image, including: Performing convolution, pooling and full connection processing on the road image to obtain a second feature image; Generating a plurality of anchor points in the second feature image, wherein the anchor points include bounding boxes of different scales and aspect ratios, and the target objects include at least: turning points and traffic signs; Determine whether each of the anchor points is the target object and predict the position offset of the target object relative to each of the anchor points to obtain classification and regression results; Generating a plurality of candidate regions according to the classification and regression results; Mapping the plurality of candidate regions to corresponding positions in the second feature image to obtain a plurality of candidate features; Convert each of the candidate features into a feature vector of a fixed size to obtain multiple feature vectors; The plurality of feature vectors are input into a fully connected layer, objects in the candidate region are classified, and the position of the bounding box is adjusted according to the initial position in the candidate region and the feature vector to obtain a recognition result of the target object.

4. The method according to claim 1, characterized in that: Generating virtual navigation information based on the target object, including: When the distance between the vehicle and the turning point in the target object is less than a preset distance, generating arrow indication information for indicating the driving direction of the vehicle; When it is identified that the traffic sign in the target object includes a preset field, a reminder message corresponding to the preset field is generated, wherein the preset field includes: a school area, a speed limit area, and a pedestrian dense area.

5. The method according to claim 4, characterized in that When the distance between the vehicle and the turning point in the target object is less than a preset distance, generating arrow indication information for indicating the driving direction of the vehicle includes: When the distance between the vehicle and the turning point is less than a first threshold and greater than a second threshold, generating green arrow indication information; When the distance between the vehicle and the turning point is less than the second threshold, red arrow indication information is generated.

6. The method according to claim 1, characterized in that Displaying the virtual navigation information at a preset position includes: Using simultaneous positioning and mapping technology, a three-dimensional model of the vehicle's environment is constructed in real time superimposing the virtual navigation information into the three-dimensional model in the form of a virtual image, and rendering the superimposed three-dimensional model to obtain a target three-dimensional model; The target three-dimensional model is displayed at a preset position in the vehicle.

7. The method according to claim 1, characterized in that Before identifying the buildings in the road image, the method further includes: performing denoising and illumination correction processing on the road image.

8. A vehicle navigation device, characterized in that: include: An acquisition module, used to acquire a road image acquired by an image acquisition device in a vehicle; A first recognition module is used to recognize buildings in the road image and, based on the buildings, recognize the road type of the road where the vehicle is located; A second recognition module is used to recognize the target object in the road image by using a one-stage target detection algorithm when the road type is an urban road; A third recognition module is used to recognize the target object in the road image by using a two-stage target detection algorithm when the road type is a rural road; The generating module is used to generate virtual navigation information based on the target object and display the virtual navigation information at a preset position.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the vehicle navigation method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the vehicle navigation method according to any one of claims 1 to 7 is implemented.

Images