Image rapid processing method and device during intelligent driving and medium

Abstract

The invention relates to an image processing technology, and discloses a rapid image processing method during intelligent driving, which comprises the following steps: respectively carrying out targeted analysis on front view image data, all-round view image data and side view image data acquired in the driving process of a target vehicle to obtain road marking data, surrounding vehicle data and road pedestrian data; the method comprises the following steps: converting road marking data, surrounding vehicle data and road pedestrian data into coordinate data, carrying out three-dimensional space rapid modeling based on the coordinate data to obtain an intelligent driving space model, generating a driving suggestion according to the intelligent driving space model, carrying out three-dimensional rough modeling according to collected wide-angle image data to obtain an intelligent driving rough space model, and carrying out three-dimensional modeling according to the collected wide-angle image data to obtain a driving suggestion. And comparing the intelligent driving space model with the intelligent driving rough space model in real time to obtain a model error, and performing driving suggestion adaptive broadcasting based on the model error. The invention also provides an image rapid processing device during intelligent driving and a medium. The image processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method, device and medium for rapid image processing during intelligent driving. Background Art

[0002] With the continuous increase in the number of automobiles and the growing demand for travel safety and convenience, intelligent driving technology is gradually moving from concept to reality and becoming the core driving force for the transformation of the automotive industry and even the entire transportation field. The intelligent driving system aims to achieve the functions of autonomous driving or assisted driving of vehicles through a series of advanced sensors, algorithms and electronic control units, thereby greatly reducing human driving errors, reducing the risk of traffic accidents, and improving traffic efficiency at the same time.

[0003] Image recognition and processing technology, as a key link in intelligent driving, plays a crucial role. Intelligent driving vehicles are usually equipped with multiple cameras distributed at different parts of the vehicle body to capture visual information around the vehicle in all directions. The massive amount of image data collected by these cameras in real time needs to be efficiently processed within an extremely short time to provide timely and accurate basis for the decision-making and control of the vehicle. For example, when the vehicle is driving at high speed, the front-view camera may collect dozens or even hundreds of frames of images per second. If these images cannot be quickly analyzed to identify key elements such as the road, vehicles, pedestrians and traffic signs ahead, the vehicle will not be able to make reasonable driving decisions such as accelerating, decelerating or turning, thus endangering driving safety.

[0004] Currently, traditional image processing methods have exposed many limitations in intelligent driving scenarios. On the one hand, most existing image algorithms are based on general computer vision technology and are insufficiently optimized for the specific needs of intelligent driving. When processing high-resolution images in complex road conditions, the computational complexity is high, resulting in slow processing speed and difficult to meet the real-time requirements. For example, in urban congestion sections, vehicles, pedestrians, non-motor vehicles are mixed, and road signs appear frequently. The processing time of traditional algorithms for front-view images often exceeds dozens of milliseconds, while the intelligent driving system requires the processing time of each frame of image to be controlled within a few milliseconds to ensure that the vehicle can respond in time. Summary of the Invention

[0005] The present invention provides a method, device and medium for rapid image processing during intelligent driving, and its main purpose is to solve the problem of low response efficiency of existing image processing methods in intelligent driving scenarios.

[0006] To achieve the above object, a method for rapid image processing during intelligent driving provided by the present invention includes: Real-time collecting front-view image data, surround-view image data and side-view image data during the driving process of the target vehicle; Perform targeted analysis on road markings based on the forward-looking image data to obtain road marking data; Perform targeted analysis on the positions of surrounding vehicles based on the panoramic image data to obtain surrounding vehicle data; Perform targeted recognition of road pedestrians based on the side-looking image data to obtain road pedestrian data; Convert the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data, and perform rapid 3D space modeling based on the comprehensive coordinate data to obtain an intelligent driving space model; Generate driving suggestions based on the intelligent driving space model; Real-time collect the wide-angle image data of the target vehicle, and perform 3D rough modeling based on the wide-angle image dataset to obtain a rough intelligent driving space model; Compare the intelligent driving space model with the rough intelligent driving space model in real time to obtain a model error; Determine whether the model error is greater than or equal to a preset error threshold; If the model error is less than the error threshold, broadcast the driving suggestions to the user; If the model error is greater than or equal to the error threshold, after adjusting the driving suggestions in real time according to the model error, return to the step of broadcasting the driving suggestions to the user.

[0007] Optionally, the performing targeted analysis on road markings based on the forward-looking image data to obtain road marking data includes: Use a pre-trained lightweight road segmentation model to segment the forward-looking image to obtain a forward-looking road image; Use the HSV color space to enhance the color of the forward-looking road image to obtain an enhanced road image; Perform edge contour detection on the enhanced road image to obtain marking contour data; Detect straight lines and curves in the enhanced road image based on the Hough transform to obtain shape data; Locate the road markings in the forward-looking road image based on the marking contour data and the shape data to obtain road marking data.

[0008] Optionally, the using a pre-trained lightweight road segmentation model to segment the forward-looking image to obtain a forward-looking road image includes: Perform normalization processing on the forward-looking image to obtain a normalized forward-looking image; Perform dimension conversion processing on the normalized forward-looking image to obtain a converted image; Infer the transformed image by invoking the forward propagation function of the lightweight road segmentation model to obtain a probability distribution tensor; Based on a preset probability threshold, convert the probability distribution tensor into a two-dimensional array, and use the two-dimensional array to segment the forward-looking image to obtain a forward-looking road image.

[0009] Optionally, the analyzing the surrounding vehicle positions according to the panoramic image data to obtain surrounding vehicle data includes: Perform vehicle target recognition on the panoramic image data to obtain target recognition data; Use a multi-target tracking algorithm to track the positions of vehicles in the panoramic image data based on the target recognition data to obtain vehicle trajectories; Perform short-term trajectory prediction on the vehicles in the panoramic image data based on the vehicle trajectories to obtain predicted trajectories; Summarize the target recognition data, the vehicle trajectories, and the predicted trajectories to obtain the surrounding vehicle data.

[0010] Optionally, the converting the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data includes: Obtain the first marking distance and the second marking distance included in the road marking data, and calculate the marking position according to the first marking distance, the second marking distance, and a preset forward-looking camera distance; Obtain the first vehicle distance and the second vehicle distance included in the surrounding vehicle data, and calculate the vehicle position according to the first vehicle distance, the second vehicle distance, and a preset panoramic camera distance; Obtain the first pedestrian distance and the second pedestrian distance included in the road pedestrian data, and calculate the pedestrian position according to the first pedestrian distance, the second pedestrian distance, and a preset side-looking camera distance; Establish a two-dimensional rectangular coordinate system with the target vehicle as the origin and parallel to the horizontal plane; Based on the two-dimensional rectangular coordinate system, convert the marking position, the vehicle position, and the pedestrian position into coordinate data to obtain comprehensive coordinate data.

[0011] Optionally, the method of collecting the wide-angle image data of the target vehicle in real time and performing three-dimensional rough modeling according to the wide-angle image data set to obtain a rough intelligent driving space model includes: Perform wide-angle distortion correction processing on the wide-angle image data to obtain a corrected image; Extract features of target objects from the corrected image to obtain target object features; Perform fast approximate feature matching using the target object features to obtain target recognition data.

[0012] Use a 3D reconstruction algorithm to obtain the target object in the corrected image based on the target recognition data, and obtain the 3D coordinates of the target object; Perform 3D modeling based on the 3D coordinates of the target object to obtain a rough spatial model for intelligent driving.

[0013] Optionally, the real-time comparison of the intelligent driving spatial model and the rough spatial model for intelligent driving to obtain a model error includes: Obtain the number of surrounding vehicles and the number of surrounding pedestrians in the intelligent driving spatial model, and obtain the rough number of surrounding vehicles and the rough number of surrounding pedestrians in the rough spatial model for intelligent driving.

[0014] Establish a 3D coordinate system with the target vehicle as the origin; Based on the 3D coordinate system, obtain the coordinate data of the intelligent driving spatial model and the rough spatial model for intelligent driving to obtain model 3D data and rough model 3D data; Determine the model volume and the model spatial range of the rough spatial model for intelligent driving according to the model 3D data; Determine the rough model volume and the rough model spatial range of the rough spatial model for intelligent driving according to the rough model 3D data; Calculate the volume that does not overlap between the intelligent driving spatial model and the rough spatial model for intelligent driving according to the model volume, the model spatial range, the rough model volume, and the rough model spatial range to obtain a non-overlapping volume; Calculate the model error according to the model volume, the rough model volume, the non-overlapping volume, the rough number of surrounding vehicles, and the rough number of surrounding pedestrians.

[0015] Optionally, the calculation of the model error according to the model volume, the rough model volume, the non-overlapping volume, the rough number of surrounding vehicles, and the rough number of surrounding pedestrians includes: Calculate the model error using the following formula: Where, is the model error, is the model volume, is the rough model volume, is the non-overlapping volume, is the base of the natural logarithm, is the number of surrounding pedestrians, is the rough number of surrounding pedestrians, is the number of surrounding vehicles, is the approximate number of surrounding vehicles.

[0016] To solve the above problems, the present invention also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned method for rapid image processing during intelligent driving.

[0017] To solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned method for rapid image processing during intelligent driving.

[0018] In the embodiment of the present invention, by collecting front-view image data, surround-view image data, and side-view image data during the driving process of the target vehicle in real time, performing targeted analysis on road markings based on the front-view image data to obtain road marking data, performing targeted analysis on the positions of surrounding vehicles based on the surround-view image data to obtain surrounding vehicle data, and performing targeted recognition of road pedestrians based on the side-view image data to obtain road pedestrian data, the efficiency of image data analysis can be improved. Convert the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data, perform rapid three-dimensional space modeling based on the comprehensive coordinate data to obtain an intelligent driving space model, generate driving suggestions according to the intelligent driving space model, collect the wide-angle image data of the target vehicle in real time, perform three-dimensional rough modeling based on the wide-angle image data set to obtain a rough intelligent driving space model, compare the intelligent driving space model with the rough intelligent driving space model in real time to obtain a model error, and determine whether the model error is greater than or equal to a preset error threshold. If the model error is less than the error threshold, the driving suggestion is broadcast to the user. If the model error is greater than or equal to the error threshold, after adjusting the driving suggestion in real time according to the model error, return to the step of broadcasting the driving suggestion to the user. It can improve the safety of intelligent driving. Therefore, the method, device, and medium for rapid image processing during intelligent driving proposed by the present invention can solve the problem of low response efficiency of existing image processing methods in intelligent driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic flowchart of a method for rapid image processing during intelligent driving provided by an embodiment of the present invention; Figure 2 A schematic flowchart for analyzing road markings provided by an embodiment of the present invention; Figure 3 A schematic flowchart for converting coordinate data provided by an embodiment of the present invention; Figure 4 A schematic structural diagram of an electronic device for implementing the method for rapid image processing during intelligent driving provided by an embodiment of the present invention.

[0020] The realization, functional features and advantages of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] An embodiment of the present application provides a method for rapid image processing during intelligent driving. The execution subject of the method for rapid image processing during intelligent driving includes, but is not limited to, at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for rapid image processing during intelligent driving can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0023] Referring to Figure 1 As shown, it is a schematic flowchart of the method for rapid image processing during intelligent driving provided by an embodiment of the present invention. In this embodiment, the method for rapid image processing during intelligent driving includes: S1. Real-time collect front view image data, panoramic view image data and side view image data during the driving process of the target vehicle.

[0024] In the embodiment of the present invention, the front view image data is collected by a front view camera preset in the target vehicle.

[0025] In the embodiment of the present invention, the panoramic view image data is collected by a panoramic view camera preset in the target vehicle.

[0026] In the embodiment of the present invention, the side view image data is collected by a side view camera preset in the target vehicle.

[0027] Specifically, the forward-looking camera is usually installed behind the front windshield of the vehicle. The number of the forward-looking cameras can be two, and the forward-looking image data can be two sets of image data.

[0028] Specifically, the surround-view cameras are usually installed under the front bumper and under the rear bumper of the vehicle. There can be two surround-view cameras installed under the front bumper and two under the rear bumper of the vehicle respectively, and the surround-view image data includes multiple sets of images.

[0029] Specifically, the side-view cameras are usually installed on the sides of the vehicle. There can be two side-view cameras installed on each of the left and right sides of the vehicle, and the side-view image data includes multiple sets of images.

[0030] In the embodiments of the present invention, the forward-looking image data, the surround-view image data, and the side-view image data are all collected according to a preset acquisition frequency. The forward-looking image data, the surround-view image data, and the side-view image data all include consecutive images. Specifically, the preset acquisition frequency can be once per second.

[0031] S2. Perform road marking target analysis based on the forward-looking image data to obtain road marking data.

[0032] In the embodiments of the present invention, refer to Figure 2 As shown, the present invention provides a flow diagram of road marking target analysis. The performing road marking target analysis based on the forward-looking image data to obtain road marking data includes: S21. Segment the forward-looking image using a pre-trained lightweight road segmentation model to obtain a forward-looking road image; S22. Perform color enhancement on the forward-looking road image in the HSV color space to obtain an enhanced road image; S23. Detect the edge contours of the enhanced road image to obtain marking contour data; S24. Detect straight lines and curves in the enhanced road image based on the Hough transform to obtain shape data; S25. Locate the road markings in the forward-looking road image based on the marking contour data and the shape data to obtain road marking data.

[0033] Specifically, the HSV color space consists of three dimensions: hue, saturation, and value.

[0034] Specifically, the use of the HSV color space to perform color enhancement on the forward-looking road image to obtain an enhanced road image means enhancing the saturation to improve the purity of the colors in the forward-looking road image, and then enhancing the colors corresponding to the road markings (usually white and yellow) in the forward-looking road image by adjusting a specific hue range.

[0035] Specifically, the edge contour detection of the enhanced road image to obtain the marking contour data is performed based on a color threshold. Since the colors in the enhanced road image have been enhanced, the contour can be confirmed by identifying the regions with a large difference in color intensity in the enhanced road image.

[0036] Specifically, the Hough transform is a technique widely used in computer vision and image processing, mainly used to detect geometric shapes in images, such as straight lines, circles, and other parametric curves. The Hough transform transforms the shapes (such as straight lines) in the image from the spatial domain to the parameter domain. In the parameter domain, the "votes" or intersections of each possible shape parameter are recorded through an accumulator array, thereby detecting the shapes in the image. For straight line detection, the accumulator array records the slope and intercept of the straight line.

[0037] In the embodiment of the present invention, the positioning of the road markings in the forward-looking road image based on the marking contour data and the shape data to obtain the road marking data means identifying the regions in the forward-looking road image with colors of white or yellow and shapes of straight lines or curves to obtain the road marking data.

[0038] Specifically, the lightweight road segmentation model is a neural network model that can, while maintaining a high segmentation accuracy, optimize the network structure and computational efficiency to enable it to run in real time on resource-constrained devices (such as embedded systems or mobile devices).

[0039] In the embodiment of the present invention, the use of the pre-trained lightweight road segmentation model to segment the forward-looking image to obtain the forward-looking road image includes: Performing normalization processing on the forward-looking image to obtain a normalized forward-looking image; Performing dimension transformation processing on the normalized forward-looking image to obtain a transformed image; Performing inference on the transformed image by calling the forward propagation function of the lightweight road segmentation model to obtain a probability distribution tensor; Based on a preset probability threshold, converting the probability distribution tensor into a two-dimensional array, and using the two-dimensional array to segment the forward-looking image to obtain the forward-looking road image.

[0040] Specifically, the normalization process maps the pixel value range of the image to between [0, 1] or [-1, 1] to improve the processing efficiency of the model.

[0041] Specifically, the dimensionality conversion process for the pre-normalized front view image converts the dimensions of the image from (height, width, channels) to (channels, height, width).

[0042] Specifically, the probability distribution tensor refers to the distribution of the probability that any pixel point in the image is a road in the road segmentation task.

[0043] Specifically, the conversion of the probability distribution tensor into a two-dimensional array based on a preset probability threshold means converting the elements in the probability distribution tensor that are greater than the probability threshold into 1 and those less than the probability threshold into 0.

[0044] Specifically, the use of the two-dimensional array to segment the front view image means segmenting the front view image according to the part where the value in the two-dimensional array is 1.

[0045] In the embodiment of the present invention, by performing a targeted analysis of road markings based on the front view image data to obtain road marking data, the efficiency of identifying road markings can be improved.

[0046] S3. Perform a targeted analysis of the positions of surrounding vehicles based on the surround view image data to obtain surrounding vehicle data.

[0047] In the embodiment of the present invention, the targeted analysis of the positions of surrounding vehicles based on the surround view image data to obtain surrounding vehicle data means identifying and positioning the surrounding vehicles based on the surround view image data, and analyzing and predicting the running trajectories by analyzing the position changes of the vehicles in consecutive pictures.

[0048] In the embodiment of the present invention, the targeted analysis of the positions of surrounding vehicles based on the surround view image data to obtain surrounding vehicle data includes: Perform vehicle target recognition on the surround view image data to obtain target recognition data; Use a multi-target tracking algorithm to track the positions of the vehicles in the surround view image data based on the target recognition data to obtain vehicle trajectories; Perform short-term trajectory prediction on the vehicles in the surround view image data based on the vehicle trajectories to obtain predicted trajectories; Summarize the target recognition data, the vehicle trajectories, and the predicted trajectories to obtain the surrounding vehicle data.

[0049] Specifically, the vehicle target recognition of the surround view image data can be achieved through a pre-trained lightweight target detection model. The lightweight target detection model can be YOLOv8n, which is a lightweight version in the YOLOv8 series and is designed for scenarios that require efficient computing resources. It inherits the core advantages of YOLOv8, such as high accuracy and fast inference speed, and further reduces the number of parameters by optimizing the network structure. YOLOv8n supports multiple tasks, including target detection, image segmentation, and pose estimation, and is suitable for embedded devices and real-time application scenarios.

[0050] Specifically, the position tracking of the vehicles in the surround view image data by using the multi-object tracking algorithm based on the target recognition data is to extract the bounding boxes of the vehicles in each picture according to the target recognition data, track the positions of the same vehicle in consecutive surround view images, and extract key features such as trajectory length, driving speed, and acceleration according to the vehicle coordinates and timestamp information obtained by the tracking.

[0051] In the embodiment of the present invention, the short-term trajectory prediction of the vehicles in the surround view image data based on the vehicle trajectories is to establish a vehicle motion model based on the Kalman filter algorithm and predict the running trajectories based on the driving speed and acceleration of the vehicle at each timestamp.

[0052] Specifically, the Kalman Filter is an efficient recursive estimation algorithm for the state estimation of dynamic systems. It provides the optimal estimation (minimum mean square error) of the system state in the presence of noise by combining the mathematical model of the system and the observed data.

[0053] In the embodiment of the present invention, by performing a targeted analysis of the positions of surrounding vehicles according to the surround view image data to obtain surrounding vehicle data, the efficiency of identifying surrounding vehicles can be improved.

[0054] S4. Perform targeted recognition of road pedestrians according to the side view image data to obtain road pedestrian data.

[0055] In the embodiment of the present invention, the targeted recognition of road pedestrians according to the side view image data to obtain road pedestrian data is to perform target recognition on the side view image data based on a neural network model to obtain the position information of road pedestrians, and then perform pedestrian trajectory analysis and short-term trajectory prediction based on the positions of road pedestrians in consecutive images.

[0056] S5. Convert the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data, and perform rapid three-dimensional space modeling based on the comprehensive coordinate data to obtain an intelligent driving space model.

[0057] In an embodiment of the present invention, the road marking data, the surrounding vehicle data, and the road pedestrian data are converted into coordinate data to obtain comprehensive coordinate data, and based on the comprehensive coordinate data, rapid three-dimensional space modeling is performed to obtain an intelligent driving space model.

[0058] In an embodiment of the present invention, since the road marking data, the surrounding vehicle data, and the road pedestrian data are obtained by shooting and identifying with multiple cameras, binocular positioning can be performed on the same target through two images at the same timestamp.

[0059] In an embodiment of the present invention, the road marking data includes distance data identified from different pictures of the same marking.

[0060] In an embodiment of the present invention, referring to Figure 3 As shown, a flowchart of coordinate data conversion provided by an embodiment of the present invention is shown. The conversion of the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data includes: S31. Obtain the first marking distance and the second marking distance included in the road marking data, and calculate the marking position according to the first marking distance, the second marking distance, and the preset front view camera distance; S32. Obtain the first vehicle distance and the second vehicle distance included in the surrounding vehicle data, and calculate the vehicle position according to the first vehicle distance, the second vehicle distance, and the preset surround view camera distance; S33. Obtain the first pedestrian distance and the second pedestrian distance included in the road pedestrian data, and calculate the pedestrian position according to the first pedestrian distance, the second pedestrian distance, and the preset side view camera distance; S34. Establish a two-dimensional rectangular coordinate system with the target vehicle as the origin and parallel to the horizontal plane; S35. Based on the two-dimensional rectangular coordinate system, convert the marking position, the vehicle position, and the pedestrian position into coordinate data to obtain comprehensive coordinate data.

[0061] In an embodiment of the present invention, the rapid three-dimensional space modeling based on the comprehensive coordinate data to obtain an intelligent driving space model is to select a vehicle model, a pedestrian model, and a road marking model from a pre-established model library, and then place the vehicle model, the pedestrian model, and the road marking model into a pre-established three-dimensional model space based on the coordinate data, and perform short-term prediction trajectory simulation on the vehicle and the pedestrian based on the vehicle prediction trajectory data and the pedestrian prediction trajectory data included in the surrounding vehicle data and the road pedestrian data.

[0062] In the embodiments of the present invention, by converting the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data, the efficiency of model establishment can be improved. By performing rapid three-dimensional space modeling based on the comprehensive coordinate data to obtain an intelligent driving space model, the accuracy of generating driving suggestions can be improved.

[0063] S6. Generate driving suggestions according to the intelligent driving space model.

[0064] In the embodiments of the present invention, since the surrounding vehicle data includes the speed data and trajectory prediction data of surrounding vehicles, and the road pedestrian data includes pedestrian trajectory prediction data. Therefore, the intelligent driving space model can simulate the predicted trajectories of surrounding vehicles and pedestrians, and combined with the vehicle speed of the target vehicle, can generate more accurate and reasonable driving suggestions.

[0065] In the embodiments of the present invention, the generation of driving suggestions according to the intelligent driving space model means identifying the information of the vehicles (including position, speed, future short-term trajectory) and pedestrians (including position and movement trajectory) around the target vehicle according to the intelligent driving space model, and generating driving suggestions based on these position information. For example, when a pedestrian appears in front of the target vehicle, the generated driving suggestion is "Pay attention to decelerating and avoiding the pedestrian in front". When the vehicle speed of the vehicle in front is too slow and there is no vehicle on the left or the vehicle behind in the left lane has not accelerated, the generated driving suggestion is to accelerate and overtake the vehicle in front from the left lane. When the vehicle speed of the vehicle in front suddenly drops, the generated driving suggestion is "Search for the vehicle in front, pay attention to avoidance".

[0066] In the embodiments of the present invention, by generating driving suggestions according to the intelligent driving space model, the driving safety can be improved.

[0067] S7. Real-time collect the wide-angle image data of the target vehicle, and perform three-dimensional rough modeling according to the wide-angle image data set to obtain a rough intelligent driving space model.

[0068] In the embodiments of the present invention, the real-time collection of the wide-angle image data of the target vehicle and the performance of three-dimensional rough modeling according to the wide-angle image data set to obtain a rough intelligent driving space model include: Perform wide-angle distortion correction processing on the wide-angle image data to obtain a corrected image; Perform feature extraction of target objects on the corrected image to obtain target object features; Perform fast approximate feature matching using the target object features to obtain target recognition data.

[0069] Use a three-dimensional reconstruction algorithm to obtain the target objects in the corrected image according to the target recognition data to obtain the three-dimensional coordinates of the target objects. Perform 3D modeling based on the 3D coordinates of the target object to obtain a rough intelligent driving space model.

[0070] Specifically, the use of the target object features for fast approximate feature matching to obtain target recognition data means sequentially matching the target object features with the features in a preset feature database, and when the matching degree reaches the preset matching degree threshold, it is determined that the matching is successful.

[0071] Specifically, the feature database contains a large number of road marking features, vehicle contour features, and pedestrian contour features.

[0072] Specifically, the size of the preset matching degree threshold can be set according to the task requirements. The larger the matching degree threshold, the higher the matching efficiency, but the lower the matching accuracy.

[0073] S8. Compare the intelligent driving space model with the rough intelligent driving space model in real time to obtain a model error.

[0074] In the embodiment of the present invention, the comparison of the intelligent driving space model with the rough intelligent driving space model in real time to obtain a model error includes: Obtain the number of surrounding vehicles and the number of surrounding pedestrians in the intelligent driving space model, and obtain the rough number of surrounding vehicles and the rough number of surrounding pedestrians in the rough intelligent driving space model.

[0075] Establish a 3D coordinate system with the target vehicle as the origin; Based on the 3D coordinate system, obtain the coordinate data of the intelligent driving space model and the rough intelligent driving space model to obtain model 3D data and rough model 3D data; Determine the model volume and model space range of the rough intelligent driving space model according to the model 3D data; Determine the rough model volume and rough model space range of the rough intelligent driving space model according to the rough model 3D data; Calculate the volume that does not overlap between the intelligent driving space model and the rough intelligent driving space model according to the model volume, the model space range, the rough model volume, and the rough model space range to obtain a non-overlapping volume; Calculate the model error according to the model volume, the rough model volume, the non-overlapping volume, the rough number of surrounding vehicles, and the rough number of surrounding pedestrians.

[0076] Specifically, calculating the model error according to the model volume, the rough model volume, the non-coincident volume, the rough number of surrounding vehicles, and the rough number of surrounding pedestrians includes: Calculating the model error using the following formula: where, is the model error, is the model volume, is the rough model volume, is the non-coincident volume, is the base of the natural logarithm, is the number of surrounding pedestrians, is the rough number of surrounding pedestrians, is the number of surrounding vehicles, is the rough number of surrounding vehicles.

[0077] S9. Determine whether the model error is greater than or equal to a preset error threshold.

[0078] If the model error is less than the error threshold, then execute S10. Broadcast the driving advice to the user.

[0079] In the embodiments of the present invention, when the model error is less than the error threshold, it indicates that the accuracy of the intelligent driving space model is high, and the driving advice generated according to the intelligent driving space model is highly reliable.

[0080] In the embodiments of the present invention, broadcasting the driving advice to the user may be to convert the driving advice into voice data and then broadcast it using a speaker provided in the target vehicle.

[0081] If the model error is greater than or equal to the error threshold, then execute S11. Adjust the driving advice in real time according to the model error.

[0082] In the embodiments of the present invention, when the model error is greater than or equal to the error threshold, it indicates that the road conditions where the target vehicle is located are relatively complex, the intelligent driving space model may have errors, and the driving advice generated according to the intelligent driving space model is less reliable. Therefore, it is necessary to adjust the driving advice.

[0083] In the embodiments of the present invention, the real-time adjustment of the driving advice according to the model error means adjusting the driving advice to a safer and more conservative one. For example, when the speed of the vehicle ahead is low, the advice to overtake from the left lane will only be given when there is no vehicle in the left lane. If there is a vehicle behind or in front in the left lane, the advice to overtake from the left lane will not be generated. For example, when there is a vehicle in front or behind, lowering the threshold for triggering the reminder to keep a safe distance will make it easier to generate the driving advice of "pay attention to keeping a safe distance".

[0084] In the embodiments of the present invention, after the driving advice is adjusted in real time according to the model error, return to S10 and announce the driving advice to the user.

[0085] In the embodiments of the present invention, when the model error is greater than or equal to the error threshold, by re-executing the step of announcing the driving advice to the user after adjusting the driving advice in real time according to the model error, the safety of intelligent driving in complex road conditions is greatly improved.

[0086] As Figure 4 shown, it is a schematic structural diagram of an electronic device for the method of rapid image processing during intelligent driving provided by the embodiments of the present invention.

[0087] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and operable on the processor 10, such as a program for rapid image processing during intelligent driving.

[0088] Among them, in some embodiments, the processor 10 may be composed of integrated circuits. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and circuits, and by running or executing programs or modules stored in the memory 11 (such as the program for rapid image processing during intelligent driving, etc.), and calling the data stored in the memory 11, to perform various functions of the electronic device and process data.

[0089] The memory 11 includes at least one type of readable storage medium, which includes flash memory, mobile hard disks, multimedia cards, card-type memories (such as SD or DX memories, etc.), magnetic memories, magnetic disks, optical disks, etc. The memory 11 can be an internal storage unit of the electronic device in some embodiments, such as the mobile hard disk of the electronic device. The memory 11 can also be an external storage device of the electronic device in some other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device. Further, the memory 11 can also include both the internal storage unit and the external storage device of the electronic device. The memory 11 can be used not only to store application software installed on the electronic device and various types of data, such as the code of the image fast processing program during intelligent driving, etc., but also to temporarily store the data that has been output or will be output.

[0090] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable the connection and communication between the memory 11 and at least one processor 10, etc.

[0091] The communication interface 13 is used for the communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually used to establish a communication connection between this electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)). Optionally, the user interface can also be a standard wired interface and a wireless interface. Optionally, in this embodiment, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display can also be appropriately referred to as a display screen or a display unit, and is used to display the information processed in the electronic device and to display a visual user interface.

[0092] Only an electronic device with components is shown in the figure. Those skilled in the art can understand that the structure shown in the figure does not constitute a limitation on the electronic device, and it may include fewer or more components than shown in the figure, or combine some components, or have different component arrangements.

[0093] For example, although not shown, the electronic device may further include a power source (such as a battery) for supplying power to each component. Preferably, the power source can be logically connected to the at least one processor 10 through a power management device, so as to implement functions such as charging management, discharging management, and power consumption management through the power management device. The power source may also include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, and a power status indicator. The electronic device may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0094] It should be understood that the embodiments are only for illustrative purposes and are not limited by this structure in the scope of the patent application.

[0095] The intelligent driving image fast processing program stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can implement: Real-time collect the front view image data, surround view image data, and side view image data during the driving process of the target vehicle; Perform road marking target analysis based on the front view image data to obtain road marking data; Perform surrounding vehicle position target analysis based on the surround view image data to obtain surrounding vehicle data; Perform road pedestrian target recognition based on the side view image data to obtain road pedestrian data; Convert the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data, and perform three-dimensional space fast modeling based on the comprehensive coordinate data to obtain an intelligent driving space model; Generate driving suggestions according to the intelligent driving space model; Real-time collect the wide-angle image data of the target vehicle, and perform three-dimensional rough modeling based on the wide-angle image data set to obtain a rough intelligent driving space model; Perform real-time comparison between the intelligent driving space model and the rough intelligent driving space model to obtain a model error; Judge whether the model error is greater than or equal to a preset error threshold; If the model error is less than the error threshold, broadcast the driving suggestions to the user; If the model error is greater than or equal to the error threshold, after adjusting the driving advice in real time according to the model error, return to the step of broadcasting the driving advice to the user.

[0096] Specifically, for the specific implementation method of the above instructions by the processor 10, reference can be made to the description of the relevant steps in the corresponding embodiments of the attached drawings, which will not be elaborated here.

[0097] Furthermore, if the modules / units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory).

[0098] The present invention also provides a computer-readable storage medium, where the readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, it can implement: Real-time collect the front view image data, surround view image data, and side view image data during the driving of the target vehicle; Perform road marking target analysis based on the front view image data to obtain road marking data; Perform surrounding vehicle position target analysis based on the surround view image data to obtain surrounding vehicle data; Perform road pedestrian target recognition based on the side view image data to obtain road pedestrian data; Convert the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data, and perform three-dimensional space rapid modeling based on the comprehensive coordinate data to obtain an intelligent driving space model; Generate driving advice according to the intelligent driving space model; Real-time collect the wide-angle image data of the target vehicle, and perform three-dimensional rough modeling according to the wide-angle image data set to obtain a rough intelligent driving space model; Compare the intelligent driving space model with the rough intelligent driving space model in real time to obtain a model error; Judge whether the model error is greater than or equal to a preset error threshold; If the model error is less than the error threshold, broadcast the driving advice to the user; If the model error is greater than or equal to the error threshold, after adjusting the driving advice in real time according to the model error, return to the step of broadcasting the driving advice to the user.

[0099] In the embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0100] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0102] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0103] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed within the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights.

[0104] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0105] In addition, obviously, the term "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or apparatuses stated in the system claims can also be implemented by one unit or apparatus through software or hardware. The terms such as first and second are used to represent names and do not represent any specific order.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for rapid image processing during intelligent driving, characterized in that, The method includes: Real-time collecting front view image data, panoramic view image data, and side view image data during the driving process of the target vehicle; Performing targeted analysis on road markings based on the front view image data to obtain road marking data; Performing targeted analysis on the positions of surrounding vehicles based on the panoramic view image data to obtain surrounding vehicle data; Performing targeted recognition of road pedestrians based on the side view image data to obtain road pedestrian data; Converting the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data, and performing rapid three-dimensional space modeling based on the comprehensive coordinate data to obtain an intelligent driving space model; Generating driving suggestions according to the intelligent driving space model; Real-time collecting wide-angle image data of the target vehicle, and performing three-dimensional rough modeling based on the wide-angle image data set to obtain a rough intelligent driving space model; Performing real-time comparison between the intelligent driving space model and the rough intelligent driving space model to obtain a model error; Judging whether the model error is greater than or equal to a preset error threshold; If the model error is less than the error threshold, then broadcasting the driving suggestions to the user; If the model error is greater than or equal to the error threshold, then after adjusting the driving suggestions in real time according to the model error, returning to the step of broadcasting the driving suggestions to the user.

2. The method for fast image processing during intelligent driving according to claim 1, wherein The performing targeted analysis on road markings based on the front view image data to obtain road marking data includes: Using a pre-trained lightweight road segmentation model to segment the front view image to obtain a front view road image; Performing color enhancement on the front view road image using the HSV color space to obtain an enhanced road image; Performing edge contour detection on the enhanced road image to obtain marking contour data; Detecting straight lines and curves in the enhanced road image based on the Hough transform to obtain shape data; Locating road markings in the front view road image based on the marking contour data and the shape data to obtain road marking data.

3. The method for rapid image processing during intelligent driving according to claim 2, wherein, The using a pre-trained lightweight road segmentation model to segment the front view image to obtain a front view road image includes: Performing normalization processing on the front view image to obtain a normalized front view image; Performing dimension transformation processing on the normalized front view image to obtain a transformed image; Inferring the transformed image by calling the forward propagation function of the lightweight road segmentation model to obtain a probability distribution tensor; Converting the probability distribution tensor into a two-dimensional array based on a preset probability threshold, and using the two-dimensional array to segment the front view image to obtain a front view road image.

4. The method for rapid image processing during intelligent driving according to claim 1, characterized in that, The performing targeted analysis on the positions of surrounding vehicles based on the panoramic view image data to obtain surrounding vehicle data includes: Performing vehicle target recognition on the panoramic view image data to obtain target recognition data; Using a multi-target tracking algorithm to track the positions of vehicles in the panoramic view image data based on the target recognition data to obtain vehicle trajectories; Performing short-term trajectory prediction on the vehicles in the panoramic view image data based on the vehicle trajectories to obtain predicted trajectories; Summarize the target recognition data, the vehicle trajectory, and the predicted trajectory to obtain the surrounding vehicle data.

5. The method for rapid image processing during intelligent driving according to claim 1, characterized in that, Converting the road marking data, the surrounding vehicle data, and the road pedestrian data into coordinate data to obtain comprehensive coordinate data includes: Obtain the first marking distance and the second marking distance included in the road marking data, and calculate the marking position according to the first marking distance, the second marking distance, and the preset front view camera distance; Obtain the first vehicle distance and the second vehicle distance included in the surrounding vehicle data, and calculate the vehicle position according to the first vehicle distance, the second vehicle distance, and the preset surround view camera distance; Obtain the first pedestrian distance and the second pedestrian distance included in the road pedestrian data, and calculate the pedestrian position according to the first pedestrian distance, the second pedestrian distance, and the preset side view camera distance; Establish a two-dimensional rectangular coordinate system with the target vehicle as the origin and parallel to the horizontal plane; Based on the two-dimensional rectangular coordinate system, convert the marking position, the vehicle position, and the pedestrian position into coordinate data to obtain comprehensive coordinate data.

6. The method for rapid image processing during intelligent driving according to claim 1, characterized in that Collect the wide-angle image data of the target vehicle in real time, and perform three-dimensional rough modeling based on the wide-angle image data set to obtain a rough intelligent driving space model, including: Perform wide-angle distortion correction processing on the wide-angle image data to obtain a corrected image; Perform feature extraction of the target object on the corrected image to obtain the target object features; Use the target object features for fast approximate feature matching to obtain target recognition data; use a three-dimensional reconstruction algorithm to obtain the target object in the corrected image according to the target recognition data to obtain the three-dimensional coordinates of the target object; Perform three-dimensional modeling based on the three-dimensional coordinates of the target object to obtain a rough intelligent driving space model.

7. The method for rapid image processing during intelligent driving according to claim 1, characterized in that Compare the intelligent driving space model with the rough intelligent driving space model in real time to obtain a model error, including: Obtain the number of surrounding vehicles and the number of surrounding pedestrians in the intelligent driving space model, and obtain the rough number of surrounding vehicles and the rough number of surrounding pedestrians in the rough intelligent driving space model; Establish a three-dimensional coordinate system with the target vehicle as the origin; Based on the three-dimensional coordinate system, obtain the coordinate data of the intelligent driving space model and the rough intelligent driving space model to obtain the model three-dimensional data and the rough model three-dimensional data; Determine the model volume and the model space range of the rough intelligent driving space model according to the model three-dimensional data; Determine the rough model volume and the rough model space range of the rough intelligent driving space model according to the rough model three-dimensional data; Calculate the volume that does not overlap between the intelligent driving space model and the rough intelligent driving space model according to the model volume, the model space range, the rough model volume, and the rough model space range to obtain the non-overlapping volume; Calculate the model error according to the model volume, the rough model volume, the non-overlapping volume, the rough number of surrounding vehicles, and the rough number of surrounding pedestrians.

8. The method for rapid image processing during intelligent driving according to claim 7, characterized in that Calculating a model error according to the model volume, the rough model volume, the non-coincident volume, the approximate number of surrounding vehicles, and the approximate number of surrounding pedestrians includes: The model error is calculated using the following formula: where is the model error, is the model volume, is the rough model volume, is the non - overlapping volume, is the base of the natural logarithm, is the number of surrounding pedestrians, is the rough number of surrounding pedestrians, is the number of surrounding vehicles, is the rough number of surrounding vehicles.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for fast image processing during intelligent driving according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for fast image processing during intelligent driving according to any one of claims 1 to 7.

Images