A method and system for visual environment perception of intelligent connected vehicles
Through the intelligent connected vehicle visual environment perception method, image analysis is performed using vision sensors and multiple neural network models, the problem of expensive lidar sensors and poor performance in bad weather is solved, and efficient and accurate environmental perception and safety improvement is achieved.
Patent Information
- Application Number
- CN202411623141.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-03-14
- Estimated Expiration
- 2044-11-14
AI Technical Summary
Existing smart car environment perception systems rely on expensive lidar sensors, and their performance and stability are affected in severe weather.
The intelligent connected vehicle visual environment perception method is adopted, and the image data is obtained using the automotive vision sensor, and the feature attributes are preprocessed, segmented and identified, and transmitted to multiple neural network models for image analysis, and the processing results are integrated to output environmental information.
It improves image processing efficiency and analysis accuracy, enhances the flexibility and adaptability of the system, and improves the safety of intelligent connected vehicles.
Smart Images
Figure CN119131738B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of autonomous driving and relates to a method and system for visual environment perception of an intelligent networked vehicle. Background Art
[0002] Self-driving cars, also known as car self-driving systems, mainly collect information about the car's surrounding environment through sensors, and then make decisions on the car's actuators after analysis, fusion and processing by the computer system, so as to realize the acceleration, deceleration, left and right steering and braking of the car. The car environment perception system is to continuously collect information about the external environment of the car through the environment perception sensors installed on the car body, identify static and moving objects in the surrounding environment, detect and track the identified objects, and then use the corresponding designed algorithms to determine whether the object is a target object and the degree of threat to the car. At present, the common solution of the intelligent car environment perception system is to produce 3D street view map technology through the 360-degree scanning laser radar on the roof, and realize the intelligent car environment perception with laser radar sensor technology as the core. However, the price of laser radar sensors is expensive, and the preparation and R&D costs are high. In addition, the performance and stability of laser radar are easily affected in severe weather such as fog, heavy rain and blizzard. Summary of the invention
[0003] In order to solve the problems existing in the background technology, the present invention proposes a method and system for visual environment perception of an intelligent connected vehicle.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0005] A method for visual environment perception of an intelligent connected vehicle, comprising:
[0006] First, the car vision sensor is used to obtain image data around the car.
[0007] The image data around the car is preprocessed to obtain a first image, and the first image is segmented to obtain a second image set.
[0008] Each image in the second image set is identified and its corresponding feature attribute is marked.
[0009] Each image in the second image set is transmitted to the corresponding neural network according to its characteristic attributes to perform image analysis operations.
[0010] A plurality of first neural network models are provided, each of which has different processing capabilities and processes different data, and each first neural network model processes each image data differently according to the characteristic attributes of each image in the second image set.
[0011] Multiple first neural network models can process the same image data, the first neural network model processes a single frame of the image data, and the second neural network model processes multiple frames of the image data in a time series.
[0012] The second neural network model determines whether the multiple frames of images in the time series being processed include the single frame image processed by the first neural network model. If included, the second neural network model receives the results of the processing by the first neural network model during the processing process, and makes corrections during the processing of the multiple frames of images in the time series.
[0013] After the processing is completed, the processing results are uploaded to the third neural network model, and the third neural network model integrates the analysis results of all the first neural network models.
[0014] Post-process each image analysis result in the second image set and output the vehicle surrounding environment information.
[0015] Furthermore, the specific method of preprocessing the image data around the car to obtain the first image is:
[0016] First, filter the image data around the car and define a filter window.
[0017] By using the mean value of the pixel points of the image data around the car in the window, all pixels in the image data around the car are filtered. The specific formula is:
[0018]
[0019] Where I(m,n) is the pixel value of the original pixel in the image data around the car in the window, I'(x,y) is the pixel value after denoising, K is the number of pixels in the image data around the car in the window, and S is the coordinate set of the pixel points in the window. By moving the window position, the filtering operation of the entire image data around the car is completed to obtain the first image.
[0020] Furthermore, the specific method of segmenting the first image is:
[0021] In the first image, a plurality of seed pixels (p1, p2, ..., p i ) as a starting point.
[0022] Determine the standard for judging similarity and set the threshold T to judge the similarity between multiple pixels. The specific formula is:
[0023] |I(p)-I(p i )| <T;
[0024] Where I(p) is the grayscale value of pixel p.
[0025] Check the seed pixel p i The neighboring pixels of the region are added to the region. If the neighboring pixels meet the similarity criteria, they are added to the region and used as new seed pixels. The seed pixels are expanded and the expansion is stopped when no more neighboring pixels can be added.
[0026] The first image is segmented according to the expansion area of each seed pixel to obtain a set of second images.
[0027] Furthermore, the specific method of identifying each image in the second image set and marking its corresponding characteristic attribute is:
[0028] Perform grayscale conversion on each image in each second image set, convert all images in the second image set into grayscale images, calculate the gradient value of each image in the x direction and the y direction, and use I x and I y To express.
[0029] Using the gradient value I of each image in the x direction x And the gradient value I of each image in the y direction y The calculation result tensor M is as follows:
[0030]
[0031] Each image in the second image set is converted to grayscale and the following response function is used to detect corner points:
[0032] R(x,y)=det(M)-k·(trace(M)) 2 ;
[0033] in, det(M) is the determinant of the matrix M, trace(M) is the trace of the matrix M, and k is an empirical constant.
[0034] By setting the threshold T, pixels whose response value R is greater than T are selected as corner points:
[0035] R(x,y)>T;
[0036] In this way, the features of the corner points of each image in the second image set are extracted, and corresponding feature attributes are added to each image according to the features of the corner points of each image.
[0037] Furthermore, the specific method of transmitting each image in the second image set to the corresponding neural network according to its characteristic attributes to perform the image analysis operation is:
[0038] A plurality of first neural network models are provided, each of which has different processing capabilities and processes different data, and each first neural network model processes each image data differently according to the characteristic attributes of each image in the second image set.
[0039] Multiple first neural network models can process the same image data, the first neural network model processes a single frame of the image data, and the second neural network model processes multiple frames of the image data in a time series.
[0040] The second neural network model determines whether the multiple frames of images in the time series being processed include the single frame image processed by the first neural network model. If included, the second neural network model receives the results of the processing by the first neural network model during the processing process, and makes corrections during the processing of the multiple frames of images in the time series.
[0041] After the processing is completed, the processing results are uploaded to the third neural network model, and the third neural network model integrates the analysis results of all the first neural network models.
[0042] Furthermore, the first neural network model includes a convolutional neural network and a recurrent neural network.
[0043] Convolutional neural networks are used to detect objects in single-frame images acquired by visual sensors.
[0044] Recurrent neural networks are used to process data of multiple frames of continuous images acquired by visual sensors in time series.
[0045] On the other hand, the present invention also provides a visual environment perception system for an intelligent networked vehicle, comprising:
[0046] The data acquisition module uses the vehicle vision sensor to acquire the image data around the vehicle.
[0047] The data preprocessing module preprocesses the image data around the car and segments the preprocessed image data around the car.
[0048] The data processing module identifies each image, marks its corresponding feature attributes, and transmits them to the corresponding neural network based on the feature attributes of the image to perform image analysis operations.
[0049] Data output module outputs information about the car's surroundings.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] The present invention improves the efficiency of image processing and ensures the accuracy of subsequent analysis by systematically preprocessing and segmenting the acquired image data. At the same time, the present invention also introduces multiple neural network models for division of labor, which can not only perform detailed analysis on single-frame images, but also comprehensively process multi-frame time series data. The second neural network model can receive and correct the results of the first neural network in real time, which improves the accuracy and response speed of data processing and enhances the flexibility and adaptability of the system.
[0052] By accurately identifying surrounding environment information, the system can provide effective support for driving decisions, thereby improving the safety of intelligent connected vehicles. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is a flow chart of the method of the present invention.
[0054] Figure 2 It is a system structure diagram of the present invention. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0056] like Figure 1 As shown, a method for visual environment perception of an intelligent connected vehicle includes the following steps:
[0057] First, the car vision sensor is used to obtain image data around the car.
[0058] The image data around the car is preprocessed to obtain a first image, and the first image is segmented to obtain a second image set.
[0059] Each image in the second image set is identified and its corresponding feature attribute is marked.
[0060] Each image in the second image set is transmitted to the corresponding neural network according to its characteristic attributes to perform image analysis operations.
[0061] Post-process each image analysis result in the second image set and output the vehicle surrounding environment information.
[0062] First, the car's visual sensors are used to obtain rich image data around the car in all directions, which is the basis for subsequent processing.
[0063] Subsequently, the collected image data around the car is subjected to fine preprocessing operations, and after algorithm filtering and optimization, a clearer and more accurate first image is obtained.
[0064] Perform filtering operations on the image data around the car and define a filtering window. The size of the filtering window can be customized by the user.
[0065] By using the mean value of the pixel points of the image data around the car in the window, all pixels in the image data around the car are filtered. The specific formula is:
[0066]
[0067] Where I(m,n) is the pixel value of the original pixel in the image data around the car in the window, I'(x,y) is the pixel value after denoising, K is the number of pixels in the image data around the car in the window, and S is the coordinate set of the pixel points in the window. By moving the window position, the filtering operation of the entire image data around the car is completed to obtain the first image.
[0068] The first image after the filtering operation eliminates the noise in the original image, and the characteristic signals or image details in the first image are more prominent.
[0069] Next, the first image is segmented reasonably using advanced image segmentation technology to form a second image set containing information of different regions.
[0070] In the first image, a plurality of seed pixels (p1, p2, ..., p i ) as a starting point.
[0071] Determine the standard for judging similarity and set the threshold T to judge the similarity between multiple pixels. The specific formula is:
[0072] |I(p)-I(p i )| <T;
[0073] The calculation is performed using the grayscale value of the pixel, where I(p) is the grayscale value of pixel p.
[0074] Check the seed pixel p i The neighboring pixels of the region are added to the region. If the neighboring pixels meet the similarity criteria, they are added to the region and used as new seed pixels. The seed pixels are then expanded until there are no more neighboring pixels to be added. The expansion process is stopped.
[0075] The first image is segmented according to the expansion area of each seed pixel to obtain a set of second images.
[0076] The advantage of this image segmentation method is that the steps are clear and definite. From the selection of the starting point to the setting of the similarity judgment standard, to the expansion of the seed pixels and the final image segmentation, each link has a specific operation process, which makes the whole process highly logical and operational, and helps to accurately segment the image and improve the efficiency and accuracy of image analysis.
[0077] Then, each image in the second image set is deeply identified, and its corresponding feature attributes are accurately marked using a professional image recognition algorithm.
[0078] For each image in each second image set, a grayscale value conversion operation needs to be performed. Specifically, all images in the second image set are converted into grayscale images through specific algorithms and processing methods. After completing the grayscale conversion of the image, the gradient values of each image in the horizontal direction (x direction) and the vertical direction (y direction) are further calculated. This calculation process involves the analysis of the change in the grayscale value of the image pixels. By calculating the difference in the grayscale values of adjacent pixels in different directions, the gradient value of the image in this direction is determined. This is of great significance for subsequent tasks such as image processing, feature extraction, and image analysis, and can provide basic data and key information for a deeper understanding and processing of images. For example, in image edge detection, the calculation of gradient values can help accurately locate the edge contours of objects in the image, thereby realizing operations such as recognition and analysis of target objects in the image. The gradient values of the image in the horizontal direction (x direction) and the vertical direction (y direction) are represented by I x and I y To express.
[0079] Using the gradient value I of each image in the x direction x And the gradient value I of each image in the y direction y The calculation result tensor M is as follows:
[0080]
[0081] Each image in the second image set is converted to grayscale and the following response function is used to detect corner points:
[0082] R(x,y)=det(M)-k·(trace(M)) 2 ;
[0083] in, det(M) is the determinant of the matrix M, trace(M) is the trace of the matrix M, and k is an empirical constant.
[0084] By setting the threshold T, pixels whose response value R is greater than T are selected as corner points:
[0085] R(x,y)>T;
[0086] In this way, the features of the corner points of each image in the second image set are extracted, and corresponding feature attributes are added to each image according to the features of the corner points of each image.
[0087] Corner points are mainly detected based on the geometric structure of the image, that is, the change pattern of pixel grayscale. Under a certain degree of lighting change, as long as the geometric structure at the corner point is not destroyed, it can still be detected. For example, in an image of an outdoor scene, the light intensity and angle will change with the time of day, but the corner position and features of a building or object will usually not disappear due to these lighting changes.
[0088] Afterwards, based on these characteristic attributes, each image is accurately transmitted to the corresponding neural network, allowing the neural network to exert its powerful computing and analysis capabilities to perform image analysis operations.
[0089] In this method, multiple secondary neural network models are configured, each of which has a unique processing capability and processes different data. The secondary neural network model includes a first neural network model and a second neural network model.
[0090] Each first neural network model will perform different processing operations on the corresponding image data based on the characteristic attributes of each image in the second image set.
[0091] The multiple first neural network models have the ability to process the same image data. For example, the first neural network model specifically processes a single frame of the image data, focusing on analyzing various types of information in the single frame.
[0092] The second neural network model is responsible for processing the multiple frames of the image data in the time series. In addition, the second neural network model will determine whether the multiple frames of the time series it processes contain the single frame image processed by the first neural network model. If so, during the processing, the second neural network model will receive the result of the processing completed by the first neural network model, and make corrections in the processing of the multiple frames of the time series based on this result.
[0093] When all secondary neural network models have completed processing, they will upload their respective processing results to the third neural network model. The third neural network model plays an integrating role as the main neural network model, and comprehensively and systematically integrates the analysis results of all secondary neural network models.
[0094] The advantages of this architecture are very significant. Multiple secondary neural network models have clear division of labor and carry out targeted processing according to the different characteristics and needs of the image, which improves the professionalism and efficiency of the processing. At the same time, the collaboration and information transmission between different models, such as the reception and use of the results of the first neural network model by the second neural network model, enhances the comprehensive analysis ability of image data and can understand and process image information more comprehensively and accurately. Finally, the integrated processing of the main neural network model enables the results of each secondary model to be unified and coordinated, providing a more complete and reliable basis for subsequent decision-making or further analysis, and improving the performance and intelligence level of the entire system.
[0095] The secondary neural network model covers convolutional neural networks and recurrent neural networks. Among them, the first neural network model in the secondary neural network model is a convolutional neural network, which plays an important role. It is mainly responsible for accurately detecting objects in single-frame images acquired by visual sensors. Through its unique algorithm and structure, it can effectively identify various object features in the image, providing a basis for subsequent analysis. The second neural network model in the secondary neural network model is a recurrent neural network, which focuses on data processing of multiple frames of continuous images acquired by visual sensors in a time series. It is good at capturing the changes and correlations of images in the time dimension, so as to better understand information such as the dynamic behavior of objects. The advantage of this combination is that the convolutional neural network can perform a detailed analysis of a single-frame image and provide accurate features of the object; the recurrent neural network can further mine the information of multiple frames of images from the perspective of time series. The two cooperate with each other, making the secondary neural network more comprehensive, accurate and efficient in processing visual image data, and improving the understanding and processing capabilities of the entire system for image information.
[0096] Finally, each image analysis result in the second image set is comprehensively post-processed, including data integration, screening, optimization and other steps, and ultimately outputs detailed and accurate information about the vehicle's surrounding environment, providing strong support for the vehicle's driving decisions.
[0097] On the other hand, Figure 2 As shown, the present invention also provides a visual environment perception system for an intelligent networked vehicle, comprising:
[0098] The data acquisition module uses the vehicle vision sensor to acquire the image data around the vehicle.
[0099] The data preprocessing module preprocesses the image data around the car and segments the preprocessed image data around the car.
[0100] The data processing module identifies each image, marks its corresponding feature attributes, and transmits them to the corresponding neural network based on the feature attributes of the image to perform image analysis operations.
[0101] Data output module outputs information about the car's surroundings.
[0102] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for visual environment perception of an intelligent connected vehicle, characterized in that: Included are: Firstly, the car visual sensor is used to obtain the image data around the car; Preprocessing the image data around the car to obtain a first image, and segmenting the first image to obtain a second image set; Identify each image in the second image set and mark its corresponding feature attribute; Transmitting each image in the second image set to a corresponding neural network according to its characteristic attributes to perform image analysis operations; A plurality of first neural network models are provided, each of which has different processing capabilities and processed data, and each of which processes each image data differently according to a characteristic attribute of each image in the second image set; The multiple first neural network models can process the same image data, the first neural network model processes a single frame of the image data, and the second neural network model processes multiple frames of the image data in a time series; The second neural network model determines whether the multiple frames of images in the time series being processed include the single frame of image processed by the first neural network model. If included, the second neural network model receives the result of the processing completed by the first neural network model during the processing process, and makes corrections during the processing of the multiple frames of images in the time series; After the processing is completed, the processing results are uploaded to the third neural network model, and the third neural network model integrates the analysis results of all the first neural network models; Post-process each image analysis result in the second image set and output the vehicle surrounding environment information.
2. The method for visual environment perception of an intelligent connected vehicle according to claim 1, characterized in that: The specific method of preprocessing the image data around the car to obtain the first image is: First, filter the image data around the car and define a filter window; By using the mean value of the pixel points of the image data around the car in the window, all pixels in the image data around the car are filtered. The specific formula is: Where I(m,n) is the pixel value of the original pixel in the image data around the car in the window, I'(x,y) is the pixel value after denoising, K is the number of pixels in the image data around the car in the window, and S is the coordinate set of the pixel points in the window. By moving the window position, the filtering operation of the entire image data around the car is completed to obtain the first image.
3. The method for visual environment perception of an intelligent connected vehicle according to claim 1, characterized in that: The specific method for segmenting the first image is: In the first image, a plurality of seed pixels (p1, p2, ..., p i ) as the starting point; Determine the standard for judging similarity and set the threshold T to judge the similarity between multiple pixels. The specific formula is: |I(p)-I(p i )|<T; Where I(p) is the gray value of pixel p; Check the seed pixel p i Neighborhood pixels of the region are added to the region. If the neighborhood pixels meet the similarity criteria, they are added to the region and used as new seed pixels. The seed pixels are expanded. When no more neighborhood pixels can be added, the expansion stops. The first image is segmented according to the expansion area of each seed pixel to obtain a set of second images.
4. The method for visual environment perception of an intelligent connected vehicle according to claim 1, characterized in that: The specific method of identifying each image in the second image set and marking its corresponding feature attribute is: Perform grayscale conversion on each image in each second image set, convert all images in the second image set into grayscale images, calculate the gradient value of each image in the x direction and the y direction, and use I x and I y To express; Using the gradient value I of each image in the x direction x And the gradient value I of each image in the y direction y The calculation result tensor M is as follows: Each image in the second image set is converted to grayscale and the following response function is used to detect corner points: R(x,y)=det(M)-k·(trace(M)) 2 ; in, det(M) is the determinant of the matrix M, trace(M) is the trace of the matrix M, k is an empirical constant; By setting the threshold T, pixels whose response value R is greater than T are selected as corner points: R(x,y)>T; In this way, the features of the corner points of each image in the second image set are extracted, and corresponding feature attributes are added to each image according to the features of the corner points of each image.
5. The method for visual environment perception of an intelligent connected vehicle according to claim 1, characterized in that: The first neural network model is a convolutional neural network, which is used to detect objects in a single frame image acquired by a visual sensor; The second neural network model is a recurrent neural network, which is used for data processing of multiple frames of continuous images acquired by a visual sensor in a time series.
6. A visual environment perception system for an intelligent networked vehicle, executing the visual environment perception method for an intelligent networked vehicle according to claim 1, characterized in that: Included are: A data acquisition module uses a vehicle vision sensor to acquire image data around the vehicle; A data preprocessing module, which preprocesses the image data around the car and segments the preprocessed image data around the car; The data processing module identifies each image, marks its corresponding feature attributes, and transmits them to the corresponding neural network based on the feature attributes of the image to perform image analysis operations; Data output module outputs information about the car's surroundings.
Citation Information
Patent Citations
Automobile anti-collision early warning method, system and device based on machine vision and medium
CN118372820A