Visual environment perception method and system for intelligent connected automobile

Abstract

According to the intelligent network connection automobile visual environment perception method and system, surrounding image data are obtained through an automobile visual sensor, and detailed environment information is finally output through preprocessing, image segmentation, feature recognition and neural network analysis. The method comprises the following specific steps: firstly, filtering image data to obtain a clear first image; segmenting the first image through a seed pixel expansion method to form a second image set; performing feature recognition on each image, and marking feature attributes; and finally, the image is transmitted to a neural network for analysis, and environmental information is output after results are integrated. According to the method, the convolutional neural network and the recurrent neural network are adopted to process a single-frame image and a multi-frame image respectively, so that the accuracy and the real-time performance of environment perception are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving and relates to a method and system for visual environment perception of an intelligent networked vehicle. Background Art

[0002] Self-driving cars, also known as car self-driving systems, mainly collect information about the car's surrounding environment through sensors, and then make decisions on the car's actuators after analysis, fusion and processing by the computer system, so as to realize the acceleration, deceleration, left and right steering and braking of the car. The car environment perception system is to continuously collect information about the external environment of the car through the environment perception sensors installed on the car body, identify static and moving objects in the surrounding environment, detect and track the identified objects, and then use the corresponding designed algorithms to determine whether the object is a target object and the degree of threat to the car. At present, the common solution of the intelligent car environment perception system is to produce 3D street view map technology through the 360-degree scanning laser radar on the roof, and realize the intelligent car environment perception with laser radar sensor technology as the core. However, the price of laser radar sensors is expensive, and the preparation and R&D costs are high. In addition, the performance and stability of laser radar are easily affected in severe weather such as fog, heavy rain and blizzard. Summary of the invention

[0003] In order to solve the problems existing in the background technology, the present invention proposes a method and system for visual environment perception of an intelligent connected vehicle.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows.

[0005] A method for visual environment perception of an intelligent connected vehicle, comprising: Firstly, the car visual sensor is used to obtain the image data around the car; Preprocessing the image data around the car to obtain a first image, and segmenting the first image to obtain a second image set; Identify each image in the second image set and mark its corresponding feature attribute; Transmitting each image in the second image set to a corresponding neural network according to its characteristic attributes to perform image analysis operations; Post-process each image analysis result in the second image set and output the vehicle surrounding environment information.

[0006] Furthermore, the specific method of preprocessing the image data around the car to obtain the first image is: First, filter the image data around the car and define a filter window; By using the mean value of the pixel points of the image data around the car in the window, all pixels in the image data around the car are filtered. The specific formula is: ;in is the pixel value of the original pixel in the image data around the car in the window, is the pixel value after denoising, is the number of pixels in the image data around the car in the window, is a set of pixel coordinates in the window. By moving the window position, the filtering operation of the entire image data around the car is completed to obtain the first image.

[0007] Furthermore, the specific method of segmenting the first image is: Select a number of seed pixels in the first image As a starting point; Determine the criteria for judging similarity and set the threshold To judge the similarity between multiple pixels, the specific formula is: ; in Pixel Gray value of Checking the Seed Pixels Neighborhood pixels of the region are added to the region. If the neighborhood pixels meet the similarity criteria, they are added to the region and used as new seed pixels. The seed pixels are expanded. When no more neighborhood pixels can be added, the expansion stops. The first image is segmented according to the expansion area of ​​each seed pixel to obtain a set of second images.

[0008] Furthermore, the specific method of identifying each image in the second image set and marking its corresponding characteristic attribute is: Convert the grayscale value of each image in each second image set, convert all images in the second image set into grayscale images, and calculate the grayscale value of each image in Direction and The gradient values ​​in the direction are respectively expressed as and To express; Using each image Gradient value in direction and each image in Gradient value in direction Compute result tensor , the specific formula is: .

[0009] Detection uses the following response function to detect corner points: ; in, , is a matrix The determinant of , is a matrix traces, is an empirical constant.

[0010] By setting the threshold , select the response value Greater than The pixel points are taken as corner points: ; In this way, the features of the corner points of each image in the second image set are extracted, and corresponding feature attributes are added to each image according to the features of the corner points of each image.

[0011] Furthermore, the specific method of transmitting each image in the second image set to the corresponding neural network according to its characteristic attributes to perform the image analysis operation is: A plurality of first neural network models are provided, each of which has different processing capabilities and processed data, and each of which processes each image data differently according to a characteristic attribute of each image in the second image set; The multiple first neural network models can process the same image data, the first neural network model processes a single frame of the image data, and the second neural network model processes multiple frames of the image data in a time series; The second neural network model determines whether the multiple frames of images in the time series being processed include the single frame of image processed by the first neural network model. If included, the second neural network model receives the result of the processing completed by the first neural network model during the processing process, and makes corrections during the processing of the multiple frames of images in the time series; After the processing is completed, the processing results are uploaded to the third neural network model, and the third neural network model integrates the analysis results of all the first neural network models.

[0012] Further, the first neural network model includes a convolutional neural network and a recurrent neural network; Convolutional neural networks are used to detect objects in single-frame images acquired by visual sensors; Recurrent neural networks are used to process data of multiple frames of continuous images acquired by visual sensors in time series.

[0013] On the other hand, the present invention also provides a visual environment perception system for an intelligent networked vehicle, comprising: A data acquisition module uses a vehicle vision sensor to acquire image data around the vehicle; A data preprocessing module, which preprocesses the image data around the car and segments the preprocessed image data around the car; The data processing module identifies each image, marks its corresponding feature attributes, and transmits them to the corresponding neural network based on the feature attributes of the image to perform image analysis operations; Data output module outputs information about the car's surroundings.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention improves the efficiency of image processing and ensures the accuracy of subsequent analysis by systematically preprocessing and segmenting the acquired image data. At the same time, the present invention also introduces multiple neural network models for division of labor, which can not only perform detailed analysis on single-frame images, but also comprehensively process multi-frame time series data. The second neural network model can receive and correct the results of the first neural network in real time, which improves the accuracy and response speed of data processing and enhances the flexibility and adaptability of the system.

[0015] By accurately identifying surrounding environment information, the system can provide effective support for driving decisions, thereby improving the safety of intelligent connected vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method of the present invention.

[0017] Figure 2 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, a method for visual environment perception of an intelligent connected vehicle includes the following steps: Firstly, the car visual sensor is used to obtain the image data around the car; Preprocessing the image data around the car to obtain a first image, and segmenting the first image to obtain a second image set; Identify each image in the second image set and mark its corresponding feature attribute; Transmitting each image in the second image set to a corresponding neural network according to its characteristic attributes to perform image analysis operations; Post-process each image analysis result in the second image set and output the vehicle surrounding environment information.

[0020] First, the car's visual sensors are used to obtain rich image data around the car in all directions, which is the basis for subsequent processing.

[0021] Subsequently, the collected image data around the car is subjected to fine preprocessing operations, and after algorithm filtering and optimization, a clearer and more accurate first image is obtained.

[0022] Perform filtering operations on the image data around the car and define a filtering window. The size of the filtering window can be customized by the user.

[0023] By using the mean value of the pixel points of the image data around the car in the window, all pixels in the image data around the car are filtered. The specific formula is: ; in is the pixel value of the original pixel in the image data around the car in the window, is the pixel value after denoising, is the number of pixels in the image data around the car in the window, is a set of pixel coordinates in the window. By moving the window position, the filtering operation of the entire image data around the car is completed to obtain the first image.

[0024] The first image after the filtering operation eliminates the noise in the original image, and the characteristic signals or image details in the first image are more prominent.

[0025] Next, the first image is segmented reasonably using advanced image segmentation technology to form a second image set containing information of different regions.

[0026] Select a number of seed pixels in the first image As a starting point; Determine the criteria for judging similarity and set the threshold To judge the similarity between multiple pixels, the specific formula is: ; It is calculated by the gray value of the pixel, where Pixel The gray value of .

[0027] Checking the Seed Pixels The neighboring pixels of the region are added to the region. If the neighboring pixels meet the similarity criteria, they are added to the region and used as new seed pixels. The seed pixels are then expanded. When no more neighboring pixels can be added, the expansion stops.

[0028] The first image is segmented according to the expansion area of ​​each seed pixel to obtain a set of second images.

[0029] The advantage of this image segmentation method is that the steps are clear and definite. From the selection of the starting point to the setting of the similarity judgment standard, to the expansion of the seed pixels and the final image segmentation, each link has a specific operation process, which makes the whole process highly logical and operational, and helps to accurately segment the image and improve the efficiency and accuracy of image analysis.

[0030] Then, each image in the second image set is deeply identified, and its corresponding feature attributes are accurately marked using a professional image recognition algorithm.

[0031] For each image in each second image set, a grayscale value conversion operation needs to be performed. Specifically, all images in the second image set need to be converted into grayscale images through a specific algorithm and processing method. After the grayscale conversion of the image is completed, the grayscale value of each image in the horizontal direction ( direction) and vertical direction ( This calculation process involves analyzing the grayscale value changes of image pixels. By calculating the difference in grayscale values ​​of adjacent pixels in different directions, the gradient value of the image in that direction is determined. This is of great significance for subsequent image processing, feature extraction, and image analysis tasks, and can provide basic data and key information for a deeper understanding and processing of images. For example, in image edge detection, the calculation of gradient values ​​can help accurately locate the edge contours of objects in the image, thereby realizing operations such as recognition and analysis of target objects in the image. The image is in the horizontal direction ( direction) and vertical direction ( The gradient values ​​in the direction are respectively and To express.

[0032] Using each image Gradient value in direction and each image in Gradient value in direction Compute result tensor , the specific formula is: .

[0033] Detection uses the following response function to detect corner points: ; in, , is a matrix The determinant of , is a matrix traces, is an empirical constant.

[0034] By setting the threshold , select the response value Greater than The pixel points are taken as corner points: ; In this way, the features of the corner points of each image in the second image set are extracted, and corresponding feature attributes are added to each image according to the features of the corner points of each image.

[0035] Corner points are mainly detected based on the geometric structure of the image, that is, the change pattern of pixel grayscale. Under a certain degree of lighting change, as long as the geometric structure at the corner point is not destroyed, it can still be detected. For example, in an image of an outdoor scene, the light intensity and angle will change with the time of day, but the corner position and features of a building or object will usually not disappear due to these lighting changes.

[0036] Afterwards, based on these characteristic attributes, each image is accurately transmitted to the corresponding neural network, allowing the neural network to exert its powerful computing and analysis capabilities to perform image analysis operations.

[0037] In this method, multiple secondary neural network models are configured, each of which has a unique processing capability and processes different data. The secondary neural network model includes a first neural network model and a second neural network model.

[0038] Each first neural network model will perform different processing operations on the corresponding image data based on the characteristic attributes of each image in the second image set.

[0039] The multiple first neural network models have the ability to process the same image data. For example, the first neural network model specifically processes a single frame of the image data, focusing on analyzing various types of information in the single frame.

[0040] The second neural network model is responsible for processing the multiple frames of the image data in the time series. In addition, the second neural network model will determine whether the multiple frames of the time series it processes contain the single frame image processed by the first neural network model. If so, during the processing, the second neural network model will receive the result of the processing completed by the first neural network model, and make corrections in the processing of the multiple frames of the time series based on this result.

[0041] When all secondary neural network models have completed processing, they will upload their respective processing results to the third neural network model. The third neural network model plays an integrating role as the main neural network model, and comprehensively and systematically integrates the analysis results of all secondary neural network models.

[0042] The advantages of this architecture are very significant. Multiple secondary neural network models have clear division of labor and carry out targeted processing according to the different characteristics and needs of the image, which improves the professionalism and efficiency of the processing. At the same time, the collaboration and information transmission between different models, such as the reception and use of the results of the first neural network model by the second neural network model, enhances the comprehensive analysis ability of image data and can understand and process image information more comprehensively and accurately. Finally, the integrated processing of the main neural network model enables the results of each secondary model to be unified and coordinated, providing a more complete and reliable basis for subsequent decision-making or further analysis, and improving the performance and intelligence level of the entire system.

[0043] The secondary neural network model covers convolutional neural networks and recurrent neural networks. Among them, the first neural network model in the secondary neural network model is a convolutional neural network, which plays an important role. It is mainly responsible for accurately detecting objects in single-frame images acquired by visual sensors. Through its unique algorithm and structure, it can effectively identify various object features in the image, providing a basis for subsequent analysis. The second neural network model in the secondary neural network model is a recurrent neural network, which focuses on data processing of multiple frames of continuous images acquired by visual sensors in a time series. It is good at capturing the changes and correlations of images in the time dimension, so as to better understand information such as the dynamic behavior of objects. The advantage of this combination is that the convolutional neural network can perform a detailed analysis of a single-frame image and provide accurate features of the object; the recurrent neural network can further mine the information of multiple frames of images from the perspective of time series. The two cooperate with each other, making the secondary neural network more comprehensive, accurate and efficient in processing visual image data, and improving the understanding and processing capabilities of the entire system for image information.

[0044] Finally, each image analysis result in the second image set is comprehensively post-processed, including data integration, screening, optimization and other steps, and ultimately outputs detailed and accurate information about the vehicle's surrounding environment, providing strong support for the vehicle's driving decisions.

[0045] On the other hand, Figure 2 As shown, the present invention also provides a visual environment perception system for an intelligent networked vehicle, comprising: A data acquisition module uses a vehicle vision sensor to acquire image data around the vehicle; A data preprocessing module, which preprocesses the image data around the car and segments the preprocessed image data around the car; The data processing module identifies each image, marks its corresponding feature attributes, and transmits them to the corresponding neural network based on the feature attributes of the image to perform image analysis operations; Data output module outputs information about the car's surroundings.

[0046] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for visual environment perception of intelligent connected vehicles, It is characterized in that The method includes: firstly, using the automobile visual sensor to obtain the image data around the automobile; Preprocess the image data around the car to obtain a first image, segment the first image to obtain a second image set; identify each image in the second image set and mark its corresponding feature attributes; segment each image in the second image set according to its feature attributes, Transmit to the corresponding neural network for image analysis operation; A plurality of first neural network models are provided, each of which has different processing capabilities and processed data, and each of which processes each image data differently according to a characteristic attribute of each image in the second image set; The multiple first neural network models can process the same image data, the first neural network model processes a single frame of the image data, and the second neural network model processes multiple frames of the image data in a time series; The second neural network model determines whether the multiple frames of images in the time series being processed include the single frame of image processed by the first neural network model. If included, the second neural network model receives the result of the processing completed by the first neural network model during the processing process, and makes corrections during the processing of the multiple frames of images in the time series; After the processing is completed, the processing results are uploaded to the third neural network model, and the third neural network model integrates the analysis results of all the first neural network models; Post-process each image analysis result in the second image set and output the vehicle surrounding environment information.

2. According to claim 1, a method for visual environment perception of an intelligent connected vehicle, It is characterized in that The specific method of preprocessing the image data around the car to obtain the first image is: first, filtering the image data around the car, first defining a filtering window; by using the mean of the pixel points of the image data around the car in the window, filtering all pixels in the image data around the car, the specific formula is: ;in is the pixel value of the original pixel in the image data around the car in the window, is the pixel value after denoising, is the number of pixels in the image data around the car in the window, is a set of pixel coordinates in the window. By moving the window position, the filtering operation of the entire image data around the car is completed to obtain the first image.

3. The method for visual environment perception of an intelligent connected vehicle according to claim 1, It is characterized in that The specific method for segmenting the first image is: Select a number of seed pixels in the first image As a starting point; Determine the criteria for judging similarity and set the threshold To judge the similarity between multiple pixels, the specific formula is: ; in Pixel Gray value of Checking the Seed Pixels Neighborhood pixels of the region are added to the region. If the neighborhood pixels meet the similarity criteria, they are added to the region and used as new seed pixels. The seed pixels are expanded. When no more neighborhood pixels can be added, the expansion stops. The first image is segmented according to the expansion area of ​​each seed pixel to obtain a set of second images.

4. The method for visual environment perception of an intelligent connected vehicle according to claim 1, It is characterized in that The specific method of identifying each image in the second image set and marking its corresponding feature attribute is: Convert the grayscale value of each image in each second image set, convert all images in the second image set into grayscale images, and calculate the grayscale value of each image in Direction and The gradient values ​​in the direction are respectively expressed as and To express; Using each image Gradient value in direction and each image in Gradient value in direction Calculation result tensor , the specific formula is: ; Detection uses the following response function to detect corner points: in, , is a matrix The determinant of , is a matrix traces, is an empirical constant; By setting the threshold , select the response value Greater than The pixel points are taken as corner points: ; In this way, the features of the corner points of each image in the second image set are extracted, and corresponding feature attributes are added to each image according to the features of the corner points of each image.

5. The method for visual environment perception of an intelligent connected vehicle according to claim 1, It is characterized in that The first neural network model is a convolutional neural network, which is used to detect objects in a single frame image acquired by a visual sensor; The second neural network model is a recurrent neural network, which is used for data processing of multiple frames of continuous images acquired by a visual sensor in a time series.

6. A visual environment perception system for intelligent connected vehicles, It is characterized in that The method comprises: a data acquisition module, which uses a vehicle visual sensor to acquire image data around the vehicle; a data preprocessing module, which preprocesses the image data around the vehicle and segments the preprocessed image data around the vehicle; The data processing module identifies each image, marks its corresponding feature attributes, and transmits them to the corresponding neural network based on the feature attributes of the image to perform image analysis operations; Data output module outputs information about the car's surroundings.

Images