An efficient road panorama detection method based on model fusion

Abstract

The present invention belongs to the field of image processing technology, and discloses an efficient road panoramic detection method based on model fusion, which obtains image data and performs pre-training; constructs a network model and performs training, and the network model structure includes a feature extraction module, a road segmentation module and a target recognition module; the feature extraction module performs feature extraction on the image to be processed; the road segmentation module searches for edge segments in the input feature map, and connects the segments as boundaries to identify the road surface area; the target recognition module uses a decision tree to detect a variety of vehicle models; uses pruning to fine-tune the network model, and adds L1 and L2 regularization to the scaling factor of the BN layer of the feature extraction module; uses an artificial intelligence camera to measure the distance between vehicles; and detects the picture data taken by the artificial intelligence camera. Compared with the prior art, the present invention can quickly perform road segmentation and vehicle detection, and output clear images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an efficient road panorama detection method based on model fusion. Background Art

[0002] With the development of science and technology and the progress of the times, more and more products are becoming more automated and intelligent. Since the 21st century, the functions of road detection have been continuously improved. Ordinary road detection is usually divided into two parts: road segmentation and target detection.

[0003] Road segmentation technology is one of the keywords in the field of computer vision today, and it paves the way for a complete understanding of real-time scenes. In 2014, Long et al. proposed the FCN network. FCN transforms the fully connected layers in the traditional CNN into convolutional layers. After multiple convolutions (and pooling), the resulting images become smaller and smaller, and the resolution becomes lower and lower. FCN uses upsampling to restore the low-resolution rough images to the resolution of the original image. Compared with the traditional method of using CNN for image segmentation, the advantage of FNC is that it can accept output images of any size without requiring all training images and test images to have the same size. However, the disadvantages of FCN are also very obvious. First, the results obtained are still not fine enough. The results obtained by upsampling are still relatively fuzzy and smooth, and are not sensitive to the details in the image. Secondly, the relationship between pixels is not considered, and there is a lack of spatial consistency.

[0004] Object detection is currently a hot topic in digital image processing. In recent years, object detection methods based on deep convolutional neural networks have greatly improved detection accuracy, but they still face great challenges in practical applications. Based on the latest research trends in the field of object detection, object detection currently faces problems such as a large range of object sizes, real-time detection problems, and uneven samples. Summary of the invention

[0005] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides an efficient road panoramic detection method based on model fusion, which can quickly segment the road surface and detect vehicles, and output clear images.

[0006] Technical solution: The present invention provides an efficient road panorama detection method based on model fusion, comprising the following steps:

[0007] Step 1: Obtain image data and perform pre-training;

[0008] Step 2: construct a network model and perform model training, wherein the network model structure includes a feature extraction module, a road segmentation module, and a target recognition module;

[0009] The feature extraction module extracts features from the processed image, uses CNN to extract image features, and uses the CONV+BN+Relu mode to perform two convolution operations. The first convolution layer uses a convolution layer and a BN layer fusion, and the BN layer is directly connected after the first convolution layer, and the scale conversion and mean shift operation of the BN layer for data are directly applied to the convolution kernel; the second convolution layer adds another convolution layer on the basis of the first convolution operation; and after the convolution, a spatial pyramid is added;

[0010] The road segmentation module searches for edge segments in the input feature map and then connects these segments as boundaries to identify road areas;

[0011] The target recognition module uses a decision tree to detect various types of vehicles appearing on the road;

[0012] Step 3: Use pruning to fine-tune the network model described in step 2, that is, add L1 and L2 regularization to the scaling factor of the BN layer of the feature extraction module, and then perform training and fine-tuning to form the final network model;

[0013] Step 4: Use the AI ​​camera to measure the distance between vehicles;

[0014] Step 5: Use the network model constructed in step 3 to detect the image data taken by the artificial intelligence camera.

[0015] Furthermore, the specific operations of obtaining image data and performing pre-training in step 1 are:

[0016] (1) Collect real-time images of different road surfaces, including images under different conditions such as day, night, and rainstorms;

[0017] (2) Use labelm to label. Labels are divided into two categories: road surface and various vehicles.

[0018] (3) After the annotation is completed, a json file is generated for conversion and mask generation;

[0019] (4) The data will be enhanced after scaling, rotating, and cropping the mask label image and the original image;

[0020] (5) Generate training set, validation set and test set from the enhanced data set.

[0021] Furthermore, the specific operation of the feature extraction module in step 2 is:

[0022] Set each image to 28*28, the first convolution layer has 256 channels, each channel has a 9*9 convolution kernel, and when the BN layer is directly connected after the convolution layer, the convolution operation and the BN layer operation can be merged, that is, the scale transformation and mean shift operation of the BN layer on the data can be directly applied to the convolution kernel, and the activation function RELU is used. The result of the first convolution layer is called Conv1 ReLU;

[0023] After the first convolution layer, another convolution layer is added, which uses a 9*9 convolution kernel to perform convolution operation on Conv1 ReLU;

[0024] After obtaining the image features, a spatial pyramid is used and the idea of ​​maximum pooling is adopted to ensure the consistency of the input features.

[0025] Furthermore, the specific operation of the road segmentation module in step 2 is:

[0026] 1) First find the edge segment:

[0027] 1.1) Use Gaussian filter to smooth the input image:

[0028] Let G(x, y) be a Gaussian function:

[0029] Where, x, y: template coordinates of the pixel, σ is the spatial constant;

[0030] Convolve G and f to get a smooth image: f s (x, y) = G(x, y) * f(x, y)

[0031] Among them, f(x, y) represents the input image, G(x, y) represents the Gaussian function, and the smoothed image f s (x, y);

[0032] 1.2) Calculate the gradient magnitude image and direction image:

[0033] Amplitude:

[0034] in, Represented as a gradient image, g x (x, y), g y (x, y) means calculating the partial derivative for each pixel position to obtain the gradient of an image;

[0035] angle:

[0036] in,

[0037] 1.3) Use non-maximum suppression to refine the wider edges in the image;

[0038] Let d1, d2, d3, and d4 be the four edge directions of the road image area. The direction of the edge d k ;

[0039] Let K denote The value at (x, y) if k is less than d k A point (x, y) in the direction of one or two adjacent points value, then let g N (x, y) = 0 (suppression), otherwise g N (x, y) = k, where is the gradient image, g N (x, y) represents the non-maximum suppressed image;

[0040] 1.4) For g N (x, y) is thresholded;

[0041] 2) Then connect the line segments as boundaries:

[0042] 2.1) Analyze the pixel characteristics in a small area at each point (x, y), connect all the familiar points according to the defined conjecture, and form an edge of pixels with similar properties;

[0043] 2.2) Let S xy represents a set of coordinates of a region in the image with point (x, y) as the center point. If |M(s, t)-M(x, y)|≤F, and F is a positive threshold, then S xy The edge pixel with coordinates (s, t) is similar in magnitude to the pixel at (x, y); if S xy The edge pixel with coordinates (s, t) is similar in angle to the pixel at (x, y); if both the magnitude and direction criteria are met, then S xy The pixel with coordinates (s, t) is connected to the pixel at (x, y), and then this process is repeated to obtain the boundary area;

[0044] 3) Use Hough transform to take straight lines, and after obtaining the boundary area, draw the parameter space straight lines of all points in the road image. The main straight lines in the plane can be found by identifying the points where a large number of straight lines intersect in the space. For the points in the vertical direction, the formula x cosθ+y sinθ=ρ (θ=0 on the horizontal line, ρ represents the positive x intercept) is used to obtain them, and a threshold is set to filter out valid straight lines. Then, the extension lines at both ends of the straight lines are drawn, and their intersection points are obtained. The two-point formula is used to find the intersection points with the edge of the image. Finally, the ROI area mask image is merged with the original image to obtain the final segmentation result.

[0045] Furthermore, the specific operation of the target recognition module in step 2 using a decision tree to detect multiple vehicle types appearing on the road is:

[0046] 1) Import data of all vehicle types on the road and process and obtain their names and category labels;

[0047] 2) Select the most common class as the category label;

[0048] 3) Calculate information entropy, the formula is:

[0049] Where D represents the vehicle dataset, k represents the number of categories, y represents the number of current results, and p k Indicates the proportion of k-type samples, log2 p k represents the entropy of an equiprobable uniform distribution;

[0050] 4) Construct a sub-dataset based on the calculated information entropy;

[0051] 5) Calculate the information gain by subtracting the information entropy after the partition from the information entropy before the partition:

[0052]

[0053] Where v represents the set of possible values, D represents the entire data set, and D v Represents a sample set with a round number value of v, and calculates the information gain of the front height and wheelbase in turn;

[0054] 6) According to the set threshold, if the value of information gain is greater than the set threshold, it is selected as our feature value;

[0055] 7) Generate a decision tree, select the independent variable with the largest information gain as the root node, and select the other eigenvalues ​​as internal nodes in turn.

[0056] Furthermore, the L1 and L2 regularizations in step 3 are respectively:

[0057] The formula for L1 regularization is:

[0058] The formula for L2 regularization is:

[0059] Among them, h θ (x (i) ) represents the predicted value, (y (i) ) represents the true value, Represents the entire loss function, represents regularization, θ jIt can be regarded as a weight parameter, and λ plays a balancing role, where L1 plays a role in sparsity and feature selection, and L2 plays a role in smoothing features.

[0060] Furthermore, the final BN layer in step 3 is transformed as follows:

[0061]

[0062]

[0063] Among them, z in represents the input of the BN layer, μ β and σ β are the mean and standard deviation of the input activations on the BN layer, γ and β represent the scale and displacement, and the γ parameter in the BN layer is used as the required scaling factor.

[0064] Furthermore, in step 4, the distance between vehicles is measured using an artificial intelligence camera. First, the relevant parameters of the camera are debugged; secondly, the camera is used to capture pictures of vehicles on the road, and the captured target information and depth information are packaged into the host side of the device to create a 3D vector from the camera to the object, wherein the target information and depth information include three dimensions x: horizontal coordinate, y: vertical coordinate, z: distance coordinate; then the position information is mapped into 2D space, and only the x and y coordinates need to be considered; finally, the distance is calculated, and the distance between vehicles is calculated using the regular 3D Euclidean distance.

[0065] Furthermore, in step 2, when training the model, the loss value can be generated through forward propagation training, and the values ​​of all parameters can be updated in back propagation, and a transferable learning model can be finally generated through continuous iteration.

[0066] Beneficial effects:

[0067] 1. The present invention adopts a model fusion approach to construct a new network to efficiently implement road panoramic detection, adopts a pruning method to trim the network structure, reduces the file size, and is convenient to embed into a Raspberry Pi device. An artificial intelligence camera is used to measure the distance between vehicles, giving full play to the advantages of artificial intelligence cameras, which can not only make the picture clearer during the day or at night, but also avoid collisions between two vehicles and keep them at a safe distance.

[0068] 2. The present invention adopts two layers of convolution for the feature extraction module. The convolution of the first layer adopts the fusion of the BN layer and the convolution layer. When the BN layer is directly connected after the convolution of the first layer, the convolution operation and the BN layer operation can be merged, and the scale conversion and mean shift operation of the BN layer for the data can be directly applied to the convolution kernel. And the BN layer will be optimized during the subsequent pruning operation. The second layer of convolution solves the problem that the first layer of convolution is not enough to extract suitable features. The present invention needs a larger receptive field to obtain a panoramic view of the road. Therefore, a two-layer convolution method is adopted for feature extraction. After completing the two layers of convolution, spp is added. The features extracted after the convolution are added with spp (equivalent to the maximum pooling operation), and these feature maps are pooled into feature maps of the same size, and then the feature maps of the same size are stacked together to obtain the consistency of the input features.

[0069] 3. The road segmentation of the present invention adopts pspNet. The difficulty of realizing road segmentation lies in the mismatch of contextual relationships. Similar categories are easily confused, and unclear categories (very large objects may exceed the perception of the neural network, and very small objects are difficult to find). Therefore, pspNet is improved on the basis of FCN and introduces more context information to solve it. The way pspNet introduces more context is to use full-value uniform pooling operation to increase the receptive field.

[0070] 4. The purpose of the present invention to detect vehicles by using a decision tree algorithm is that: 1) the decision tree has a simple structure, a small amount of computation, and is very suitable for embedded platforms. 2) the present invention is used for panoramic road detection, and the tree structure visualization can clearly present the types of vehicles on the road. 3) the decision tree is efficient, and the decision tree only needs to be built once and can be used repeatedly. 4) feasible and effective results can be obtained for large data sources in a relatively short time.

[0071] 5. The present invention will eventually run on a Raspberry Pi, but it will consume 700M of space when training CNN, which will be a huge resource burden for embedded devices. The method adopted this time is to add L1 regularization to the scaling factor in the BN layer. This approach is easier to implement. It does not require any changes to the existing CNN framework. Using L1 regularization to push the value of the BN scaling factor to 0 allows us to identify insignificant channels. Then each scaling factor corresponds to a specific convolution channel, and the final network model structure is compact. The size of the model obtained after pruning will be reduced by more than half, and the running time will be greatly reduced.

[0072] 6. The present invention uses the principle of binocular distance measurement to detect the distance between vehicles. The effect achieved is that when vehicles are too close to each other, a reminder will be issued to ensure road safety. The artificial intelligence camera can obtain very clear images regardless of day or night, with a long working distance, precise detection of the road surface, a large viewing angle, and is not prone to missed detection at night. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a flow chart of the road panoramic detection method of the present invention;

[0074] Figure 2 A decision tree for vehicle type identification of the present invention;

[0075] Figure 3 A schematic diagram of measuring vehicle distance according to the present invention;

[0076] Figure 4 This is a structural diagram of the network model of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0078] Reference Figures 1 to 4 The present invention discloses an efficient road panorama detection method based on model fusion, and its specific implementation scheme is as follows:

[0079] 1. Image data collection and pre-training:

[0080] (1) Collect real-life images of different road surfaces, including images under different conditions such as day, night, and rainstorms; (2) Use the labelm labeling tool to label, and the main labels are divided into two categories, road surface and various vehicles; (3) After labeling is completed, generate a json file for conversion and generate a mask (4) After scaling, rotating, and cropping the mask label image and the original image, the data will be enhanced; (5) Generate the training set, validation set, and test set path txt files from the enhanced data set and save them

[0081] 2. Build a network model and train the model

[0082] The entire network framework is as follows Figure 4As shown in the figure, it is mainly divided into three parts. The first part is the feature extraction module, which uses CNN to extract image features and uses the CONV+BN+Relu mode to perform two convolution operations. Each picture is set to 28*28. There are 256 channels in the first convolution layer, and each channel has a 9*9 convolution kernel. When the BN layer is directly connected after the convolution layer, the convolution operation and the BN layer operation can be combined, that is, the scale transformation and mean shift operation of the BN layer on the data can be directly applied to the convolution kernel, and the activation function RELU is used. The purpose is to speed up the convergence speed of the entire model during training. The result of the first layer is called Conv1 ReLU. Since a layer of the network cannot fully extract the appropriate features, a convolution layer is added after the first convolution layer. This layer uses a 9*9 convolution kernel to perform a convolution operation on Conv1 ReLU. After the first part, we got the features of the image, but the size of the feature map of each image is different. SPP (spatial pyramid) is used to effectively avoid the problem of image distortion caused by cropping and scaling of the image area, and solve the problem of neural network extraction of repeated features related to the image. The idea of ​​maximum pooling is used to ensure the consistency of input features.

[0083] The second part is the road segmentation module. The road segmentation algorithm is described in detail as follows:

[0084] A. First find the edge segment:

[0085] 1. Use Gaussian filter to smooth the input image:

[0086] (11) Let G(x, y) be a Gaussian function: Where x, y: template coordinates of pixels, σ: spatial constant;

[0087] (12) Use g and f to convolve to get a smooth image: f s (x, y) = G(x, y) * f(x, y), where f(x, y) represents the input image, G(x, y) represents the Gaussian function, and the smoothed image f s (x, y);

[0088] 2. Calculate the gradient magnitude image and direction image:

[0089] Amplitude:

[0090] in, Represented as a gradient image, g x (x, y), g y (x, y) means calculating the partial derivative for each pixel position to obtain the gradient of an image.

[0091] angle:

[0092] in,

[0093] 3. Use non-maximum suppression to refine the wider edges in the image

[0094] (31) Let d1, d2, d3, and d4 be the four edge directions of the road image area. The direction of the edge d k .

[0095] (32), let K denote The value at (x, y) if k is less than d k A point (x, y) in the direction of one or two adjacent points value, then let g N (x, y) = 0 (suppression), otherwise g N (x, y) = k.

[0096] 4. For g N (x, y) is thresholded to reduce false edge points

[0097] (41)Thresholding can be imagined as creating two additional images:

[0098] g NH (x, y) = g N (x, y) ≥ T H

[0099] g NL (x, y) = g N (x, y) ≥ T L

[0100] Initially NH (x, y) and g NL (x, y) are all set to zero by:

[0101] g NL (x, y) = g NL (x, y)-g NH (x, y)

[0102] From g NL (x, y) removes all NH The non-zero pixels at (x, y) are used to reduce false edge points.

[0103] B. Then connect the line segments as boundaries:

[0104] 1. Analyze the pixel characteristics in a small area at each point (x, y), connect all familiar points according to the defined inference, and form an edge of pixels with similar properties.

[0105] 2. Let S xy Represents a set of coordinates of a region centered at point (x, y) in the image

[0106] If |M(s, t)-M(x, y)|≤F:

[0107] F is a positive threshold, then S xy The edge pixel with coordinates (s, t) is similar in magnitude to the pixel at (x, y).

[0108] if

[0109] S xy The edge pixel with coordinates (s, t) is similar in angle to the pixel at (x, y).

[0110] If both the magnitude and direction criteria are met, then S xy The pixel with coordinates (s, t) is connected to the pixel at (x, y). Then repeat this process to get the boundary area.

[0111] C. Finally, use Hough transform to take the straight line. After obtaining the boundary area, the method usually adopted is to draw the parameter space straight line of all points in the road image. The main straight line in the plane can be found by identifying the points where a large number of straight lines intersect in the space. For the points in the vertical direction, the formula x cosθ+y sinθ=ρ is used to obtain. Set the threshold to filter out the valid straight line and then draw the extension line at both ends of the straight line, and find its intersection point. Use the two-point formula to find the intersection point with the edge of the image. Finally, take the ROI area mask image and merge it with the original image to get the final segmentation result.

[0112] The third part is the target recognition module. In order to fully capture the measurement of driving on the road, this invention adopts the concept of decision tree to detect various types of cars, trucks, buses, etc. on the road. See Table 1 below. Vehicle classification is divided into three categories. The steps of using the decision tree algorithm are as follows:

[0113] 1. Import data of all vehicle types on the road and process and obtain their names and category labels.

[0114] 2. Select the most common class as the category label.

[0115] 3. Calculate information entropy, the formula is:

[0116] Where D represents the data set, k represents the number of categories, y represents the number of current results, and p k Indicates the proportion of k-type samples, log2 p k Represents the entropy of an equiprobable uniform distribution.

[0117] 4. Construct sub-datasets based on the calculated information entropy.

[0118] 5. Calculate the information gain. Information gain is the information entropy before division minus the information entropy after division:

[0119]

[0120] Where v represents the set of possible values, D represents the entire data set, and D v Represents the sample set with round number v. Calculate the information gain of the front height and wheelbase in sequence.

[0121] 6. According to the set threshold, if the value of information gain is greater than the set threshold, it is selected as our feature value.

[0122] 7. Generate a decision tree. Select the independent variable with the largest information gain as the root node. The other feature values ​​are selected as internal nodes in turn.

[0123] Table 1: Vehicle type classification

[0124]

[0125] The above method is used to test various vehicle models. Figure 2 , attached Figure 2 Identify the decision tree for a specific vehicle model.

[0126] After the network model is built, a large amount of sample data and corresponding label data are continuously iterated to train the model's prediction results to minimize the loss function. When training the model, the loss value can be generated through forward propagation training, and the values ​​of all parameters can be updated in the back propagation. Through continuous iteration, a model that can be transferred and learned is finally generated.

[0127] 3. Fine-tune the network model using pruning

[0128] As described above, in the feature extraction module, the convolution method is adopted, and the BN layer is added after the convolution and activation function. In addition, the pruning fine-tuning method is used to add L1 and L2 regularization to the scaling factor of the BN layer. The role of L1 and L2 regularization is not only to prevent the model from overfitting, but also to play a role in sparseness.

[0129] L1 regularization, sparseness and feature selection for weight parameters:

[0130] The formula for L1 regularization is:

[0131] The formula for L2 regularization is:

[0132] Among them, hθ (x (i) ) represents the predicted value, (y (i) ) represents the true value, Represents the entire loss function, represents regularization, θ j It can be regarded as a weight parameter, and λ plays a balancing role, where L1 plays a role in sparsity and feature selection, and L2 plays a role in smoothing features.

[0133] Finally, BN performs the following transformation: Among them, z in Indicates that μ β and σ β are the mean and standard deviation of the input activations on the BN layer, γ and β represent the scale and shift, which provide the possibility to linearly transform the normalized activations back to any scale, using the γ parameter in the BN layer as the required scaling factor.

[0134] The workflow of pruning is to first train the data set and then perform pruning to filter out the important convolutional layers. The model after filtering cannot be used directly and needs to be fine-tuned during training before it can be used in embedded devices.

[0135] 4. Use artificial intelligence camera (OAK) to measure the distance between vehicles:

[0136] 1. First, debug the relevant parameters of the camera

[0137] (1) The distance between two points in the left and right images is measured by the OAK camera, which is called parallax.

[0138] (2) Use parallax to calculate the focal length of the camera (the baseline of OAK is 7.5 and the HFOV is 71.9):

[0139]

[0140] (3) Calculate the depth of the camera

[0141]

[0142] (4) Calculate the minimum stereo depth distance. From the above, we know that the baseline is 7.5 cm. The focal length is 441.25 pixels, and the maximum parallax value is 95 by default. Then:

[0143]

[0144] (5) Similarly, calculate the maximum stereo depth distance

[0145]

[0146] And set the camera parameters according to the above calculation results.

[0147] 2. Secondly, use the camera to capture the image of the vehicle on the road, and pack the captured target information and depth information (including three dimensions x: horizontal coordinate, y: vertical coordinate, z: distance coordinate) into the host side of the device to create a 3D vector from the camera to the object. Then map the position information into 2D space, only considering the x and y coordinates.

[0148] like Figure 3 As shown, the distance is finally calculated, and the distance between vehicles is calculated by the regular 3D Euclidean distance. The formula is as follows:

[0149]

[0150] 3. Finally, if the distance between any two cars is less than the threshold we specify, it means that both cars are considered dangerous and approaching each other, and a warning is issued.

[0151] 5. Convert the constructed network model into a blob file format, turn on the OAK artificial intelligence camera, build the Raspberry Pi environment, and install the Raspberry Pi image that OAK depends on. After the burning is complete, insert the Raspberry Pi to start, download the Raspberry Pi software, select the Raspberry Pi model and the memory card to be burned, and then start burning. Convert and input commands to start.

[0152] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient road panorama detection method based on model fusion, characterized in that: The steps include: Step 1: Obtain image data and preprocess it; Step 2: construct a network model and perform model training, wherein the network model structure includes a feature extraction module, a road segmentation module, and a target recognition module; The feature extraction module extracts features from the processed image, uses CNN to extract image features, and performs two convolution operations using the CONV+BN+Relu mode. The first convolution layer uses a fusion of a convolution layer and a BN layer. The BN layer is directly connected after the first convolution layer, and the scale conversion and mean shift operations of the BN layer on the data are directly applied to the convolution kernel; The second convolution layer adds another convolution layer based on the first convolution operation; and after the convolution, a spatial pyramid is added; The road segmentation module searches for edge segments in the input feature map and then connects these segments as boundaries to identify road surface areas; The target recognition module uses a decision tree to detect various types of vehicles appearing on the road; Step 3: Use pruning to fine-tune the network model described in step 2, that is, add L1 and L2 regularization to the scaling factor of the BN layer of the feature extraction module, and then perform training and fine-tuning to form the final network model; Step 4: Use the AI ​​camera to measure the distance between vehicles; Step 5: Use the network model constructed in step 3 to detect the image data taken by the artificial intelligence camera.

2. The efficient road panorama detection method based on model fusion according to claim 1, characterized in that: The specific operations of obtaining image data and performing preprocessing in step 1 are: (1) Collect real-time images of different road surfaces, including images under different conditions such as day, night, and rainstorms; (2) Use labelm to label. Labels are divided into two categories: road surface and various vehicles. (3) After the annotation is completed, a json file is generated for conversion and mask generation; (4) The data will be enhanced after scaling, rotating, and cropping the mask label image and the original image; (5) Generate training set, validation set and test set from the enhanced data set.

3. The efficient road panorama detection method based on model fusion according to claim 1, characterized in that: The specific operation of the feature extraction module in step 2 is: Each image is set to 28*28, the first convolution layer has 256 channels, each channel has a 9*9 convolution kernel, and when the BN layer is directly connected after the convolution layer, the convolution operation and the BN layer operation are merged, that is, the scale transformation and mean shift operation of the BN layer on the data are directly applied to the convolution kernel, and the activation function RELU is used. The result of the first convolution layer is called Conv1 ReLU; After the first convolution layer, another convolution layer is added, which uses a 9*9 convolution kernel to perform convolution operation on Conv1 ReLU; After obtaining the image features, a spatial pyramid is used and the idea of ​​maximum pooling is adopted to ensure the consistency of the input features.

4. The efficient road panorama detection method based on model fusion according to claim 1, characterized in that: The specific operation of the road segmentation module in step 2 is: 1) First find the edge segment: 1.1) Use Gaussian filter to smooth the input feature map: Let G(x,y) be a Gaussian function: Where x, y: template coordinates of the pixel, σ is the spatial constant; Convolve G and f to get a smooth image: f s (x,y)=G(x,y)*f(x,y) Among them, f(x,y) represents the input feature map, G(x,y) represents the Gaussian function, and the smoothed image f s (x,y); 1.2) Calculate the gradient magnitude image and direction image: Amplitude: in, Represented as a gradient image, g x (x,y),g y (x, y) means calculating the partial derivative for each pixel position to obtain the gradient of an image; gradient: in, 1.3) Use non-maximum suppression to refine the wider edges in the image; Let d1, d2, d3, and d4 be the four edge directions of the road image area. The direction of the edge d k ; Let K denote The value at (x,y) if k is less than d k A point (x, y) in the direction of one or two adjacent points value, then let g N (x,y)=0, otherwise g N (x,y)=k, where is the gradient image, g N (x,y) represents the non-maximum suppressed image; 1.4) For g N (x,y) is thresholded; 2) Then connect the line segments as boundaries: 2.1) Analyze the pixel characteristics in a small neighborhood of each point (x, y), connect all similar points according to the defined conjecture, and form an edge of pixels with similar properties; 2.2) Let S xy represents a set of coordinates of a region in the image with point (x, y) as the center point. If |M(s, t)-M(x, y)|≤F, and F is a positive threshold, then S xy The edge pixel with coordinates (s, t) is similar in magnitude to the pixel at (x, y); if S xy The edge pixel with coordinates (s, t) is similar in angle to the pixel at (x, y); if both the magnitude and direction criteria are met, then S xy The pixel with coordinates (s, t) is connected to the pixel at (x, y), and then this process is repeated to obtain the boundary area; 3) Use Hough transform to take the straight line, and after obtaining the boundary area, draw the parameter space straight line of all the points in the road image, and the straight line in the plane is found by marking the points where the straight lines intersect in the space. For the points in the vertical direction, the formula xcosθ+ysinθ=ρ is used to obtain them. On the horizontal line, θ=0, ρ represents the positive x-intercept, and the threshold is set to filter out the valid straight line. Then draw the extension lines at both ends of the straight line and find their intersection. Use the two-point formula to find the intersection with the edge of the image, and finally take the ROI area mask image and merge it with the original image to get the final segmentation result.

5. The efficient road panorama detection method based on model fusion according to claim 1, characterized in that: The specific operation of the target recognition module in step 2 using a decision tree to detect multiple vehicle types on the road is: 1) Import data of all vehicle types on the road and process and obtain their names and category labels; 2) Select the most common class as the category label; 3) Calculate information entropy, the formula is: Where D represents the vehicle dataset, k represents the kth category, y represents the number of current results, and p k Indicates the proportion of k-type samples, log2p k represents the entropy of an equiprobable uniform distribution; 4) Construct a sub-dataset based on the calculated information entropy; 5) Calculate the information gain by subtracting the information entropy after the partition from the information entropy before the partition: Where v represents the set of possible values, D represents the entire data set, and D v Represents a sample set with a round number value of v, and calculates the information gain of the front height and wheelbase in turn; 6) According to the set threshold, if the value of information gain is greater than the set threshold, it is selected as our feature value; 7) Generate a decision tree, select the eigenvalue with the largest information gain as the root node, and select the other eigenvalues ​​as internal nodes in turn.

6. The efficient road panorama detection method based on model fusion according to claim 1, characterized in that: The L1 and L2 regularizations in step 3 are respectively: The formula for L1 regularization is: The formula for L2 regularization is: Among them, h θ (x (i) ) represents the predicted value, (y (i) ) represents the true value, Represents the entire loss function, represents regularization, θ j is the weight parameter, and λ plays a balancing role. The role of L1 is sparse and feature selection, and the role of L2 is smoothing features.

7. The efficient road panorama detection method based on model fusion according to claim 6 is characterized in that: The final BN layer in step 3 is transformed as follows: Among them, z in represents the input of the BN layer, μ β and σ β are the mean and standard deviation of the input activations on the BN layer, γ and β represent the scale and displacement, and the γ parameter in the BN layer is used as the required scaling factor.

8. The efficient road panorama detection method based on model fusion according to claim 1, characterized in that: In step 4, the distance between vehicles is measured using an artificial intelligence camera. First, the relevant parameters of the camera are debugged; secondly, the camera is used to capture images of vehicles on the road, and the captured target information and depth information are packaged into the host side of the device to create a 3D vector from the camera to the object, wherein the target information and depth information include three dimensions: x: horizontal coordinate, y: vertical coordinate, and z: distance coordinate; then the position information is mapped into 2D space, and only the x and y coordinates need to be considered; finally, the distance is calculated, and the distance between vehicles is calculated using the regular 3D Euclidean distance.

9. The efficient road panorama detection method based on model fusion according to claim 1, characterized in that: In step 2, when training the model, the loss value is generated by forward propagation training, and the values ​​of all parameters are updated in the back propagation. Through continuous iteration, a transferable learning model is finally generated.

Images