Solution of the Migration-based Target Detection Network Search Model to the Problem of Intelligent Vehicle Marker Detection

Abstract

The solution to the smart car landmark detection problem based on the migration target detection network search model is to implicitly associate the classification problem with the target detection problem through model migration, so that a high-performance target detection network can be obtained by combining the search time of the classification problem with a small amount of target detection network training time, which greatly reduces the resource consumption of the search and provides a high-performance customized network for the smart car landmark detection problem.

Description

Technical Field

[0001] This paper involves the problem of smart car landmark detection in the field of intelligent driving, and proposes an automated network search model to solve the problem. Background Art

[0002] In recent years, with the development of artificial intelligence, the concept of intelligent driving has been proposed and has become one of the hot topics of current research. By utilizing technologies such as artificial intelligence, computer vision (Computer Vision), and convolutional neural networks (CNNs), vehicles can partially perceive their surroundings and perform autonomous or assisted driving, thus compensating for the shortcomings of human drivers. Among them, object detection, as a classic problem in computer vision, is widely used in intelligent driving systems as the basis for image analysis. The object detection task is to extract features from images or videos for analysis and mark "sensitive" areas based on specific network structures or detection tools. For example, in street driving scenarios, specialized object detection algorithms can detect the classification and location of vehicles, pedestrians, or other obstacles, thereby providing visual information for autonomous driving.

[0003] For real-world scenarios like intelligent driving, models must strike a balance between accuracy, inference speed, and parameter size. Most existing models struggle to achieve these trade-offs, making it difficult to meet these demands. While performance can be improved to some extent through optimization methods like quantization, pruning algorithms, and knowledge distillation, the ultimate optimization results still depend on the network's baseline. Therefore, even with the right optimization method, model performance is still affected by the network structure itself, which undoubtedly poses a significant challenge to manually constructing neural networks.

[0004] To address these issues, a currently effective approach is to design customized neural network models. One of the most effective methods is Neural Architecture Search (NAS). NAS aims to automatically update neural network structures through specific search strategies using AutoML, achieving requirements that are difficult to achieve with manual network construction.

[0005] However, the current mainstream neural network architecture search models, regardless of the search strategy they are based on, cannot quickly provide good technical support for the data collected by smart cars. This is because the target detection models deployed on smart cars have high requirements for many factors such as parameter size and inference speed, while conventional search models often have low efficiency and high consumption, and cannot fully utilize the advantages of neural network architecture search.

[0006] Therefore, taking this as the starting point, combining the idea of model migration, and using neural network architecture search as a tool, we can propose a network structure with excellent performance for the smart car landmark detection problem in a short period of time, and also provide a feasible solution for the practical application of neural network architecture search. Summary of the Invention

[0007] Given the aforementioned shortcomings of manually designed customized networks, a neural network architecture search model was selected to provide customized networks for the smart car landmark detection problem. Therefore, to achieve the purpose of the invention, a migration-based target detection network search model was proposed for the smart car landmark detection problem. This model implicitly links the classification problem with the target detection problem through model migration. This model only requires the search time for the classification problem combined with a small amount of personalized target detection network design to obtain a high-performance target detection network, significantly reducing resource consumption and proposing an effective solution for the development of customized networks for the smart car landmark detection problem.

[0008] Take the following approach:

[0009] The approach consists of three steps. The first step prioritizes pre-training on a classification dataset, using a traffic scene dataset and an efficient search model to identify modules sensitive to intelligent driving. Second, after obtaining these high-performance modules, they are stacked to form a deep neural network for feature extraction from images. Third, this backbone network is combined with a feature pyramid module and fine-tuned using an object detection loss function to achieve the goal of solving the intelligent vehicle landmark detection problem. The following sections describe each of these steps.

[0010] (1) Pre-training search phase

[0011] To conduct customized network search based on model transfer, the first step is to solve the classification problem. A traffic scene landmark classification dataset was used to search the backbone network. Furthermore, the search strategy abandoned the previous work's supernet combined with reinforcement learning training method. This method requires the entire network to be stored in memory, combined with the corresponding loss function and pruning, and searched in a pathfinding-like manner. This is not only demanding on device resources, but also takes too long to complete a search, which does not reflect the advantages of the search model. Therefore, a proxy is used to search in combination with a gradient strategy. The proxy mode refers to using a small structure to represent the entire network. In neural network architecture search, the proxy mode can abstract the search of the entire network into a hierarchical search. Using this method, it is proposed to piece together a micro-network from modules to represent the large network for search, with the intention of searching for high-performance modules on the intelligent driving classification dataset, and then stacking them to form a deep neural network.

[0012] Considering that multi-scale features can more effectively extract image information, we designed and searched for two types of modules: "normal modules" and "downsampling modules" to build the final architecture. The "normal modules" maintain the same input and output feature maps, while the "downsampling modules" downsample the feature maps with a stride of 2, halving their dimensions. This allows for better extraction of deeper information from the image.

[0013] In the design of the module, each module is represented as a directed acyclic graph (DAG) consisting of K nodes arranged in sequence. K is set to 7. This variable affects the search speed of the model. When set to 7, it can effectively ensure the balance between accuracy and speed. Each node corresponds to a feature matrix. The edge from node i to node j is represented as e i,j , the input of this edge is feature x i , the output is recorded as O i,j (x i ), node j receives all inputs from all input edges, and forms the feature matrix x of node j after feature concatenation j , as shown in Formula 1.

[0014]

[0015] Let O be the search space of the model, representing the edge e i,j The set of all candidate operations, For each set of operations O i,j , set a set of coefficients So in fact during training, we use a mixture of operations in the search space The selection of operations can be abstracted as a softmax mixture in the search space, and the selection of each edge is shown in Formula 2:

[0016]

[0017] In the above formula 2, That is, according to each operating coefficient The softmax weight coefficient, o(x) represents the operation in the search space The search weight of , so Formula 2 is the softmax mixture of each operation in the search space. So far, the input feature matrix x can be realized i -Operation O i,j -Output feature matrix x j The connected path of each node is combined through the above path for feature extraction. At the same time, the shared weight search mode is used in the network optimization process, that is, the model weight update and architecture optimization are performed simultaneously, by alternately training the architecture parameter α and the network weight ω *Initialize α to a K*(K-3)*8-dimensional matrix with a value of 0.125, where K-3 is to remove two input nodes and one output node, and 8 is the number of search space operation types. Use LR logistic regression to minimize the loss term and To find the optimal module structure, LR logistic regression can effectively characterize the difference between the training set data and the prediction, so LR can be used to optimize the network parameters ω * .

[0018] Take 6 modules for search, 6 modules can effectively maintain the balance between search speed and accuracy. Then, the 6 modules are spliced with a fully connected layer for classification, thus forming a small-scale neural network. Set the training cycle Ep = 40, and When both are less than 1 or the architectural parameters of each edge When the softmax results of are all greater than 0.3, it can be considered that the model search phase has converged successfully. At this time, the optimized ω is the trained network weight. The optimization formula is shown in Formula 3. In Formula 3, α represents the architecture parameter, which is also in The collection of ω * represents the weight, ω * (α) represents the weight value under the architecture parameter α, and Represents the validation set loss term and the training set loss term, which are obtained by LR logistic regression with ω and α as input. The two parameters are optimized simultaneously in the search phase, with the purpose of reducing and The two loss terms enable us to find the optimal structure α and obtain the weight ω of the optimal performance at the same time. * (α):

[0019]

[0020] (2) Module stacking

[0021] After obtaining high-performance modules in step (1), the modules are stacked to form a deep backbone network for feature extraction. We select a total of N modules for stacking, of which M are downsampling modules and the rest are normal modules. The downsampling modules are inserted into the 1 / (M+1) position of the deep network to realize the extraction of multi-scale features. In the experiment, N=20 and M=2. The value of N is negatively correlated with the inference speed. The higher the value of M, the stronger the multi-dimensional information is, which is positively correlated with the accuracy. The combination of N=20 and M=2 in the task can achieve the desired effect.

[0022] After forming the backbone network, the fully connected layer and the classifier are concatenated (deleted during model migration), and the classification weights are trained using the landmark dataset from step (1). This is to make the concatenated backbone network sensitive to the characteristics of intelligent driving landmarks. This method is faster than first migrating to the target detection network and then training. After training, the weights are saved in dictionary format to facilitate weight extraction in subsequent steps.

[0023] (3) Model migration

[0024] In step (2), the deep backbone network and pre-trained weights for feature extraction are obtained and used for feature extraction. The extracted features are passed as input to the feature pyramid and the one-stage detector to process the detection task. A total of three feature maps are designed. The first feature map is 32 times the original image downsampled, the second feature map is 16 times the original image downsampled, and the third feature map is 8 times the original image downsampled. After the input image passes through the network in (2) to form a feature matrix, it is divided into two uses. The first use is formed after several 3*3 convolutions and 1*1 convolutions to form a feature matrix. Figure 1 , the second one is subjected to 1*1 convolution and upsampling operation, and concat with the intermediate layer result of (2) network to form feature Figure 2 The same operation is repeated to generate features Figure 3 , so far the multi-dimensional feature matrix used to represent image features has been obtained.

[0025] In the above operations, concat refers to directly concatenating multiple feature maps according to the channel dimension. For example, concatenating an 8*8*16 feature map with an 8*8*16 feature map generates an 8*8*32 feature map. Upsampling refers to using the nearest neighbor interpolation algorithm to generate a larger image from a smaller feature matrix without affecting the number of channels in the feature map.

[0026] After obtaining the multi-dimensional features, the Yolo one-stage detector is used for feature processing. In terms of the detector, the input is multi-dimensional features, and the output is a feature matrix with (B x(5+C)) entries, where B represents the number of bounding boxes that each unit can predict. Each bounding box has 5+C attributes, which respectively describe the center coordinates, size, objectness score, and confidence of C classes of each bounding box. Three bounding boxes are predicted for each unit, that is, B=3, and there are 7 classification categories, that is, C=7. The larger B is, the better the prediction effect, but it will affect the network training speed, so it is generally not more than 6. B=3 can complete effective candidate box extraction, and C is the total category of objects to be detected. At the same time, through the public

[0027] Formula 4 is used to perform bounding box prediction preprocessing.

[0028]

[0029] Among them, b x The predicted center coordinate x-axis coordinate, b y is the vertical coordinate, b w is the prediction box width, b h is the predicted box height. x is the x-axis coordinate after preprocessing, t y is the vertical coordinate after preprocessing, t w is the processed width, t h is the height after processing. c x is the x coordinate of the upper left corner of the grid, c y is the y coordinate. w is the width of the bounding box predicted by the network, p h is the height, σ is the sigmoid function, which forces the output value to be compressed between 0 and 1. Because if the output value is greater than 1, the predicted border may exceed the current border, which is not in line with the original design intention. For example, when the predicted center offset is (0.4, 0.8) at the grid (1, 1), the true representative coordinates are (1.4, 1.8), but if the offset is greater than 1, assuming it is (0.4, 1.2), then the true representative coordinates are (1.4, 2.2), which exceeds the (1, 1) grid and violates the original design intention.

[0030] Therefore, according to the above design, if the input image is 416*416, the three designed feature maps will generate a total of ((52×52)+(26×26)+13×13))×3=10647 bounding boxes. Next, we need to select the optimal solution from these bounding boxes. Non-maximum suppression is used for this selection. That is, the scores of all boxes are traversed, the highest score and corresponding bounding box in each category are selected, and the bounding boxes with a higher than a certain intersection-over-union ratio with the selected box are deleted. Because the threshold is exceeded, the objects in the two boxes can be considered to belong to the same category. The object detection task only requires retaining the box with the highest probability of one category. The intersection-over-union ratio th=0.5 is used as the threshold. th>0.5 is often used for the detection and screening of small targets and does not involve the detection of smaller objects. Therefore, th=0.5 is directly used for the detection box screening.

[0031] Finally, use Formula 5 to fine-tune the network weights. Since we have obtained the pre-trained weights of the backbone network in step (2), fine-tuning after model transfer only requires a few cycles to reach convergence.

[0032]

[0033] Where t is the label, which only contains 0 and 1, o is the input, which contains a decimal between 0 and 1, n represents the total number of images, and i represents the i-th image. Then t[i] represents whether the current input item matches the label of the i-th image, corresponding to 0 for no match and 1 for match, respectively. o[i] represents the probability that the data input to the detector after the neural network prediction matches the label of the i-th image. The two have the same dimension, which represents the information entropy of the image. In each training cycle, o[i] changes accordingly to minimize the loss. When the loss is less than 1, it can be considered to have converged.

[0034] A migration-based target detection network search model is proposed, which implicitly associates classification and target detection tasks, searches for customized networks with higher efficiency, achieves better results than existing methods, and provides a high-performance target detection network for smart car detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a flow chart of the invention method.

[0036] Figure 2 Schematic diagram of the search phase method.

[0037] Figure 3 Schematic diagram of the network model structure finally formed by the stacked search module.

[0038] Figure 4 Schematic diagram of the model migration stage.

[0039] Figure 5 Schematic diagram of the target detection network structure.

[0040] Figure 6 Collect test result images for smart cars. DETAILED DESCRIPTION

[0041] The following further describes the invention with reference to the accompanying drawings.

[0042] Figure 1 This is the specific implementation process of the proposed migration-based target detection network search model for the solution to the smart car landmark detection problem, including:

[0043] Step 1: Classification task search phase, such as Figure 2As shown in Figure 2, the designed module consists of p = 7 different feature nodes. The input of a node is the output of the previous two nodes and the operation searched in the search space. There are 8 types of operations in the search space, namely 3*3 convolution, 5*5 convolution, 3*3 separable convolution, 5*5 separable convolution, skip connection, no connection, average pooling, and maximum pooling. The selection method is shown in Formula 2, where O is the search space of the model. The selection of operations can be abstracted as the scoring result of the softmax result in the search space. At the same time, two different search modules are considered in the search phase: normal modules and downsampling modules. The dimensions of the input and output of the normal module are consistent. The output of the downsampling module is downsampled with a stride of 2 relative to the input, that is, the dimension of the feature map is halved. All convolutions are preceded by a ReLu operation, followed by batch normalization, and padding is added to preserve the spatial size of the feature map.

[0044] During the search phase, all operations were assigned a weight of 0.125. The same seven initial modules were used for the proxy search: the first three were normal modules, the fourth was a downsampling module, and the last three were normal modules. All normal modules were updated identically during the search process, and the normal and downsampling modules were updated synchronously. The weights of all operations were optimized using a gradient strategy with a search period of E = 40. Upon completion of the search, the structural parameters of the current normal and downsampling modules were saved for subsequent use.

[0045] Step 2: Deep network splicing stage. After completing the search stage, two types of high-performance modules are obtained and stacked in a specific way: a total of f = 20 modules are selected for stacking, of which 18 are normal modules and 2 are downsampling modules. The downsampling module is inserted at one-third of the network, and the rest are normal modules. Therefore, the final backbone network is 6*normal modules + downsampling modules + 6*normal modules + downsampling modules + 6*normal modules, as shown in the figure. Figure 3 shown.

[0046] After stacking is completed, the formed backbone network is temporarily spliced with the fully connected layer and the classifier, and trained using the Chinese Traffic Sign Classification Dataset (CTSRD). CTSRD contains more than 6,000 images, which are enhanced and expanded to more than 15,000 images through horizontal flipping, angle flipping, saturation adjustment, etc. for weight pre-training. When the training is completed, the weights are saved as a dictionary file to facilitate subsequent migration and extraction.

[0047] Step 3: Model migration phase, such as Figure 3 As shown in the figure, the backbone network in step 2 (excluding the fully connected network and classifier) is migrated, and the feature pyramid module and the Yolo one-stage detector are combined to perform the target detection task.

[0048] First, in this step, we used the smart car to collect data. Using the smart handle, we controlled the car to capture landmark image data. After acquisition, we further optimized the image quality by adjusting brightness and saturation, as real-world lighting can affect image quality. We also performed image enhancement, ultimately compiling a total of approximately 10,000 images for detection. We then used the labelimg tool to annotate each image and split it into a 4:4:2 ratio for training, validation, and test sets. This completed the object detection dataset.

[0049] Secondly, the backbone network in step 2 is spliced with the feature pyramid structure, and the intermediate output of the backbone network is combined to form three feature maps. The resulting network diagram is as follows: Figure 5 As shown. When the image is read into the network, the image size is reset to 416*416, so the dimensions of the three feature maps generated are 13*13, 26*26, and 52*52. According to the method described in the content of the invention, 10647 bounding boxes will be generated, and the bounding box with the highest confidence will be selected using the non-maximum suppression method. During the model training phase, Formula 5 is used in combination with the Adam optimizer for optimization. At the same time, the weights of the pre-training search phase are synchronously migrated to the target detection network, which saves about 10 hours compared to training from scratch. Finally, the final structure and weight model is obtained after less than 12 hours of training.

[0050] The number of module feature nodes p and the number of stacked layers f can be used to adjust the size of network model parameters, thereby balancing inference speed and accuracy. When p = 7 and f = 20 are set, target detection tests on smart car data collected on a 1080Ti GPU achieve an average accuracy of 81.2% and an inference speed of 10ms, which is better than other existing models. The detection results collected by smart cars are as follows: Figure 6 shown.

[0051] It should be understood that although this specification is described based on implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation method can also be appropriately combined and implemented according to the understanding of those skilled in the art.

[0052] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods, and they are not used to limit the scope of protection. Any equivalent implementation methods or changes that do not deviate from the spirit of the invention should be included in the scope of protection.

Claims

1. A method for intelligent vehicle landmark detection problem based on a migration-based target detection network search model, characterized in that: It includes the following three steps: Step (1) Pre-training search phase Design and search for two types of modules, namely "normal modules" and "downsampling modules", to build the final architecture. The "normal modules" have the same input and output feature maps, while the "downsampling modules" downsample the feature maps with a stride of 2, halving the feature map dimensions. In the design of the module, each module is represented as a directed acyclic graph consisting of K nodes arranged in sequence, where K is set to 7, and each node corresponds to a feature matrix; the edge from node i to node j is represented as e i,j , the input of this edge is feature x i , the output is recorded as O i,j (x i ), node j receives all inputs from all input edges, and forms the feature matrix x of node j after feature concatenation j , as shown in formula 1; x j =∑ i＜j O i,j (x i ) (1) Let O be the search space of the model, representing the edge e i,j The set of all candidate operations, For each set of operations O i,j , set a set of coefficients So in fact, during the training process, a hybrid operation in the search space is used. The selection of the operation is abstracted as a softmax mixture in the search space, and the selection of each side is shown in Formula 2: In the above formula 2, That is, according to each operating coefficient The softmax weight coefficient, o(x) represents the operation in the search space The search weight of , so Formula 2 is the softmax mixture of each operation in the search space; so far, the input feature matrix x i Operation O i,j Output feature matrix x j The connected path of each node is combined through the above path for feature extraction; at the same time, the shared weight search mode is used in the network optimization process, that is, the model weight update and architecture optimization are performed simultaneously, by alternately training the architecture parameter α and the network weight ω * Initialize α to a K*(K-3)*8-dimensional matrix with a value of 0.125, where K-3 is to remove two input nodes and one output node, and 8 is the number of search space operation types; use LR logistic regression to minimize the loss term and To find the optimal module structure, LR logistic regression can effectively characterize the difference between the training set data and the prediction, so LR is used to optimize the network parameters ω * , Take 6 modules for search and connect them to a fully connected layer for classification, thus forming a small-scale neural network. Set the training cycle Ep=40. and When both are less than 1 or the architectural parameters on each side When the softmax results of all are greater than 0.3, the model search phase is considered to have converged successfully, and the optimized ω is the trained network weight; The optimization formula is shown in Formula 3. In Formula 3, α represents the architecture parameter, which is also in The collection of ω * represents the weight, ω * (α) represents the weight value under the architecture parameter α, and Represent the validation set loss term and the training set loss term respectively, which are obtained by using LR logistic regression with ω and α as input; the two parameters are optimized simultaneously in the search phase, with the purpose of reducing and The two loss terms enable us to find the optimal structure α and obtain the weight ω of the optimal performance at the same time. * (α): Step (2) Module stacking After obtaining the modules in step (1), the modules are stacked to form a deep backbone network for feature extraction; N modules are selected for stacking, of which M are downsampling modules and the rest are normal modules. The downsampling modules are inserted into the 1 / (M+1) position of the deep backbone network to realize multi-scale feature extraction, where N=20 and M=2; After forming the backbone network, the fully connected layer and the classifier are spliced together, and the classification weights are trained using the marker dataset in step (1); Step (3) Model migration In step (2), a deep backbone network and pre-trained weights for feature extraction are obtained, which are used for feature extraction, and the extracted features are passed as input to the feature pyramid and the one-stage detector for processing the detection task; a total of three feature maps are designed, the first feature map is 32 times the original image downsampled, the second feature map is 16 times the original image downsampled, and the third feature map is 8 times the original image downsampled; the input image is divided into two uses after the feature matrix is formed by the deep backbone network obtained in step (2), the first use is formed after several 3*3 convolutions and 1*1 convolutions to form feature map 1, the second use is formed after 1*1 convolution and upsampling operations, and concat with the intermediate layer result of the deep backbone network to form feature map 2; the same operation is repeated to generate feature map 3, and the multi-dimensional feature matrix used to represent the image features has been obtained; In the above operation, concat refers to directly splicing multiple sets of feature maps according to the channel dimension; After obtaining the multi-dimensional features, the Yolo one-stage detector is used for feature processing. In terms of the detector, the input is the multi-dimensional features, and the output is a feature matrix with (B*(5+C)), where B represents the number of bounding boxes that each unit can predict. Each bounding box has 5+C attributes, which respectively describe the center coordinates, size, object score, and confidence of C classes of each bounding box. Three bounding boxes are predicted for each unit, that is, B=3. There are 7 classification categories, that is, C=7. B=3 can complete effective candidate box extraction. C is the total category of objects to be detected. Use formula 4 to perform bounding box prediction preprocessing; b x =σ(t x )+c x b y =σ(t y )+c y Among them, b x The predicted center coordinate x-axis coordinate, b y is the vertical coordinate, b w is the prediction box width, b h is the prediction box height; t x is the x-axis coordinate after preprocessing, t y is the vertical coordinate after preprocessing, t w is the processed width, t h is the height after processing; c x is the x coordinate of the upper left corner of the grid, c y is the y coordinate; p w is the width of the bounding box predicted by the network, p h is the height, σ is the sigmoid function, which forces the output value to be compressed between 0 and 1, because if the output value is greater than 1, the predicted border will exceed the current border, which is not in line with the original design intention. Select the optimal solution in the bounding box; use non-maximum suppression to select it, that is, traverse the scores of all boxes, select the highest score and corresponding bounding box in each category, and delete the bounding boxes with a higher intersection-with-union ratio than a certain value with the selected box. Because it exceeds the set threshold, the intersection-with-union ratio th=0.5 is used as the threshold. th>0.5 is often used for detection and screening of small targets and does not involve detection of smaller objects, so th=0.5 is directly used for screening detection boxes; Finally, use Formula 5 to fine-tune the network weights; Where t is the label, which only contains 0 and 1, o is the input, which contains a decimal between 0 and 1, n represents the total number of images, and i represents the i-th image. Then t[i] represents whether the current input item matches the label of the i-th image, corresponding to 0 for no match and 1 for match, respectively. o[i] represents the probability that the data input to the detector after the neural network prediction matches the label of the i-th image. The two have the same dimension, which represents the information entropy of the image. In each training cycle, o[i] changes accordingly to minimize the loss. When the loss is less than 1, it is considered to have converged.

Images