Driving scene graph building method based on neural network architecture search
Through the search method of neural network architecture based on TreeNAS, the feature extraction network and TreeNAS-YOLO model are designed, and the problem of low efficiency and accuracy of driving scene map construction is solved, and more efficient and accurate driving scene map construction is achieved, which is suitable for a variety of driving Scene.
Patent Information
- Application Number
- CN202411064776.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-22
AI Technical Summary
The existing technology has problems of low construction efficiency and low accuracy when building driving scenario maps. Especially prior methods based on empirical statistics require a large amount of labeled data. The cross-modal mapping of the language prior methods is difficult, and the deep learning-based method is based on Method The network structure design is complex.
A neural network architecture search method based on TreeNAS is used to design a feature extraction network, build a TreeNAS-YOLO model, detect object categories and relationships through object detection algorithms, build a scene map, and use feature pyramid structures to fuse feature maps at different scales, combining adaptive growth and greed The strategy dynamically adjusts the network depth.
It improves the construction efficiency and accuracy of driving scenario maps, reduces dependence on data quality and language description, enhances generalization ability, is suitable for different driving scenarios and conditions, and overcomes the problem of difficult cross-modal mapping.
Smart Images

Figure CN119007206A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving, and in particular to a method for constructing a driving scene graph based on a neural network architecture search method. Background Art
[0002] In recent years, with the rapid development of electric vehicles, autonomous driving technology has also made significant breakthroughs. In the hazard prediction task of autonomous driving, hazard prediction algorithms based on driving scene graph understanding have become an emerging trend. The so-called driving scene graph refers to a form of relational multi-graph data representation of road scenes from the driver's perspective. The nodes of the relational multi-graph represent objects, the node attributes identify their object categories, the edges represent the connections between nodes, and the edge attributes represent the relationship between objects. Rapidly constructing a driving scene graph is a key prerequisite for the subsequent realization of hazard prediction tasks.
[0003] There are still many challenges in the existing technology when constructing spatiotemporal dynamic scene graphs. For example, the prior method based on empirical statistics needs to be trained from a large amount of labeled data to generate object detection and relationship reasoning models, which is difficult to generalize; the method based on language prior uses natural language processing technology to realize the conversion of language description to scene graph, but cross-modal mapping is difficult; the method based on deep learning uses neural networks to extract features and then realize object detection and relationship reasoning, but the network structure design is complex. These problems lead to low efficiency and low precision in the construction of driving scene graphs, which limits its scope of application. Therefore, it is particularly necessary to study a better method for constructing driving scene graphs. Summary of the invention
[0004] The purpose of the present invention is to provide a driving scene graph construction method based on a neural network architecture search method, to construct a target detection algorithm TreeNAS-YOLO network based on the neural network architecture search algorithm of TreeNAS, to design object categories, attributes and their relationships in the driving scene, and to use the target detection algorithm TreeNAS-YOLO model network to build a scene graph, thereby improving the construction efficiency and accuracy of the driving scene graph.
[0005] To achieve the above purpose, the present invention adopts the following technical solutions:
[0006] A method for constructing a driving scene graph based on a neural network architecture search method comprises the following steps:
[0007] S1. Design a feature extraction network architecture, where the feature extraction network includes an input layer, a Conv2D_BN_Relu layer, a Maxpool layer, a multi-size feature extraction module, an AdaptiveAvgPool2d layer, a Flatten layer, and a Linear layer in sequence; the multi-size feature extraction module architecture is obtained using a neural network architecture search algorithm based on TreeNAS;
[0008] S2, based on the S1 feature extraction network, designs the TreeNAS-YOLO model, an object detection algorithm for building a three-dimensional scene graph from the perspective of the vehicle.
[0009] S3, using the TreeNAS-YOLO model of the target detection algorithm to obtain the target detection result of each frame of the input image, and using the lane detection algorithm based on template matching to detect the lane line information to obtain the lane detection result;
[0010] S4. Build a scene graph based on the target detection result and lane detection result obtained in step S3, as well as the danger attribute, orientation attribute, and distance attribute of the object in the pre-set driving scene.
[0011] Furthermore, the method of obtaining a multi-scale feature extraction module architecture by using the neural network architecture search algorithm of TreeNAS in S1 comprises the following steps:
[0012] S1.1. Design multiple candidate Block modules and multiple downsampling Block modules to construct a search space. The structure and computational complexity of the Block module and the downsampling Block module are similar to those of the Bottleneck in ResNet. The multiple Block modules include three categories: ordinary convolution, teacher network, and dilated convolution. Each Block module has different hyperparameters.
[0013] S1.2, encoding and combining the multiple candidate Block modules obtained in step S1.1 by arranging the candidate Block modules in parallel, so as to expand the search space;
[0014] S1.3. Use a hierarchical search strategy as the search strategy, and find the best network architecture from the expanded search space according to the search strategy. The hierarchical search strategy refers to using the search results of the previous layer as the basis for the current layer search, that is, the current layer is obtained by continuing to search on the network architecture determined by the search results of the previous layer; the search results are divided into unit structures according to [4, 4, 5, remaining layers] and downsampling layers are inserted; in the forward propagation process, the data will be downsampled once before entering each unit to reduce the image scale and expand the number of channels.
[0015] Furthermore, in the hierarchical search process of step S1.3, an asymmetric information strategy, an adaptive growth strategy and a greedy strategy are introduced to dynamically adjust the network depth;
[0016] The asymmetric information strategy is used to train the teacher network. The implementation method is as follows: first remove the downsampling layer in the teacher network, disassemble the teacher network by treating every two layers as a unit, train each unit for 5 to 10 epochs, and obtain the performance information of each unit; the training epoch is related to the position of the unit, and the specific relationship is shown in the following formula:
[0017] epoch = 5 + (unit position number) 0.5 ;
[0018] The implementation method of the adaptive generation strategy is as follows: in each layer of search process, the performance information of each unit obtained by the teacher network training is used as a reference benchmark and candidate. If no candidate block beats the performance information of the unit of the teacher network, then one more layer of depth is added, that is, one more search process is added until the preset depth threshold of growth is reached or the performance information of the layer of the teacher network is beaten;
[0019] The implementation method of the greedy strategy is: in the process of determining the results of each layer, directly select the candidate Block module with the best performance in the search process of this layer as the final architecture result; in the process of adaptive growth, adopt a greedy strategy to select the best performance as the previous layer before adding a layer of depth.
[0020] Furthermore, the method of designing a target detection algorithm TreeNAS-YOLO model for establishing a three-dimensional scene graph from the perspective of the vehicle in step S2 includes the steps of:
[0021] S2.1. The designed target detection algorithm TreeNAS-YOLO model consists of a backbone network, an enhanced feature extraction network and a YoloHead module. The feature extraction network obtained in step 1 is used as the backbone network, and the outputs of the last three units in the multi-scale feature extraction module are the outputs of the backbone network. The backbone network is used to extract feature maps of three different scales from the input image.
[0022] S2.2, use the enhanced feature extraction network to perform effective feature extraction and fusion on the three different scale feature maps extracted by the backbone network, and obtain three effective feature maps to output to the YoloHead module;
[0023] S2.3. Taking the three valid feature maps output by the enhanced feature extraction network as input, the YoloHead module is used to obtain the detection results of each frame.
[0024] Furthermore, the enhanced feature extraction network uses a feature pyramid structure as a fusion method for feature maps of three different scales.
[0025] Furthermore, the enhanced feature extraction network consists of four Concat modules, four CSP modules, four Block-DownSampling modules and four UpSampling2D modules;
[0026] The first Concat module is the first input of the enhanced feature extraction network, connecting the first output of the backbone network and the output of the fourth UpSampling2D module, and the output is connected to the first end of the YoloHead and the input of the second Block-DownSampling module respectively through the first CSP module;
[0027] The second Concat module is the second input of the enhanced extraction reinforcement network, connecting the second output of the backbone network and the output of the first UpSampling2D module, and the output is connected to the input of the fourth UpSampling2D module through the fourth CSP module;
[0028] The input of the third Concat module is connected to the third output of the backbone network and the output of the third Block-DownSampling module, and the output is connected to the third end of YoloHead via the third CSP module;
[0029] The input of the fourth Concat module is connected to the output of the second Block-DownSampling module and the fourth CSP module, and the output is respectively connected to the input of the third Block-DownSampling module and the second end of the YoloHead via the second CSP module.
[0030] Furthermore, in the CSP module of the enhanced feature extraction network, its structure is consistent with the CSP module in the YOLO series, and the Block module in this structure can also be selected from the ten candidate Block modules defined in S1.1.
[0031] Furthermore, step S4 also includes using the SORT multi-target trajectory tracking algorithm to smooth the target detection results obtained in step 3, and fill in the missing target detection results in the three consecutive frames before and after; building a scene graph based on the filled target detection results, lane detection results, and the danger attributes, orientation attributes, and distance attributes of the objects in the pre-set driving scenario, and converting the scene graph into an adjacency matrix and a feature matrix.
[0032] Furthermore, the danger attribute, orientation attribute and distance attribute of the object in the driving scene preset in step S4 are defined as follows:
[0033] Danger attribute definition: Based on the camera position, camera screen size, and vehicle width information, the expected position frame of the vehicle and the lane line detection results are determined in the image captured by the camera; the image is divided into 7 areas based on the spatial position and size of the frame, and the danger level is determined based on the geometric position of the object after target detection; the danger level corresponds to the 7 areas, represented by Arabic numerals 1 to 7, and the meaning of each level is as follows: Level 1 represents "safe area, that is, an area without potential danger"; Level 2 represents "potential invasion area from the left"; Level 3 represents "potential invasion area from the right"; Level 4 represents "area ahead of this lane"; Level 5 represents "invasion area of this lane"; Level 6 represents the attribute of "left lane"; Level 7 represents the attribute of "right lane";
[0034] Definition of orientation attribute: The position of the geometric center of the object in the picture is divided into 4 levels, and the corresponding meanings of each level are: Level 1 is represented as "upper left, that is, the object is located at the upper left of the vehicle"; Level 2 is represented as "upper right, that is, the object is located at the upper right of the vehicle"; Level 3 is represented as "lower left, that is, the object is located at the lower left of the vehicle"; Level 4 is represented as "lower right, that is, the object is located at the lower right of the vehicle";
[0035] Distance attribute definition: The distance attribute is divided into 6 levels according to the Euclidean distance length in space, and the corresponding meanings of each level are: Level 1 means "the physical object is within the visible distance of the vehicle"; Level 2 means "the object is very far away from the vehicle"; Level 3 means "the object is generally far away from the vehicle"; Level 4 means "the object is near the vehicle"; Level 5 means "the object is close to the vehicle"; Level 6 means "the object is very close to the vehicle"; Based on the object category, the size of the candidate frame and the position of the candidate frame, the Euclidean distance between the object and the camera in space is calculated, and then the ratio of the distance to the size of the camera screen is used as the threshold for distance attribute judgment, and the corresponding distance attribute is selected according to the threshold size; the distance attribute calculation formula is as follows:
[0036] Distance=((x×x+y×y+5÷(length÷class_i)0.5)0.5
[0037] T=Distance÷905.1
[0038] Among them, x = center_x-camera_x; y = center_y-camera_y; center_x and center_y are the geometric centers of the object; camera_x and camera_y are the geometric centers of the camera; length is the length of the candidate box of the current object; class_i is the length of the benchmark candidate box of this class of objects.
[0039] The present invention proposes a method for building a driving scene graph based on a neural network architecture search. The method adopts a TreeNAS neural network architecture search method to obtain the optimal network architecture of a feature extraction network for a driving scene graph, and uses it as a backbone network together with an enhanced feature extraction network and a YoloHead module to form a target detection algorithm TreeNAS-YOLO model. Then, the target detection algorithm TreeNAS-YOLO model network is used to obtain the detection result, i.e., the object category and its candidate frame position and size information, and the driving scene graph is built in conjunction with the definition of the danger attribute, orientation attribute and distance attribute of the object in the set driving scene.
[0040] In the process of obtaining the optimal network architecture of the feature extraction network: by designing multiple candidate Block modules and multiple downsampling Block modules and combined encoding, the search space is greatly expanded, and the learning and fitting capabilities of each layer of structure are enhanced, thereby improving the overall performance of the neural network; by introducing adaptive generation and greedy strategies in the hierarchical search strategy, the network depth is dynamically adjusted to ensure that the search process is efficient and the results are excellent, avoiding the problem of missing potential excellent structures due to a narrow search space.
[0041] In the construction of the target detection network TreeNAS-YOLO model: its enhanced feature extraction network uses a feature pyramid structure to fuse feature maps of different scales, which improves the accuracy and robustness of detection and is particularly suitable for target detection tasks in complex driving scenarios.
[0042] In the process of using the results of the target detection algorithm model TreeNAS-YOLO model to build a scene graph: set the object judgment rules in the driving scene, and use the object judgment rules in the driving scene to define the dangerous attributes, orientation attributes and distance attributes of the objects in the driving scene in detail, so that the generated scene graph can accurately reflect the potential risks in the actual driving environment and provide a reliable basis for danger prediction.
[0043] Compared with the prior art, the present invention uses the TreeNAS neural network architecture search method to realize the design of the feature extraction network structure for the driving scene graph, which does not need to rely on data quality and representative empirical information, and can adaptively find the best architecture, has stronger generalization ability, and is suitable for different driving scenes and conditions. In the construction of the scene graph, it mainly relies on visual data, reduces the uncertainty and complexity brought by language description, and overcomes the problem of difficulty in cross-modal mapping. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 It is a schematic diagram of the process of a driving scene graph construction method based on a neural network architecture search method in an embodiment;
[0045] Figure 2 It is a flowchart of an embodiment of constructing a feature extraction network based on a TreeNAS neural network search algorithm;
[0046] Figure 3 1 is a schematic diagram of a target detection network structure designed based on the TreeNASTreeNAS neural network search algorithm of an embodiment;
[0047] Figure 4 is a schematic diagram of the division of dangerous areas constructed in the embodiment;
[0048] Figure 5 It is a schematic diagram of distance category division constructed in the embodiment. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0050] like Figure 1 As shown, the present embodiment provides a method for constructing a driving scene graph based on a neural network architecture search method, comprising the following steps:
[0051] S1. Design feature extraction network architecture
[0052] like Figure 2 As shown, the feature extraction network includes an input layer, a Conv2D_BN_Relu layer, a Maxpool layer, a multi-scale feature extraction module, an AdaptiveAvgPool2d layer, a Flatten layer, and a Linear layer in sequence. The multi-scale feature extraction module architecture is obtained using a neural network architecture search algorithm based on TreeNAS, and the specific steps include:
[0053] S1.1. Constructing the search space
[0054] Construct a variety of different types of candidate Block modules and downsampling Block modules. The structure and computational complexity of the Block modules and downsampling Block modules are similar to those of the Bottleneck in ResNet. Multiple Block modules should include three types: ordinary convolution, teacher network and hole convolution, and the hyperparameters of each Block module are different, including the convolution kernel size and convolution kernel category.
[0055] The number of candidate blocks is designed according to the requirements. In this embodiment, 10 different types of block modules and 4 different types of downsampling block modules are constructed, where the convolution kernel size of BlockBlock is 3 to 7. The specific BlockBlock structure is shown in Table 1:
[0056] Table 1 is the hyperparameters and configuration table of the candidate Block module and downsampling module in the search space of this implementation
[0057]
[0058]
[0059] According to Table 1, Block1, Block3 and Block5 are all ordinary convolutions, among which the convolution kernel size of Block1 is 5×5, the convolution kernel size of Block3 is 7×7, and the convolution kernel size of Block5 is 3×3. These three modules are the benchmarks for the other module variants. Block2 is a teacher network with residual connection and is the reference module for the design of other variants. Block4 and Block7 are both dilated convolutions; among which the dilation rate of Block4 is 2, the convolution kernel size is 3×3, and the activation function is LeakyReLU; Block7 is a dilated convolution with SE module, the dilation rate of Block7 is 2, and the convolution kernel size is 3×3. Block6 is an ordinary convolution with SE module (Squeeze-and-Excitation), and the convolution kernel size is 3×3. Block8 and Block9 are both grouped convolutions with 4 groups, among which Block8 adopts channel shuffle and residual connection, and the convolution kernel size is 3×3; Block9 adopts dense connection, and the convolution kernel size is 3×3. Block 10 is a common convolution with both residual connection and SE module, and the convolution kernel size is 3×3.
[0060] In the above structure: the SE module (Squeeze-and-Excitation) is implemented using two fully connected layers. The first layer is used to reduce the number of channels to 1 / 16, using LeakyReLU as the activation function. The second layer is used to expand the number of channels back using ReLU as the activation function. Dense connections use channel connections to implement subsequent layer inputs with channel data from all previous layers. Channel shuffling uses mathematical transformations to implement grouped convolutions where each group has channel data from the other three groups. Its mathematical description is shown in the following formula:
[0061] xshuffled=reshape(transpose(reshape(x,(B,G,CG,H,W)),(0,2,1,3,4)),(B,C,H,W))
[0062] Where CG = C / G, B is the batch size, C is the number of channels, H is the height, W is the width, and G is the number of groups.
[0063] S1.2, in order to expand the search space and enhance the learning and fitting capabilities of each layer structure, this embodiment uses the method of arranging candidate Block modules in parallel to encode and combine the multiple candidate Block modules obtained in step S1.1. The number of candidate Block modules arranged in parallel is set by default to a maximum of two Blocks arranged in parallel, which can also be customized according to requirements. The encoding combination method can be series, parallel or series-parallel, and the combination method can also be customized according to requirements.
[0064] S1.3. Use a hierarchical search strategy as the search strategy, and find the best network architecture from the search space according to the search strategy. The hierarchical search strategy refers to using the search results of the previous layer as the basis for the current layer search, that is, the current layer is obtained by continuing to search on the network architecture determined by the search results of the previous layer. The search results are divided into unit structures according to [4, 4, 5, remaining layers] and the downsampling layer is inserted. The final classification head is implemented using a global pooling layer. During the forward propagation process, the data will be downsampled once before entering each unit to reduce the image scale and expand the number of channels. The downsampling Block module in the structure can be switched or modified in S1.1 as needed.
[0065] In order to dynamically adjust the feature network depth, in this embodiment, an asymmetric information strategy, an adaptive growth strategy and a greedy strategy are introduced in the hierarchical search process.
[0066] The asymmetric information strategy for training the teacher network includes: first removing the downsampling layer in the teacher network, disassembling the teacher network as if every two layers were a unit to obtain multiple units, training each unit for 5 to 10 epochs, and obtaining the performance information of each unit; the training epoch is related to the position of the unit, and the specific relationship is shown in the following formula:
[0067] epoch = 5 + (unit position number) 0.5 .
[0068] The implementation method of the adaptive generation strategy is: in each layer of search process, the performance information of each unit obtained by the teacher network training is used as a reference benchmark and candidate. If no candidate Block module beats the performance information of the unit of the teacher network, then another layer of depth is added, that is, the search process is added once, until the preset depth threshold of growth is reached or the performance information of the layer of the teacher network is defeated.
[0069] The implementation method of the greedy strategy is: in the process of determining the results of each layer, directly select the candidate Block module with the best performance in the search process of this layer as the final architecture result. In the process of adaptive growth, the greedy strategy is also used to select the best performance as the previous layer before adding a layer of depth.
[0070] S2. Design target detection algorithm TreeNAS-YOLO model
[0071] like Figure 3 As shown in the figure, the target detection algorithm TreeNAS-YOLO model consists of a backbone network, an enhanced feature extraction network and a YoloHead module; its implementation method includes:
[0072] S2.1. The feature extraction network obtained in step 1 is used as the backbone network. The outputs of the last three units in the multi-scale feature extraction module are the outputs of the backbone network. The backbone network is used to extract feature maps of three different scales from the input image.
[0073] S2.2. Use the enhanced feature extraction network to perform effective feature extraction and fusion on the three different scale feature maps extracted by the backbone network, and obtain three effective feature maps to output to the YoloHead module.
[0074] The enhanced feature extraction network uses the feature pyramid structure as a fusion method for three feature maps of different scales. It consists of four Concat modules, four CSP modules, four Block-DownSampling modules and four UpSampling2D modules: the first Concat module is the first input of the enhanced feature extraction network, connecting the first output of the backbone network and the output of the fourth UpSampling2D module, and the output is connected to the first end of YoloHead and the input of the second Block-DownSampling module through the first CSP module.
[0075] The second Concat module is the second input of the enhanced extraction reinforcement network, connecting the second output of the backbone network and the output of the first UpSampling2D module, and the output is connected to the input of the fourth UpSampling2D module through the fourth CSP module.
[0076] The input of the third Concat module is connected to the third output of the backbone network and the output of the third Block-DownSampling module, and the output is connected to the third end of YoloHead via the third CSP module.
[0077] The input of the fourth Concat module is connected to the output of the second Block-DownSampling module and the fourth CSP module, and the output is respectively connected to the input of the third Block-DownSampling module and the second end of the YoloHead via the second CSP module.
[0078] In this embodiment, the structure of the CSP module of the enhanced feature extraction network is consistent with the CSP module in the YOLO series, with a total of three convolution processes. After each convolution, the channel is divided into two parts, one of which is directly output to the end, and the other enters the next convolution, and finally stacked according to the channel dimension. The Bottleneck module in the structure can be selected from the ten candidate blocks defined in S1.1, and the default is Block2.
[0079] S2.3. Taking the three valid feature maps output by the enhanced feature extraction network as input, the YoloHead module is used to obtain the target detection results of each frame.
[0080] S3, using the TreeNAS-YOLO model of the target detection algorithm to predict the target detection result of each frame of the input image, and using the lane detection algorithm based on template matching to detect the lane line information to obtain the lane detection result;
[0081] S4. Build a driving scene diagram
[0082] This embodiment designs the object categories, attributes and their relationships in the driving scene, uses the target detection algorithm TreeNAS-YOLO network to obtain the detection results, and implements the construction of the driving scene graph according to the detection results and the object categories, attributes and their relationships in the designed driving scene. The implementation method includes:
[0083] S4.1. Use the SORT (Simple Online and Realtime Tracking) multi-target trajectory tracking algorithm to smooth the target detection results obtained in step 3, and fill in the missing target detection information in the three consecutive frames. Specifically:
[0084] First, a tracker is established for each object in the target detection result of the previous frame, and the position is predicted using IoU (Intersection over Union). The predicted position is compared with the target detection result of this frame. If there is a corresponding target detection result in the predicted position of this frame, the result is output and the tracker is updated. If there is no corresponding target detection result in the predicted position of this frame, the result of the position is retained as the predicted position result in the previous frame. If there is still no corresponding target detection result at this position in the next frame, the result is no longer output and the tracker of this object is closed. If there is a corresponding target detection result at this position in the next frame, the result is output and the tracker status is updated. S4.2. A scene graph is generated based on the target detection result (object category and its candidate box position and size) obtained in step S3, the lane detection result obtained in step S3, and the danger attribute, orientation attribute and distance attribute of the object in the pre-set driving scene, and the scene graph is converted into an adjacency matrix and a feature matrix.
[0085] In this embodiment, the danger attribute, orientation attribute and distance attribute (see Table 2) in the driving scene are set to provide a reliable basis for object determination for the scene graph, so as to improve the accuracy of the generated scene graph in reflecting the potential risks in the actual driving environment.
[0086] Table 2 is a detailed table of the setting levels and meanings of the danger attribute, direction attribute and distance attribute.
[0087]
[0088] Definition of dangerous attributes: Dangerous attributes are divided into 7 types according to their levels: ["Safe-zone", "potential-left", "potential-right", "in-lane", "invaded-lane", "left-lane", "right-lane". Based on information such as the camera position, the camera screen size, and the width of the vehicle, the expected position box of the vehicle's advance and the lane line detection results are determined in the image captured by the camera, and the image is divided into 7 areas based on the spatial position and size of the box. Information such as the distance from the ground, the camera screen size, and the width of the vehicle can be based on actual application parameters. In this embodiment, the distance from the ground is 1.6 meters, the camera screen size is 640×640, and the width of the vehicle is 1.9 meters. The specific partitions are as follows: Figure 4 The danger level of the object is determined based on its geometric position after target detection.
[0089] Distance and orientation attribute definition: The position of the geometric center of the object in the picture is divided into four types: ["Top-left", "Top-right", "Bottom-left", "Bottom-right"], such as Figure 5 As shown. The distance attribute is divided into six types according to the Euclidean distance length in space: ["visible", "very_far", "far", "near", "close", "extremely_close"]. The Euclidean distance between the object and the camera in space is calculated based on the object category, the size of the candidate box, and the position of the candidate box. Then, the ratio of the distance to the size of the camera image is used as the threshold for distance attribute judgment, and the corresponding distance attribute is selected according to the threshold. Object categories include 20 types such as self-car "ego", car, bicycle, person, truck, bus, cat, dog, etc. Some distance attributes are shown in Figure 1. Figure 5 The calculation formula is as follows:
[0090] Distance=((x×x+y×y+5÷(length÷class_i) 0.5 ) 0.5
[0091] T=Distance÷905.1
[0092] Among them, x = center_x-camera_x; y = center_y-camera_y; center_x and center_y are the geometric centers of the object; camera_x and camera_y are the geometric centers of the camera; length is the length of the candidate box of the current object; class_i is the length of the benchmark candidate box of this class of objects.
[0093] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A method for constructing a driving scene graph based on a neural network architecture search method, characterized in that: The following steps are involved: S1. Design a feature extraction network architecture, where the feature extraction network includes an input layer, a Conv2D_BN_Relu layer, a Maxpool layer, a multi-size feature extraction module, an AdaptiveAvgPool2d layer, a Flatten layer, and a Linear layer in sequence; the multi-size feature extraction module architecture is obtained using a neural network architecture search algorithm based on TreeNAS; S2, based on the S1 feature extraction network, designs the TreeNAS-YOLO model, an object detection algorithm for building a three-dimensional scene graph from the perspective of the vehicle. S3, using the TreeNAS-YOLO model of the target detection algorithm to obtain the target detection result of each frame of the input image, and using the lane detection algorithm based on template matching to detect the lane line information to obtain the lane detection result; S4. Build a scene graph based on the target detection result and lane detection result obtained in step S3, as well as the danger attribute, orientation attribute, and distance attribute of the object in the pre-set driving scene.
2. A method for constructing a driving scene graph based on a neural network architecture search method according to claim 1, characterized in that: The method S1 uses the TreeNAS neural network architecture search algorithm to obtain a multi-size feature extraction module architecture, including the following steps: S1.
1. Design multiple candidate Block modules and multiple downsampling Block modules to construct a search space. The structure and computational complexity of the Block module and the downsampling Block module are similar to those of the Bottleneck in ResNet. The multiple Block modules include three categories: ordinary convolution, teacher network, and dilated convolution. Each Block module has different hyperparameters. S1.2, encoding and combining the multiple candidate Block modules obtained in step S1.1 by arranging the candidate Block modules in parallel, so as to expand the search space; S1.
3. Use a hierarchical search strategy as the search strategy, and find the best network architecture from the expanded search space according to the search strategy. The hierarchical search strategy refers to using the search results of the previous layer as the basis for the current layer search, that is, the current layer is obtained by continuing to search on the network architecture determined by the search results of the previous layer; the search results are divided into unit structures according to [4, 4, 5, remaining layers] and downsampling layers are inserted; in the forward propagation process, the data will be downsampled once before entering each unit to reduce the image scale and expand the number of channels.
3. The method for constructing a driving scene graph based on a neural network architecture search method according to claim 2, characterized in that: The hierarchical search process in step S1.3 also introduces an asymmetric information strategy, an adaptive growth strategy and a greedy strategy to dynamically adjust the network depth; The asymmetric information strategy is used to train the teacher network. The implementation method is as follows: first remove the downsampling layer in the teacher network, disassemble the teacher network by treating every two layers as a unit, train each unit for 5 to 10 epochs, and obtain the performance information of each unit; the training epoch is related to the position of the unit, and the specific relationship is shown in the following formula: epoch = 5 + (unit position number) 0.5 ; The implementation method of the adaptive generation strategy is as follows: in each layer of search process, the performance information of each unit obtained by the teacher network training is used as a reference benchmark and candidate. If no candidate block beats the performance information of the unit of the teacher network, then one more layer of depth is added, that is, one more search process is added until the preset depth threshold of growth is reached or the performance information of the layer of the teacher network is beaten; The implementation method of the greedy strategy is: in the process of determining the results of each layer, directly select the candidate Block module with the best performance in the search process of this layer as the final architecture result; in the process of adaptive growth, adopt a greedy strategy to select the best performance as the previous layer before adding a layer of depth.
4. The method for constructing a driving scene graph based on a neural network architecture search method according to claim 1, characterized in that: The step S2 designs a method for a target detection algorithm TreeNAS-YOLO model established for a three-dimensional scene graph from the perspective of the vehicle, comprising the steps of: S2.
1. The designed target detection algorithm TreeNAS-YOLO model consists of a backbone network, an enhanced feature extraction network and a YoloHead module. The feature extraction network obtained in step 1 is used as the backbone network, and the outputs of the last three units in the multi-scale feature extraction module are the outputs of the backbone network. The backbone network is used to extract feature maps of three different scales from the input image. S2.2, use the enhanced feature extraction network to perform effective feature extraction and fusion on the three different scale feature maps extracted by the backbone network, and obtain three effective feature maps to output to the YoloHead module; S2.
3. Taking the three valid feature maps output by the enhanced feature extraction network as input, the YoloHead module is used to obtain the detection results of each frame.
5. The method for constructing a driving scene graph based on a neural network architecture search method according to claim 4, characterized in that: The enhanced feature extraction network uses a feature pyramid structure as a fusion method for feature maps of three different scales.
6. The method for constructing a driving scene graph based on a neural network architecture search method according to claim 5, characterized in that: The enhanced feature extraction network consists of four Concat modules, four CSP modules, four Block-DownSampling modules and four UpSampling2D modules; The first Concat module is the first input of the enhanced feature extraction network, connecting the first output of the backbone network and the output of the fourth UpSampling2D module, and the output is connected to the first end of the YoloHead and the input of the second Block-DownSampling module respectively through the first CSP module; The second Concat module is the second input of the enhanced extraction reinforcement network, connecting the second output of the backbone network and the output of the first UpSampling2D module, and the output is connected to the input of the fourth UpSampling2D module through the fourth CSP module; The input of the third Concat module is connected to the third output of the backbone network and the output of the third Block-DownSampling module, and the output is connected to the third end of YoloHead via the third CSP module; The input of the fourth Concat module is connected to the output of the second Block-DownSampling module and the fourth CSP module, and the output is respectively connected to the input of the third Block-DownSampling module and the second end of the YoloHead via the second CSP module.
7. The method for constructing a driving scene graph based on a neural network architecture search method according to claim 6, characterized in that: In the CSP module of the enhanced feature extraction network, its structure is consistent with the CSP module in the YOLO series, and the Block module in the structure can be selected from the ten candidate Block modules defined in S1.
1.
8. The method for constructing a driving scene graph based on a neural network architecture search method according to claim 1, characterized in that: The step S4 also includes using the SORT multi-target trajectory tracking algorithm to smooth the target detection results obtained in step 3, and fill in the missing target detection results in the three consecutive frames before and after; building a scene graph according to the filled target detection results, lane detection results, and the danger attributes, orientation attributes and distance attributes of the objects in the pre-set driving scene, and converting the scene graph into an adjacency matrix and a feature matrix.
9. The method for constructing a driving scene graph based on a neural network architecture search method according to claim 8, characterized in that: The danger attribute, orientation attribute and distance attribute of the object in the driving scene pre-set in step S4. are defined as follows: Danger attribute definition: Based on the camera position, camera screen size, and vehicle width information, the expected position frame of the vehicle and the lane line detection results are determined in the image captured by the camera; the image is divided into 7 areas based on the spatial position and size of the frame, and the danger level is determined based on the geometric position of the object after target detection; the danger level corresponds to the 7 areas, represented by Arabic numerals 1 to 7, and the meaning of each level is as follows: Level 1 means "safe area, that is, the area without potential danger"; Level 2 means "potential invasion area from the left"; Level 3 means "potential invasion area from the right"; Level 4 means "area ahead of the lane"; Level 5 means "invasion area of the lane"; Level 6 means the attribute is "left lane"; Level 7 means the attribute is "right lane"; Direction attribute definition: The position of the geometric center of the object in the picture is divided into 4 levels, and the corresponding meanings of each level are: Level 1 is represented as "upper left, that is, the object is located at the upper left of the vehicle"; Level 2 is represented as "upper right, that is, the object is located at the upper right of the vehicle"; Level 3 is represented as "lower left, that is, the object is located at the lower left of the vehicle"; Level 4 is represented as "lower right, that is, the object is located at the lower right of the vehicle"; Distance attribute definition: The distance attribute is divided into 6 levels according to the Euclidean distance length in space, and the corresponding meanings of each level are: Level 1 means "the physical object is visible from the vehicle"; Level 2 means "the object is very far away from the vehicle"; Level 3 means "the object is generally far away from the vehicle"; Level 4 means "the object is near the vehicle"; Level 5 means "the object is close to the vehicle"; Level 6 means "the object is very close to the vehicle"; Based on the object category, the size of the candidate frame and the position of the candidate frame, the Euclidean distance between the object and the camera in space is calculated, and then the ratio of the distance to the size of the camera screen is used as the threshold for distance attribute judgment, and the corresponding distance attribute is selected according to the threshold size; the distance attribute calculation formula is as follows: Distance=((x×x+y×y+5÷(length÷class_i) 0.5 ) 0.5 T=Distance÷905.1 Among them, x = center_x-camera_x; y = center_y-camera_y; center_x and center_y are the geometric centers of the object; camera_x and camera_y are the geometric centers of the camera; length is the length of the candidate box of the current object; class_i is the length of the benchmark candidate box of this class of objects.
