A neural network search method, device, computer equipment and storage medium

Abstract

The embodiment of the present invention provides a neural network search method, device, computer equipment and storage medium. The method includes: determining a main node and redundant nodes set for one or more sensors on a vehicle, deploying a main neural network for sensor perception on the main node, setting a network structure adjacent to the main neural network as a neighborhood space, generating conditions for evaluating a secondary neural network for redundant nodes as evaluation conditions, and performing a neural network structure search in the neighborhood space with the goal of satisfying the evaluation conditions to obtain a secondary neural network. The secondary neural network is deployed on the redundant nodes for sensor perception, which can reduce the search time of the neural network structure search, reduce the consumption of computing resources, and efficiently find the local optimal solution. The neural network structure search designs corresponding evaluation conditions based on the characteristics of the redundant nodes, thereby searching for a secondary neural network suitable for deployment on the redundant nodes, ensuring that the secondary neural network can run normally on the redundant nodes.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of autonomous driving, and in particular to a neural network search method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, autonomous driving technology has also made new breakthroughs. In the process of autonomous driving moving from academia to industry and gradually integrating into people's daily lives, the issues of autonomous driving safety and stability have become increasingly prominent.

[0003] Redundant nodes are set up in autonomous driving with independent automotive-grade hardware, and algorithms for perceiving the environment can be deployed. Algorithms for perceiving the environment widely use neural networks, and their complexity and requirements for computing power are becoming increasingly higher. However, the existing automotive-grade hardware has relatively weak computing power, and it is becoming increasingly difficult to run complete algorithms for perceiving the environment on redundant nodes. Summary of the invention

[0004] The embodiments of the present invention propose a neural network search method, apparatus, computer device and storage medium to solve the problem that it is increasingly difficult to run a complete algorithm for perceiving the environment on redundant nodes of autonomous driving.

[0005] In a first aspect, an embodiment of the present invention provides a neural network search method, comprising:

[0006] Determine a master node and a redundant node set for one or more sensors on the vehicle, wherein a main neural network applied with the sensor perception is deployed on the master node;

[0007] Setting a network structure adjacent to the main neural network as a neighborhood space;

[0008] generating conditions for evaluating the secondary neural network for the redundant nodes as evaluation conditions;

[0009] With the goal of satisfying the evaluation conditions, a neural network structure search is performed in the neighborhood space to obtain a secondary neural network to be deployed at the redundant node to apply the sensor perception.

[0010] In a second aspect, an embodiment of the present invention further provides a neural network search device, comprising:

[0011] A node determination module, used to determine a primary node and a redundant node set for one or more sensors on the vehicle, wherein a primary neural network using the sensor perception is deployed on the primary node;

[0012] A neighborhood space setting module, used to set a network structure adjacent to the main neural network as a neighborhood space;

[0013] An evaluation condition generating module, used for generating conditions for evaluating the secondary neural network for the redundant nodes as evaluation conditions;

[0014] A network search module is used to perform a neural network structure search in the neighborhood space with the goal of satisfying the evaluation conditions, obtain a secondary neural network, and deploy it in the redundant node to apply the sensor perception.

[0015] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0016] one or more processors;

[0017] a memory for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the neural network search method as described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the neural network search method as described in the first aspect is implemented.

[0020] In this embodiment, the main node and redundant node set for one or more sensors on the vehicle are determined, and a main neural network for sensor perception is deployed on the main node. A network structure adjacent to the main neural network is set as a neighborhood space, and conditions for evaluating the secondary neural network are generated for the redundant nodes as evaluation conditions, with the goal of satisfying the evaluation conditions. A neural network structure search is performed in the neighborhood space to obtain the secondary neural network, which is deployed on the redundant nodes and uses sensor perception. The neighborhood space is designed with the main neural network as prior knowledge to ensure the effectiveness of the search space. The search space is converged, which can reduce the search time of the neural network structure search, reduce the consumption of computing resources, and efficiently find the local optimal solution. The neural network structure search designs corresponding evaluation conditions based on the characteristics of the redundant nodes, thereby searching for a secondary neural network suitable for deployment on the redundant nodes, ensuring that the secondary neural network can run normally on the redundant nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a neural network search method provided in Embodiment 1 of the present invention;

[0022] Figure 2 A schematic diagram of the structure of a vehicle provided in Embodiment 1 of the present invention;

[0023] FIG. 3A to FIG. 3DAn example diagram of a neighborhood space provided in Embodiment 1 of the present invention

[0024] Figure 4 is a flow chart of a neural network search method provided by Embodiment 2 of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a neural network search device provided in Embodiment 3 of the present invention;

[0026] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Embodiment 1

[0029] Figure 1 This is a flowchart of a neural network search method provided in Embodiment 1 of the present invention. This embodiment can be applied to autonomous driving, and a neural network structure search is performed based on a neural network of a main node to obtain a situation of a neural network of a redundant node. The method can be executed by a neural network search device, and the neural network search device can be implemented by software and / or hardware, and can be configured in a computer device, such as a server, a personal computer, a workstation, etc., and specifically includes the following steps:

[0030] Step 101: Determine a primary node and a redundant node set for one or more sensors on a vehicle.

[0031] The vehicle in this embodiment is equipped with an automatic driving system and can support automatic driving. The so-called automatic driving may refer to the vehicle's ability to perceive the environment, plan paths and autonomously control the vehicle, that is, to use electronic technology to control the vehicle for human-like driving.

[0032] According to the degree of control over vehicle control tasks, autonomous driving vehicles can be divided into L0 non-automation (NoAutomotion), L1 driver assistance (Driver Assistance), L2 partial automation (Partial Automation), L3 conditional automation (Conditional Automation), L4 high automation (High Automation), and L5 full automation (Full Automation).

[0033] The autonomous driving vehicle in this embodiment may refer to a vehicle that meets any requirement of L1-L5, wherein the system performs an auxiliary function at L1-L3, and when reaching L4, the vehicle driving will be handed over to the system. Therefore, the autonomous driving vehicle may be selected as a vehicle that meets any requirement of L4 or L5.

[0034] like Figure 2 As shown, the vehicle 200 may include a driving control device 201 , a body bus 202 , ECUs (Electronic Control Unit) 203 , ECU 204 , ECU 205 , sensors 206 , sensor 207 , sensor 208 , and actuators 209 , 210 , 211 .

[0035] The driving control device (also known as the vehicle brain) 201 is responsible for the overall intelligent control of the entire vehicle 200. The driving control device 201 can be a separately set controller, such as a programmable logic controller (PLC), a single-chip microcomputer, an industrial control machine, etc.; it can also be a device composed of other electronic devices with input / output ports and operation control functions; it can also be a computer device installed with vehicle driving control applications. The driving control device can analyze and process the data sent from each ECU and / or each sensor received on the body bus 202, make corresponding decisions, and send the instructions corresponding to the decisions to the body bus.

[0036] The body bus 202 may be a bus for connecting the driving control device 201, ECU 203, ECU 204, ECU 205, sensor 206, sensor 207, sensor 208 and other not shown devices of the vehicle 200. Since the high performance and reliability of the CAN (Controller Area Network) bus have been widely recognized, the body bus commonly used in motor vehicles is the CAN bus. Of course, it is understandable that the body bus may also be other types of buses.

[0037] The body bus 202 can send the instructions sent by the driving control device 201 to ECU 203, ECU 204, and ECU 205. ECU 203, ECU 204, and ECU 205 analyze and process the instructions and then send them to corresponding execution devices for execution.

[0038] Sensor 206 , sensor 207 , and sensor 208 include but are not limited to lidar, cameras, and the like.

[0039] It should be understood that Figure 2The number of vehicles, driving control devices, body buses, ECUs, actuators, and sensors in the embodiment is merely illustrative. Any number of vehicles, driving control devices, body buses, ECUs, and sensors may be provided as required.

[0040] For a sensor or a combination of multiple (two or more) sensors, independent main nodes and redundant nodes are set on the vehicle. The main node is deployed with a main neural network for sensor perception, and the redundant node is deployed with a sub-neural network for sensor perception. The structure of the main neural network is different from that of the sub-neural network, for example, a combination of LaneNet and H-Net, PolyLaneNet, VPGNet, 3D-LaneNet, etc.

[0041] Generally speaking, the main node is hardware with stronger computing power, and the redundant node is hardware with weaker computing power. Therefore, the secondary neural network is generally a lightweight model and the main neural network is generally a large-size model. That is, the structure of the secondary neural network is usually simpler than that of the main neural network, and the secondary neural network has weaker learning and generalization capabilities for sensor data, while the main neural network has stronger learning and generalization capabilities for sensor data. In addition, the accuracy of the secondary neural network is lower than that of the main neural network.

[0042] The services provided by the main neural network for autonomous driving are the same or similar to the services provided by the sub-neural network for autonomous driving, such as detecting lane lines, detecting traffic lights, detecting traffic signs, detecting other vehicles, etc. At this time, the input of the main neural network and the input of the sub-neural network are both data collected by sensors, and the output of the main neural network is the same as the output of the sub-neural network.

[0043] Redundancy is the use of two or more identical or similar, relatively independent, and switchable designs for system availability. In short, when a system fails, another system can be quickly switched to ensure that the system remains available without changing the system's operating status.

[0044] In this embodiment, a redundant design is implemented for the perception of autonomous driving. When the vehicle is autonomously driving, the priority of applying the main neural network is higher than that of the secondary neural network. That is, under normal conditions, the main neural network of the main node and the secondary neural network of the redundant node are started, but the authority is initially given to the main node, that is, the main neural network of the main node is used to provide services for autonomous driving. When the main node is abnormal (such as downtime) or the main neural network is abnormal (such as severe timeout, crash), the control is given to the redundant node, that is, the secondary neural network of the secondary node is used to provide services for autonomous driving.

[0045] Step 102: Set a network structure adjacent to the main neural network as a neighborhood space.

[0046] In this embodiment, Neural Architecture Search (NAS) is applied to design a secondary neural network for redundant nodes, that is, an effective network structure can be detected for redundant nodes as a secondary neural network without human intervention.

[0047] Generally speaking, neural network structure search is a global search with a large search space, high consumption of computing resources, long search time, and the search results may not be optimal.

[0048] In this embodiment, the main neural network can be used as a basis to find network structures adjacent to the main neural network, that is, network structures that are the same or similar to the main neural network, thereby forming a neighborhood space as the search space for neural architecture search (NAS) to realize neighborhood search.

[0049] First, compared with global search, neighborhood search has a smaller search space, so the search time is shorter and the computing resources consumed are less.

[0050] Secondly, the main neural network has verified the perception results and is a high-performance model, so it can be deployed on the main node to provide services for autonomous driving. Therefore, the main neural network of the main node can be used as a priori knowledge and considered to be a relatively excellent model in performance. Then, based on the main neural network, the neighborhood search space is designed and the neural network structure search is performed. The result of the search is the local optimum with constraints in the neighborhood of the main neural network.

[0051] Finally, global search has a larger search space. Although it theoretically contains more local optimal solutions, or even global optimal solutions, the neural network structure search itself has certain limitations. In an overly large search space, the search difficulty will be greater, and it is possible to fall into some relatively poor local optimal solutions.

[0052] In one embodiment of the present invention, step 102 includes the following steps:

[0053] Step 1021, search for the backbone network in the main neural network.

[0054] When designing the main neural network, which is an algorithm for perceiving the environment, we will consider what kind of backbone network to use. The backbone network is the basic network structure of the main neural network. For example, is it one-stage detection or two-stage detection, whether it is anchor free detection, etc.

[0055] Generally speaking, once the backbone network is determined, we can qualitatively explain what type of main neural network it is and what detection methods are used.

[0056] Step 1022: Query the first parameter set in the backbone network.

[0057] There are multiple network structures set in the backbone network, such as depth, width, operator type, etc., which are recorded as first parameters. These first parameters are also the search dimensions considered when performing neural network structure search.

[0058] Step 1023: Under the condition of maintaining the architecture of the backbone network unchanged, a second parameter adjacent to the first parameter is set as a neighborhood space.

[0059] Generally speaking, neural network structure search does not change the backbone network backbone. Therefore, under the condition of maintaining the architecture of the backbone network unchanged, other network structures adjacent to (the same or similar to) the first parameter can be found and recorded as the second parameter, so that the second parameter can be formed into a neighborhood space, that is, the second parameter can replace the first parameter and be integrated into the architecture of the backbone network.

[0060] In one example, the first parameter includes a first operator and the second parameter includes a second operator. In this example, the first operator applied in one or more layers of a backbone network backbone can be determined, and the second operator of the same type as the first operator is set as the neighborhood space.

[0061] For example, Figure 3A As shown, if the first operator is a common convolution Conv, then the convolution Conv can be replaced by a second operator such as group convolution Group Conv, separable convolution Seq Conv, etc.

[0062] In another example, the first parameter includes a first quantity and the second parameter includes a second quantity. In this example, the first quantity of channels in the backbone network backbone can be determined, a range containing the first quantity is generated, and the second quantity within the range and applicable to the channel is set as the neighborhood space.

[0063] The number of channels is also the width of the neural network. The number of channels in each layer of the neural network increases as the size of the features (such as image data) decreases. Generally, when the size of the features is halved, the number of channels doubles.

[0064] In the search space (neighborhood space) of neural network structure search, this constraint can be broken and a reasonable range of channels can be designed for each layer of network calculation, so that the search process can try more possibilities. Considering computational optimization and efficiency issues, the number of channels is generally a multiple of 8 or 16.

[0065] For example, if the first number of channels in a certain layer of the backbone network is 128, then the second number can be designed to be [64, 72, 96, 128, 144]. If the first number of channels in a certain layer is 512, then the second number can be designed to be [448, 480, 512, 576].

[0066] For example, Figure 3B As shown, the second number of channels of a certain layer of the network is set to [16, 24, 32], the second number of channels of the next layer of the network is set to [32, 40, 48], and the second number of channels of the next layer of the network is set to [64, 80, 96].

[0067] In another example, the first parameter includes the volume (also known as dimension, size) of the convolution kernel, and the second parameter includes the volume (also known as dimension, size) of the convolution kernel. In this example, it can be determined that there is a convolution kernel in the backbone network backbone, and multiple volumes are set for the convolution kernel based on the frequency of use as the neighborhood space.

[0068] In general, if Figure 3C As shown, the volumes of the most frequently used convolution kernels are 1x1 (not shown in the figure), 3x3, 5x5, and 7x7. Therefore, the second parameter can be set to 1x1, 3x3, 5x5, and 7x7.

[0069] In another example, the first parameter includes a first depth, and the second parameter includes a second depth. In this example, the first depth of the backbone network may be determined, and the first depth may be reduced to the second depth as the neighborhood space.

[0070] Generally speaking, the depth of the secondary neural network of the redundant node is lower. Therefore, the second depth can be obtained by descending based on the first depth of the backbone network backbone of the main neural network of the main node.

[0071] Furthermore, for backbone networks using ResNet, MobileNet or other similar networks with Block as the basic unit, the first depth inside each Block can also be reduced to the second depth for neural network structure search.

[0072] For example, Figure 3DAs shown, the second depth of a certain Block is set to [1, 2, 3, 4], the second depth of the next Block is set to [1, 2, 3, 4], and the second depth of the next Block is set to [1, 2, 3, 4].

[0073] Of course, the above-mentioned way of designing the neighborhood space is only used as an example. When implementing the embodiment of the present invention, other ways of designing the neighborhood space can be set according to actual conditions. For example, whether to add residuals, whether to add se structures, and the value of the expansion factor of mbblock are designed as neighborhood spaces based on the backbone network, etc., and the embodiment of the present invention does not limit this. In addition, in addition to the above-mentioned way of designing the neighborhood space, those skilled in the art can also adopt other ways of designing the neighborhood space according to actual needs, and the embodiment of the present invention does not limit this.

[0074] Step 103: Generate conditions for evaluating the secondary neural network for the redundant nodes as evaluation conditions.

[0075] In this embodiment, one or more conditions for evaluation can be set for the secondary neural network, recorded as evaluation conditions, and the evaluation conditions are used to search the neural network structure for redundant nodes, thereby realizing the neural network structure search under specific constraints, and searching for secondary neural networks suitable for deployment on redundant nodes.

[0076] In a specific implementation, conditions for evaluating the accuracy of the sub-neural network can be generated as first-level evaluation conditions, and at least one of the conditions for evaluating the total delay, computational amount, and memory consumption value of the sub-neural network can be generated as second-level evaluation conditions.

[0077] Among them, the priority of the first-level evaluation conditions is higher than the priority of the second-level evaluation conditions, that is, the neural network structure search takes accuracy as the main search target, and at least one of the total delay, computational amount, and memory consumption value as the secondary search target.

[0078] Of course, the above evaluation conditions are only used as examples. When implementing the embodiment of the present invention, other evaluation conditions can be set according to actual conditions, and the embodiment of the present invention does not limit this. In addition, in addition to the above evaluation conditions, those skilled in the art can also adopt other evaluation conditions according to actual needs, and the embodiment of the present invention does not limit this.

[0079] Step 104, with the goal of satisfying the evaluation conditions, a neural network structure search is performed in the neighborhood space to obtain a secondary neural network to be deployed on redundant nodes for sensor perception.

[0080] The neighborhood space is used as the search space. The search space determines which network structures will be evaluated. The neural network structure search is performed within the scope of the search space to search out the secondary neural network.

[0081] The evaluation conditions will provide some numbers that can reflect the efficiency of all network structures in the search space. The secondary neural network will be evaluated based on the evaluation conditions, and the secondary neural network that meets the evaluation conditions will be searched out, and then deployed in redundant nodes, using the same set of sensors as the main neural network to perceive the environment.

[0082] In one embodiment of the present invention, step 104 includes the following steps:

[0083] Step 1041: Perform a neural network structure search in the neighborhood space to obtain a network structure.

[0084] Neural network structure search depends on the search method (also known as search strategy), and the neural network structure search can be performed in the neighborhood space using a suitable search method to search for the corresponding network structure, which is used for the secondary neural network.

[0085] Among them, the search methods include but are not limited to the one-shot search method based on supernet sampling, the network search method based on reinforcement learning, the hyperparameter search method based on Bayes, and the like.

[0086] Furthermore, the search method is related to the search dimension. In order to select a suitable search method and improve the efficiency of the search, the search dimension that is adapted to the network structure in the neighborhood space can be determined, and a search method that is adapted to the search dimension can be set, so as to apply the search method to perform a neural network structure search in the neighborhood space to obtain the network structure.

[0087] For example, the Bayesian-based hyperparameter search method is suitable for hyperparameter search in continuous space, such as the second number of channels, the second depth, etc.

[0088] For example, the one-shot search method based on supernet sampling is suitable for the second operator, the volume of the convolution kernel, the second depth, etc. without affecting the network networking.

[0089] A neural network structure search often includes multiple search dimensions. Different search dimensions can be searched in the same neural network structure search or in different neural network structure searches.

[0090] For example, the Bayesian hyperparameter search method can be used to search for the second number of channels, thereby setting it as the number of channels of the sub-neural network. After determining the number of channels, the Bayesian method is then used again to search for the second depth as the depth of the sub-neural network. Finally, the supernet-based One-shot search method is used to simultaneously search for the second operator and the volume of the convolution kernel as the operator applied to each layer of the network in the sub-neural network and the volume of each convolution kernel to obtain the sub-neural network.

[0091] Step 1042: Count the indicators used for the network structure as evaluation indicators.

[0092] In this embodiment, with respect to the evaluation conditions for evaluating the sub-neural network, the same type of indicators can be generated for the network structure constituting the sub-neural network and recorded as evaluation indicators.

[0093] In one example, the evaluation condition is used to evaluate the accuracy of the sub-neural network, and the accuracy of the network structure constituting the sub-neural network can be generated as an evaluation indicator.

[0094] In another example, the evaluation condition is used to evaluate the total delay of the sub-neural network, and the total delay can be generated for the network structure constituting the sub-neural network as an evaluation indicator.

[0095] Furthermore, for the evaluation of the total delay, if real-time online measurement is used, additional hardware is usually requested specifically for the evaluation. If a sub-neural network is designed for redundant nodes, the redundant nodes are requested accordingly to evaluate the total delay of the sub-neural network. In addition, the number of requests during the execution of the neural network structure search is large, resulting in queuing, thereby reducing the efficiency of the execution of the neural network structure search.

[0096] In this example, the sub-delay of processing data at each network layer in the network structure can be measured. For example, the computational amount of the operators applied at each network layer in the network structure can be used to approximate the sub-delay, and so on. The sub-delay is recorded in a preset table, all the sub-delays in the table are added together, and the sum of all the sub-delays in the table is calculated to obtain the total delay of the network structure.

[0097] Although there is a certain difference between the predicted total delay and the actual total delay, the relative difference in total delay between different network structures can be reflected, which can help the neural network structure search converge. There is no need to evaluate the actual total delay value on redundant nodes, which greatly improves the efficiency of evaluating the total delay.

[0098] In another example, the evaluation condition is used to evaluate the computational amount of the sub-neural network, and the computational amount can be generated for the network structure constituting the sub-neural network as an evaluation indicator.

[0099] In another example, the evaluation condition is used to evaluate the memory consumption value of the sub-neural network, and a memory consumption value can be generated for the network structure constituting the sub-neural network as an evaluation indicator.

[0100] Of course, the above evaluation indicators are only used as examples. When implementing the embodiment of the present invention, other evaluation indicators can be set according to actual conditions, and the embodiment of the present invention does not limit this. In addition, in addition to the above evaluation indicators, those skilled in the art can also use other evaluation indicators according to actual needs, and the embodiment of the present invention does not limit this.

[0101] Step 1043: If the evaluation index does not meet the evaluation conditions, it is allowed to penalize the network structure in the neural network structure search.

[0102] Step 1044: If the evaluation index meets the evaluation condition, it is prohibited to penalize the network structure in the neural network structure search.

[0103] Compare the evaluation index of the network structure with the evaluation conditions.

[0104] If the evaluation index does not meet the evaluation conditions, the network structure can be penalized in the objective function used for optimization during the neural network structure search. If the evaluation index meets the evaluation conditions, the network structure can be prohibited from being penalized in the objective function used for optimization during the neural network structure search. This will tend to search for a network structure that meets the evaluation index.

[0105] For example, if the evaluation condition is that the total delay value of the secondary neural network is less than 100ms, then in the process of searching the neural network structure, for the network structure with a total delay value exceeding 100ms, a corresponding penalty will be imposed on the objective function, while for the network structure with a total delay value less than 100ms, no penalty will be imposed. In this way, the network structure with a total delay value less than 100ms will tend to be searched out.

[0106] In this embodiment, the main node and redundant node set for one or more sensors on the vehicle are determined, and a main neural network for sensor perception is deployed on the main node. A network structure adjacent to the main neural network is set as a neighborhood space, and conditions for evaluating the secondary neural network are generated for the redundant nodes as evaluation conditions, with the goal of satisfying the evaluation conditions. A neural network structure search is performed in the neighborhood space to obtain the secondary neural network, which is deployed on the redundant nodes and uses sensor perception. The neighborhood space is designed with the main neural network as prior knowledge to ensure the effectiveness of the search space. The search space is converged, which can reduce the search time of the neural network structure search, reduce the consumption of computing resources, and efficiently find the local optimal solution. The neural network structure search designs corresponding evaluation conditions based on the characteristics of the redundant nodes, thereby searching for a secondary neural network suitable for deployment on the redundant nodes, ensuring that the secondary neural network can run normally on the redundant nodes.

[0107] Embodiment 2

[0108] Figure 4 This is a flow chart of a neural network search method provided in Embodiment 2 of the present invention. This embodiment is based on the previous embodiment and further adds an operation of training a secondary neural network. The method specifically includes the following steps:

[0109] Step 401: Determine a primary node and a redundant node set for one or more sensors on a vehicle.

[0110] Among them, the main neural network with sensor perception is deployed on the main node.

[0111] Step 402: Set a network structure adjacent to the main neural network as a neighborhood space.

[0112] Step 403: Generate conditions for evaluating the secondary neural network for the redundant nodes as evaluation conditions.

[0113] Step 404, with the goal of satisfying the evaluation conditions, a neural network structure search is performed in the neighborhood space to obtain a secondary neural network to be deployed on redundant nodes for sensor perception.

[0114] Step 405: Acquire data collected by the sensor.

[0115] Step 406: Use the data as samples to train the secondary neural network.

[0116] In this embodiment, the sub-neural network is obtained through neural network structure search, and then the data collected by the sensor, such as image data, audio data, point cloud data, etc., can be obtained, and the data can be used as samples to retrain the sub-neural network. The performance of the sub-neural network can improve the accuracy of autonomous driving decisions and ensure the safety of autonomous driving.

[0117] For different types of tasks, such as target detection, classification, etc., the training of the secondary neural network can be supervised training or unsupervised training, which is not limited in this embodiment.

[0118] Furthermore, for supervised training, the data collected by the sensor may be labeled. For example, for traffic light detection, the label may be a detection frame and a category of the traffic light.

[0119] Generally, in order to ensure the accuracy of the sample, the label can be manually labeled, that is, manually input the label for the data collected by the sensor through the client. If the data collected by the sensor has been labeled in advance, the label can be directly labeled.

[0120] Of course, considering that the output of the main neural network has high accuracy, in order to increase the speed of labeling, the output of the main neural network can also be used as a label, and so on. This embodiment does not limit this.

[0121] In one training approach, the student network can be retrained.

[0122] In this training method, the data collected by the sensor can be used as samples to update the preset training set, and the data set can be used to train the adaptive neural network for the specified learning objectives.

[0123] In another training method, considering that the sub-neural network is designed based on the main neural network (such as maintaining the backbone network), the sub-neural network has some of the same structures as the main neural network. The main neural network can be used as the source network to determine some or all of the weights in the backbone network of the main neural network, and inherit some or all of the weights in the backbone network of the sub-neural network. If the inheritance is completed, the data collected by the sensor is used as a sample to train the sub-neural network and accelerate the convergence to a local optimal solution close to the main neural network.

[0124] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0125] Embodiment 3

[0126] Figure 5 This is a structural block diagram of a neural network search device provided in Embodiment 3 of the present invention, which may specifically include the following modules:

[0127] A node determination module 501 is used to determine a primary node and a redundant node set for one or more sensors on a vehicle, wherein a primary neural network using the sensor perception is deployed on the primary node;

[0128] A neighborhood space setting module 502 is used to set a network structure adjacent to the main neural network as a neighborhood space;

[0129] An evaluation condition generating module 503, used to generate conditions for evaluating the secondary neural network for the redundant nodes as evaluation conditions;

[0130] The network search module 504 is used to perform a neural network structure search in the neighborhood space with the goal of satisfying the evaluation conditions, obtain a secondary neural network, and deploy it in the redundant node to apply the sensor perception.

[0131] In one embodiment of the present invention, the input of the main neural network and the input of the sub-neural network are both data collected by the sensor;

[0132] The output of the main neural network is the same as the output of the secondary neural network;

[0133] When the vehicle is automatically driving, the priority of applying the main neural network is higher than the priority of applying the secondary neural network.

[0134] In one embodiment of the present invention, the neighborhood space setting module 502 includes:

[0135] A backbone network search module, used to search for a backbone network in the main neural network;

[0136] A first parameter query module, used to query a first parameter set in the backbone network;

[0137] The second parameter setting module is used to set a second parameter adjacent to the first parameter as a neighborhood space under the condition of maintaining the architecture of the backbone network unchanged.

[0138] In an example of an embodiment of the present invention, the second parameter setting module includes:

[0139] An operator determination module, used to determine a first operator to be applied in one or more layers of the backbone network;

[0140] An operator setting module, used for setting a second operator of the same type as the first operator as a neighborhood space;

[0141] and / or,

[0142] A quantity determination module, configured to determine a first quantity of channels in the backbone network;

[0143] A range generation module, configured to generate a range including the first quantity;

[0144] A quantity setting module, used for setting a second quantity within the range and applicable to the channel as a neighborhood space;

[0145] and / or,

[0146] A convolution kernel determination module, used to determine whether a convolution kernel exists in the backbone network;

[0147] A volume setting module, used for setting a plurality of volumes for the convolution kernel as a neighborhood space based on the frequency of use;

[0148] and / or,

[0149] A depth determination module, used to determine a first depth of the backbone network;

[0150] The depth reduction module is used to reduce the first depth to a second depth as a neighborhood space.

[0151] In one embodiment of the present invention, the evaluation condition generation module 503 includes:

[0152] A first-level generation module, used to generate conditions for evaluating the accuracy of the secondary neural network as first-level evaluation conditions;

[0153] A second-level generation module is used to generate at least one condition of the total delay, the amount of calculation, and the memory consumption value for evaluating the secondary neural network as a second-level evaluation condition;

[0154] The priority of the first-level evaluation condition is higher than the priority of the second-level evaluation condition.

[0155] In one embodiment of the present invention, the network search module 504 includes:

[0156] A network structure search module, used for performing a neural network structure search in the neighborhood space to obtain a network structure, wherein the network structure is used for a secondary neural network;

[0157] An evaluation index statistics module, used for counting the index used to evaluate the network structure as the evaluation index;

[0158] A penalty allowing module, used for allowing the network structure to be penalized in the neural network structure search if the evaluation index does not meet the evaluation condition;

[0159] A penalty prohibition module is used to prohibit the penalty of the network structure in the neural network structure search if the evaluation index meets the evaluation condition.

[0160] In one embodiment of the present invention, the network structure search module includes:

[0161] A search dimension determination module, used to determine a search dimension adapted to the network structure in the neighborhood space;

[0162] A search method setting module, used to set a search method adapted to the search dimension;

[0163] The search method application module is used to apply the search method to perform a neural network structure search in the neighborhood space to obtain a network structure.

[0164] In one embodiment of the present invention, the evaluation index statistics module includes:

[0165] A sub-delay calculation module, used to measure the sub-delay of data processing at each layer of the network structure;

[0166] A sub-delay recording module, used for recording the sub-delay in a preset table;

[0167] The sub-delay summing module is used to calculate the sum of all the sub-delays in the table to obtain the total delay of the network structure.

[0168] In one embodiment of the present invention, it also includes:

[0169] A sensing data acquisition module, used to acquire data collected by the sensor;

[0170] The network training module is used to train the secondary neural network using the data as samples.

[0171] In one embodiment of the present invention, the network training module includes:

[0172] A weight determination module, used to determine some or all of the weights in the backbone network of the main neural network;

[0173] A weight inheritance module, used to inherit part or all of the weights in the backbone network of the secondary neural network;

[0174] The inheritance training module is used to train the secondary neural network using the data as samples if the inheritance is completed.

[0175] The neural network search device provided in the embodiment of the present invention can execute the neural network search method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0176] Embodiment 4

[0177] Figure 6A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 6 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0178] like Figure 6 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0179] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0180] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0181] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0182] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0183] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0184] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the neural network search method provided in the embodiment of the present invention.

[0185] Embodiment 5

[0186] Embodiment 5 of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned neural network search method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0187] Among them, computer-readable storage media may include, for example, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0188] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A neural network search method, characterized in that: include: Determine a master node and a redundant node set for one or more sensors on the vehicle, wherein a main neural network applied with the sensor perception is deployed on the master node; Setting a network structure adjacent to the main neural network as a neighborhood space; generating conditions for evaluating the secondary neural network for the redundant nodes as evaluation conditions; With the goal of satisfying the evaluation condition, a neural network structure search is performed in the neighborhood space to obtain a secondary neural network to be deployed in the redundant node to apply the sensor perception; The network structure adjacent to the main neural network is set as a neighborhood space, including: Finding a backbone network in the main neural network; Querying a first parameter set in the backbone network; Under the condition of maintaining the architecture of the backbone network unchanged, setting a second parameter adjacent to the first parameter as a neighborhood space; The step of setting a second parameter adjacent to the first parameter as a neighborhood space comprises: Determining a first operator to be applied in one or more layers of the backbone network; Setting a second operator of the same type as the first operator as the neighborhood space; and / or, Determining a first number of channels in the backbone network; generating a range containing the first quantity; Setting a second quantity within the range and applicable to the channel as a neighborhood space; and / or, Determining that a convolution kernel exists in the backbone network; Setting a plurality of volumes for the convolution kernel based on the used frequency as a neighborhood space; and / or, Determining a first depth of the backbone network; The first depth is reduced to a second depth as a neighborhood space.

2. The method according to claim 1, characterized in that: The input of the main neural network and the input of the sub-neural network are both data collected by the sensor; The output of the main neural network is the same as the output of the secondary neural network; When the vehicle is automatically driving, the priority of applying the main neural network is higher than the priority of applying the secondary neural network.

3. The method according to claim 1, characterized in that The conditions for evaluating the secondary neural network generated for the redundant nodes, as evaluation conditions, include: generating conditions for evaluating the accuracy of the secondary neural network as first-level evaluation conditions; Generate at least one condition for evaluating the total delay, computation amount, and memory consumption value of the secondary neural network as a second-level evaluation condition; The priority of the first-level evaluation condition is higher than the priority of the second-level evaluation condition.

4. The method according to claim 1, characterized in that: The step of performing a neural network structure search in the neighborhood space to obtain a secondary neural network for deployment at the redundant node to apply the sensor perception with the goal of satisfying the evaluation condition includes: Performing a neural network structure search in the neighborhood space to obtain a network structure, wherein the network structure is used for a secondary neural network; Counting the indicators used to evaluate the network structure as evaluation indicators; If the evaluation index does not meet the evaluation condition, it is allowed to penalize the network structure in the neural network structure search; If the evaluation index satisfies the evaluation condition, it is prohibited to penalize the network structure in the neural network structure search.

5. The method according to claim 4, characterized in that The performing of a neural network structure search in the neighborhood space to obtain a network structure comprises: Determining a search dimension adapted to the network structure in the neighborhood space; Setting a search method adapted to the search dimension; The search method is applied to perform a neural network structure search in the neighborhood space to obtain a network structure.

6. The method according to claim 4, characterized in that The statistics are used for the indicators of the network structure, as evaluation indicators, including: Measuring the sub-delay of processing data at each layer of the network in the network structure; Recording the sub-delays in a preset table; The sum of all the sub-delays in the table is calculated to obtain the total delay of the network structure.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Acquiring data collected by the sensor; The secondary neural network is trained using the data as samples.

8. The method according to claim 7, characterized in that The method of training the secondary neural network using the data as samples includes: Determining some or all of the weights in a backbone network of the main neural network; Inheriting part or all of the weights in the backbone network of the secondary neural network; If the inheritance is completed, the data is used as a sample to train the secondary neural network.

9. A neural network search device, characterized in that: include: A node determination module, used to determine a primary node and a redundant node set for one or more sensors on the vehicle, wherein a primary neural network using the sensor perception is deployed on the primary node; A neighborhood space setting module, used to set a network structure adjacent to the main neural network as a neighborhood space; An evaluation condition generating module, used for generating conditions for evaluating the secondary neural network for the redundant nodes as evaluation conditions; A network search module, used to perform a neural network structure search in the neighborhood space with the goal of satisfying the evaluation condition, and obtain a secondary neural network to be deployed in the redundant node to apply the sensor perception; The neighborhood space setting module includes: A backbone network search module, used to search for a backbone network in the main neural network; A first parameter query module, used to query a first parameter set in the backbone network; A second parameter setting module, used to set a second parameter adjacent to the first parameter as a neighborhood space under the condition that the architecture of the backbone network remains unchanged; The second parameter setting module includes: An operator determination module, used to determine a first operator to be applied in one or more layers of the backbone network; An operator setting module, used for setting a second operator of the same type as the first operator as a neighborhood space; and / or, A quantity determination module, configured to determine a first quantity of channels in the backbone network; A range generation module, configured to generate a range including the first quantity; A quantity setting module, used for setting a second quantity within the range and applicable to the channel as a neighborhood space; and / or, A convolution kernel determination module, used to determine whether a convolution kernel exists in the backbone network; A volume setting module, used for setting a plurality of volumes for the convolution kernel as a neighborhood space based on the frequency of use; and / or, A depth determination module, used to determine a first depth of the backbone network; The depth reduction module is used to reduce the first depth to a second depth as a neighborhood space.

10. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the neural network search method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the search method of the neural network according to any one of claims 1 to 8 is implemented.

Images