A visual perception method and system for assisted driving

Abstract

The application discloses a visual perception method for assisted driving, comprising the following steps: S100, collecting assisted driving image data, and constructing input X after data set preprocessing; S200, constructing a multi-task neural network model F and substituting the input X into the model to obtain a plurality of branch task-specific outputs f i ; S300, calculating the final loss function L all of the multi-task neural network model F; S400, optimizing the minimum as the target of the loss function L all , and finally obtaining the optimal multi-task neural network model F * . A system using the above visual perception method for assisted driving is also disclosed, specifically comprising a data input module, a shared layer feature extraction module, a task layer structure search module and a model output module. The application has the beneficial effect that a multi-task model is constructed to effectively reduce the network parameter quantity and the calculation quantity, simultaneously reduce the online reasoning time, and is very beneficial to practical application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of assisted driving, in particular to a visual perception method and system for assisted driving. BACKGROUND

[0002] With the vigorous development of deep learning technology, the field of assisted driving has become its main landing scene, and visual perception is an important module of the assisted driving system, including lane line detection, drivable area segmentation and target detection, etc. The traditional deep learning algorithm usually only solves one of the above tasks, which cannot meet the demand of simultaneously perceiving multiple environmental factors in the process of assisted driving. The current more advanced algorithm often improves the perception efficiency through multi-task learning, and the existing multi-task assisted driving visual perception system usually improves in the training stage of the deep learning model, mainly in the learning of three angles of structure, optimization method and task relationship. Among them:

[0003] (1) The multi-task model of assisted driving visual perception first focuses on the design of network structure, which effectively improves the performance in single task. For example, FCN branch is added on the basis of Faster RCNN to generate mask of corresponding categories, so as to accurately complete the semantic segmentation task. Or, an encoder-decoder structure is used to design a multi-task model with a shared encoder and three task decoders for classification, target detection and semantic segmentation. Or, a lightweight CNN is used as an encoder to extract image features, and then the feature map is input into each decoder to complete the visual perception task, which achieves optimal performance in accuracy and speed.

[0004] (2) The optimization method of multi-task model mainly embodies the optimization of loss function and gradient. The former is mainly for weighting and adding different tasks, and the key is how to weight. At present, there are methods of task uncertainty, learning rate, model performance, return amplitude and geometric mean, etc. The latter is mainly around the optimization of gradient, which is mainly the modification of gradient to balance the training rate between tasks, such as GradNorm method, etc.

[0005] (3) Additional attention is also needed on the task relationship. Weakly related tasks may bring negative transfer effect. Therefore, the model needs to learn the task representation or the relationship between tasks, such as clustering according to similarity, and using the learned feature representation to further improve performance.

[0006] The above methods and systems have achieved certain results, but still have the following two problems:

[0007] (1) Expert experience is needed to adjust the task weight or gradient coefficient for specific scenarios or specific tasks.

[0008] (2) A lot of time is required to manually experiment to determine the model structure parameters in the system. Summary of the Invention

[0009] One objective of this application is to provide a visual perception method and system for assisted driving that can solve at least one of the problems in the aforementioned background art.

[0010] To achieve the above objectives, the technical solution adopted in this application is: a visual perception method for assisted driving, comprising the following steps:

[0011] S100: Collects assisted driving image data, and preprocesses the dataset to form input X;

[0012] S200: Construct a multi-task neural network model F and substitute the input X to obtain the outputs f specific to multiple branch tasks. i (X);

[0013] S300: Calculate the final loss function L of the multi-task neural network model F. all ;

[0014] S400: Minimize as the loss function L all The objective is optimized to obtain the optimal multi-task neural network model F. * .

[0015] Preferably, the multi-task neural network model F includes a parameter-sharing layer B and a task layer T; then step S200 includes the following process:

[0016] S210: Substitute the input X into the parameter sharing layer B;

[0017] S220: The parameter-sharing layer B extracts the image features of the input X and obtains the feature map M;

[0018] S230: Substitute the feature map M into the task layer T to obtain the output f specific to multiple branch tasks. i (X).

[0019] Preferably, the initial structural parameters of task layer T are determined as follows: First, each branch structure of task layer T is initialized to obtain the parameter α of each branch structure. i Subsequently, the parameter α for each branch structure was obtained through random combination. i0 Then, in step S230, output f. i (X) can be represented as f i (X, w(α)); where α is the branch structure parameter, w is the neural network weight, the size of w is obtained by iteratively from α, and the value of α in the first iteration is α. i0 .

[0020] Preferably, in step S300, the output loss of the multi-task neural network model F is L. out The structural loss of the multi-task neural network model F is L. arch Then the loss function L all =L out +L arch .

[0021] Preferably, the output loss of each branch task is L. i The output loss L of the multi-task neural network model F is... out The output loss L of each branch task i Weighted, structural loss L arch This is the sum of the loss differences between the various branch tasks; that is...

[0022]

[0023]

[0024] Where α is the branch structure parameter, w is the neural network weight, and N is the number of branch tasks.

[0025] Preferably, step S400 includes the following specific processes:

[0026] S410: Perform a structure search on task layer T to obtain the branch task network structure parameters that maintain balance among multiple tasks and the optimal neural network weight parameters.

[0027] S420: Minimize the loss function as the optimization objective, and train the parameters in step S410 using the algorithm;

[0028] S430: Fit the trained parameters to the multi-task neural network model F to obtain the optimal multi-task neural network model F determined by automated search. * .

[0029] Preferably, in step S410, for the structural search of task layer T, the branch structure of task layer T needs to be transformed from a discrete structure into a continuous search space; the transformed continuous search space is represented by the following formula:

[0030]

[0031] Here, node X is the implicit representation, function o(·) represents the candidate operations between nodes, and O represents the set of candidate operations. It is the weight of the structural candidate operation o between nodes (i,j).

[0032] Preferably, in step S420, by adjusting the output loss L outand structural loss L arch Perform dual optimization to achieve the loss function L all The optimization is achieved by minimizing the following objective using gradient descent:

[0033]

[0034]

[0035] Preferably, the objective is solved using an iterative optimization algorithm to obtain the optimal branch structure parameters α and network weights w; by restoring the continuous structure parameter α to discrete network node connection operations o, a multi-task neural network model F with a determined task branch structure and optimal neural network weight performance is obtained. * The formula for discretizing the continuously represented neural network structure is as follows:

[0036]

[0037]

[0038] A visual perception system for assisted driving includes a data input module, a shared layer feature extraction module, a task layer structure search module, and a model output module. The data input module receives and preprocesses images from multiple datasets. The shared layer feature extraction module extracts low-level semantic features from the images. The task layer structure search module automatically searches for network structure parameters for each task. The model output module outputs the optimal model.

[0039] Compared with the prior art, the beneficial effects of this application are as follows:

[0040] (1) This application addresses the real-time requirements of assisted driving to perceive complex environments by constructing a multi-task model that includes lane line segmentation and obstacle detection, which effectively reduces the number of network parameters and computational load, while also reducing online inference time, which is very beneficial for practical applications.

[0041] (2) The task layer structure search method proposed in this application solves the problem of unbalanced training among multiple tasks. The method is simple in principle, has low hardware requirements, and is suitable for widespread application.

[0042] (3) The multi-task method and system proposed in this application have obtained a multi-task model through automated search, which performs well in visual perception tasks and has high detection accuracy. Attached Figure Description

[0043] Figure 1 The following describes the overall working steps of the visual perception method for assisted driving according to the present invention.

[0044] Figure 2 This is a schematic diagram of the overall structure of the visual perception system for assisted driving according to the present invention.

[0045] Figure 3 This is a flowchart illustrating the workflow of the task layer structure search module in the visual perception system for assisted driving according to the present invention.

[0046] In the diagram: Data input module 100, shared layer feature extraction module 200, task layer structure search module 300, and model output module 400. Detailed Implementation

[0047] The present application will be further described below with reference to specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0048] In the description of this application, it should be noted that the directional terms such as "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", and "counterclockwise" indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. They should not be construed as limiting the specific protection scope of this application.

[0049] It should be noted that the terms "first," "second," etc., in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0050] One aspect of this application provides a visual perception method for assisted driving, such as... Figure 1 As shown, one preferred embodiment includes the following steps:

[0051] S100: Collects assisted driving image data, and then preprocesses the collected dataset to form input X.

[0052] S200: Construct a multi-task neural network model F, then substitute the input X into the multi-task neural network model F to obtain the output f specific to multiple branch tasks. i (X).

[0053] S300: Calculate the final loss function L of the multi-task neural network model F. all .

[0054] S400: Minimize as the loss function L allThe objective is optimized to obtain the optimal multi-task neural network model F. * .

[0055] In this embodiment, the number of tasks in the current scene can be set to N. When collecting image data in step S100, the order of the collected datasets is not restricted. However, for ease of annotation, they can generally be sorted according to task priority, thus obtaining a task data sequence D = {D1, D2, ..., D...}. N The task data sequence D is then preprocessed to obtain the required input X.

[0056] Understandably, data annotation can be either continuous or sparse. If sparse annotation is used, non-existent task annotations can be specially marked, such as setting the value of pixel annotations to -1 or 255.

[0057] Specifically, in the application scenarios of assisted driving, visual perception tasks generally include lane line detection, feasible area detection, and obstacle detection; therefore, the total number of tasks N = 3. In step S100, image data is generally collected by the vehicle-mounted camera; based on the collection results of the vehicle-mounted camera, the three tasks each have three different datasets, with the number of samples in each dataset being n1, n2, and n3, respectively. The sum of the three sample numbers is the total number of samples n.

[0058] In this embodiment, the samples can employ sparse labeling, meaning that the samples for a specific task only contain the labeled data for that task. For sparse labeling, data is read in batches during the preprocessing stage, and the labels are processed uniformly, with missing task labels set as marker values.

[0059] Lane detection is considered a semantic segmentation task, with nine categories ranging from 0 to 8. Non-existent lanes are marked as -1. Similarly, feasible region detection is also considered a semantic segmentation task, but with only two categories: feasible and infeasible, marked with values ​​of 0 and 1 respectively. Non-existent regions are marked as -1. Finally, obstacle detection can be performed using object detection methods. The types of obstacle detection are well-known to those skilled in the art; generally, there are seven categories, with non-existent obstacles marked as -1.

[0060] Finally, after completing the data sampling and preprocessing process, we can obtain the data sequence D = {D1, D2, D3}, which is composed of lane line detection, feasible area detection and obstacle detection in the order of task, and the corresponding input vector X.

[0061] In this embodiment, the multi-task neural network model F includes a parameter sharing layer B and a task layer T; therefore, step S200 includes the following process:

[0062] S210: Substitute the input X into the parameter sharing layer B.

[0063] S220: The parameter sharing layer B extracts the image features of the input X and obtains the feature map M.

[0064] S230: Substitute the feature map M into the task layer T to obtain the output f specific to multiple branch tasks. i (X).

[0065] It is understandable that the first part of the multi-task neural network model F can serve as a parameter-sharing layer B for extracting image features, and the parameter-sharing layer B has a fixed network structure. Furthermore, there are various ways for the parameter-sharing layer B to extract image features, all of which are well known to those skilled in the art; a common method is to use a convolutional neural network to extract image features.

[0066] Specifically, in step S220, the parameter sharing layer B can be constructed using HRNet, preferably HRNet-W32. ResNet can be used for image feature extraction; this model is based on residual blocks, and the training impact of network depth is avoided by fitting a residual mapping f(x)-x. The network uses multiple modules composed of residual blocks, each module using several residual blocks with the same number of output channels, and finally passes through a fully connected layer to obtain the output feature map M.

[0067] To facilitate understanding, the image feature extraction process in parameter-sharing layer B will be described in detail below:

[0068] First, the original image is downsampled by a factor of 4 using two 3×3 convolutional layers. Then, the number of channels is adjusted by repeatedly stacking the Bottle Neck in ResNet. Next, a series of Transition and Stage structures are used. Each Transition adds a new scale branch. For example, Transition 1, after the output of the previous step, uses two parallel 3×3 convolutional layers to obtain two different scale branches: a downsampled scale of 4x and a downsampled scale of 8x. Transition 2 adds a new downsampled scale of 16x on top of these two scale branches. This process continues, performing downsampling at four scales. In the Stage stage, for each scale branch, information from different scales is first fused through four Basic Blocks (the Basic Blocks in ResNet). The output of each scale branch is obtained by fusing the outputs from all branches. For example, for the output of the 4x downsampling branch, it adds the output of the 4x downsampling branch (without any processing), the output of the 8x downsampling branch (upsampled by 2x using Up×2), and the output of the 16x downsampling branch (upsampled by 4x using Up×4), and finally uses ReLU to obtain the fused output of the 4x downsampling branch. This process continues, corresponding to 4 transitions, resulting in 4 stages. Ultimately, the last block of Stage 4 outputs only the output of the 4x downsampling branch, retaining only the feature map M output from the highest resolution feature layer, i.e., the feature map M output from the parameter-shared layer B.

[0069] In this embodiment, the task layer T includes multiple branch structures. Since the structure of the task layer T cannot be determined in advance, the initial structure parameters of the task layer T can be determined through the following process: First, initialize each branch structure of the task layer T. After initialization, the parameter of each branch structure can be set to α. i Then, by random combination, the parameter α of each branch structure can be obtained. i0 In step S230, the obtained feature map M is fed into the branches of each task to obtain the required task output f. i (X), specifically f i (X, w(α)). Here, α is the branch structure parameter, w is the neural network weight, and the size of w is obtained iteratively from α, with α taking the value of α in the first iteration. i0 Each subsequent iteration will use the result of the previous iteration.

[0070] It is understandable that the parameter α of the branch structure... i The parameter α is obtained by random combination. i0There are various methods, all of which are well known to those skilled in the art. A common method is to perform random combinations using the most basic convolution and pooling operations.

[0071] Specifically, based on the value of the visual perception task N, the number of branches in task layer T can be determined; that is, lane detection, feasible region detection, and obstacle detection can each correspond to a separate branch structure. Since both lane detection and feasible region detection tasks employ deep learning methods for semantic segmentation, their branch structures are identical. The specific branch structure determination results are shown in the table below:

[0072]

[0073] In this embodiment, in step S300, the overall loss function L of the multi-task neural network model F is... all The output loss L of the multi-task neural network model F out and structural loss L arch The sum; that is, the loss function L all =L out +L arch .

[0074] In this embodiment, the output loss of each branch task is L. i The output loss L of the multi-task neural network model F is... out The output loss L of each branch task i Weighted, structural loss L arch This is the sum of the loss differences between the various branch tasks; that is...

[0075]

[0076] Where α is the branch structure parameter, w is the neural network weight, and N is the number of branch tasks.

[0077] It is understandable that we can set the true label to Y, and then output f according to the branch task. i (X, w(α)) can be used to obtain the output loss L of the branch task. i =L i (Y,f i (X,w(α))).

[0078] To facilitate understanding, the loss function L of the multi-task neural network model F can be described below. all The specific calculation process will be described in detail.

[0079] Specifically, in step S230, the feature map M is input into different task branches established for lane detection, feasible region detection, and obstacle detection, resulting in corresponding task outputs f1(X), f2(X), and f3(X). Then, the output loss L for each branch task is calculated. i In the calculation, the output loss for both lane detection and feasible area detection tasks is calculated using the mean square error. The specific calculation formula is as follows:

[0080]

[0081] Y1 and Y2 are the real labels for the lane line detection task and the feasible area detection task, respectively.

[0082] For obstacle detection tasks, the output loss L3 is the sum of multiple subtasks. The types of subtasks included in obstacle detection tasks are well-known to those skilled in the art, generally including object category, object size, and bounding box coordinates. Therefore, the output loss for obstacles is the sum of the category loss and the position loss. Furthermore, when calculating the output loss for obstacles, the category loss can use the cross-entropy loss function, and the position loss can use the mean absolute error. Thus, the formula for calculating the obstacle transport loss L3 can be obtained:

[0083]

[0084] Where p is the probability of class loss occurring, and Y3 is the true label of the obstacle detection task.

[0085] In summary, the output loss L can be obtained. out = L1 + L2 + L3; Structural loss L arch =|L1―L2|+|L1―L3|+|L2―L3|.

[0086] In this embodiment, as Figure 3 As shown, step S400 includes the following specific processes:

[0087] S410: Perform a structure search on task layer T to obtain the branch task network structure parameters that maintain balance among multiple tasks and the optimal neural network weight parameters.

[0088] S420: Minimize the loss function as the optimization objective, and train the parameters from step S410 using the algorithm.

[0089] S430: Fit the trained parameters to the multi-task neural network model F to obtain the optimal multi-task neural network model F determined by automated search. * .

[0090] In this embodiment, in step S410, for the structural search of task layer T, the branch structure of task layer T needs to be transformed from a discrete structure into a continuous search space. A common transformation method is Softmax, and the transformed continuous search space is represented by the following formula:

[0091]

[0092] Here, node X is the implicit representation, function o(·) represents the candidate operations between nodes, and O represents the set of candidate operations. It is the weight of the structural candidate operation o between nodes (i,j).

[0093] Understandably, after the transformation, the branch structure parameters are changed from discrete variables to learnable continuous variables.

[0094] Specifically, the candidate operation set O for the branch structure includes the specific network structure shown in the table below.

[0095] Structure name Structure meaning max_pool_3x3 Max pooling layer of size 3x3 avg_pool_3x3 Average pooling layer of size 3x3 skip_connect Skip connection between nodes sep_conv_3x3 Separable convolution layer of size 3x3 sep_conv_5x5 Separable convolution layer of size 5x5 dil_conv_3x3 Dilated convolution layer of size 3x3 dil_conv_5x5 Dilated convolution layer of size 5x5 none No connection between nodes

[0096] The candidate operations in the table above represent operations between two nodes. The task branch structure is determined by combining these operations based on the parameter α. The number of nodes in the branch structure can be set according to actual needs; generally, a branch structure can be set to contain 7 nodes, including 2 input nodes, 4 intermediate nodes, and the final output node.

[0097] In this embodiment, in step S420, as described above, the loss function L of the multi-task neural network system F... all For output loss L out and structural loss L arch The sum. Therefore, for the loss function L all The optimization process is essentially about optimizing the output loss L. out and structural loss L arch The process involves dual optimization. Specifically, the following objective can be minimized using gradient descent:

[0098]

[0099]

[0100] Specifically, when N is 3, the above target parameters can be expressed as follows:

[0101]

[0102]

[0103] Understandably, there are multiple iterative solutions for the aforementioned objective, among which the most common approach is to solve the objective using an approximate iterative optimization algorithm. In this dual optimization process, the two variables α and w are mutually constrained; where α is an external variable and w is an internal variable. This can be understood as seeking the overall loss function L of the multi-task neural network model F. all During the minimization process, one parameter α is constrained by another variable w(α). Once the optimal variable w(α) is determined, the size of variable α is also determined accordingly.

[0104] In this embodiment, after obtaining the optimal branch structure parameters α and network weights w, the continuous structure parameters α can be restored to discrete network node connection operations o, ultimately resulting in a multi-task neural network model F with a determined task branch structure and optimal neural network weight performance. * The formula for discretizing the continuously represented neural network structure is as follows:

[0105]

[0106]

[0107] Another aspect of this application provides a system utilizing the aforementioned visual perception method for assisted driving, such as... Figure 2 and Figure 3 As shown, one preferred embodiment includes a data input module 100, a shared layer feature extraction module 200, a task layer structure search module 300, and a model output module 400. The data input module 100 receives image data from multiple datasets and preprocesses it to obtain the desired input X, then sends the obtained input X to the shared layer feature extraction module 200. The shared layer feature extraction module 200 extracts low-level semantic features of the image to obtain a feature map M, which is then sent to the task layer structure search module 300. The task layer structure search module 300 automatically searches for the network structure parameters for each task to obtain the optimal branch structure parameters α and neural network weights w, and sends the obtained optimal branch structure parameters α and neural network weights w to the model output module 400. The model output module 400 fits the obtained parameters to a multi-task neural network model F to obtain the optimal multi-task neural network model F. * And output it.

[0108] The basic principles, main features, and advantages of this application have been described above. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of this application. Various changes and modifications can be made to this application without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection claimed by this application is defined by the appended claims and their equivalents.

Claims

1. A visual perception method for assisted driving, characterized in that, Includes the following steps: S100: Collects assisted driving image data, and preprocesses the dataset to form input X; S200: Construct a multi-task neural network model F and substitute the input X to obtain the outputs specific to multiple branch tasks. ; S300: Calculate the final loss function of the multi-task neural network model F. ; S4 00: Minimize as the loss function The goal is to optimize and ultimately obtain the best multi-task neural network model. ; In step S300, the output loss of the multi-task neural network model F is The structural loss of the multi-task neural network model F is Then the loss function ; The output loss of each branch task is The output loss of the multi-task neural network model F is... The output loss of N branch tasks Weighted, structural loss This is the sum of the loss differences between the N branch tasks; that is... ； ； Where α is the branch structure parameter, w is the neural network weight, the size of w is obtained by iteratively from α, and the value of α in the first iteration is... ; Step S400 includes the following specific processes: S410: Perform a structure search on task layer T to obtain the branch task network structure parameters that maintain balance among multiple tasks and the optimal neural network weight parameters; where, for the structure search of task layer T, it is necessary to transform the branch structure of task layer T from a discrete structure into a continuous search space. S420: Minimize the loss function as the optimization objective, and train the parameters in step S410 using the algorithm; S430: Fit the trained parameters to the multi-task neural network model F to obtain the optimal multi-task neural network model determined by automated search. .

2. The visual perception method for assisted driving as described in claim 1, characterized in that: The multi-task neural network model F includes a parameter-sharing layer B and a task layer T; therefore, step S200 includes the following process: S210: Substitute the input X into the parameter sharing layer B; S220: The parameter-sharing layer B extracts the image features of the input X and obtains the feature map M; S230: Substitute the feature map M into the task layer T to obtain the task-specific outputs for multiple branches. .

3. The visual perception method for assisted driving as described in claim 2, characterized in that: The initial structure parameters of task layer T are determined as follows: First, the structure of each branch of task layer T is initialized to obtain the parameters of each branch structure. Subsequently, the parameters of each branch structure were obtained through random combination. Then, in step S230, the output will be... It can be represented as .

4. The visual perception method for assisted driving as described in claim 1, characterized in that: The transformed continuous search space is represented by the following formula: ； In the formula, node X is implicitly represented, and the function... Let O represent the set of candidate operations between nodes. It is a node The weights of candidate operations o in the structure.

5. The visual perception method for assisted driving as described in claim 1, characterized in that: In step S420, by adjusting the output loss and structural loss Perform dual optimization to achieve the loss function The optimization is achieved by minimizing the following objective using gradient descent: ； 。 6. The visual perception method for assisted driving as described in claim 5, characterized in that: The objective is solved using an iterative optimization algorithm to obtain the optimal branch structure parameters α and network weights w. By restoring the continuous structure parameter α to discrete network node connection operations o, a multi-task neural network model with a defined task branch structure and optimal neural network weight performance is obtained. The formula for discretizing the continuously represented neural network structure is as follows: ； 。 7. A system utilizing the visual perception method for assisted driving according to any one of claims 1-6, characterized in that: include: The data input module is used to receive images from multiple datasets and perform preprocessing. A shared layer feature extraction module is used to extract low-level semantic features of an image; A task layer structure search module, which is used to automatically search for the network structure parameters of each task; and Model output module; the model output module is used to output the best model.

Images