Multi-task target detection method, electronic device, medium and vehicle

Abstract

The present invention relates to the field of autonomous driving technology, and specifically provides a multi-task target detection method, electronic equipment, medium and vehicle, aiming to solve the technical problem that the existing multi-task detection method has low detection accuracy or detection effect. To this end, the multi-task target detection method of the present invention includes: acquiring an image collected by a vehicle-mounted sensor; inputting the image into a multi-scale feature extraction network to extract multi-scale features; inputting the multi-scale features into a multi-scale feature fusion network to obtain fusion features, wherein the multi-scale feature fusion network includes multiple optimal fusion paths, each optimal fusion path corresponds to one of the multiple tasks; the fusion features output from each optimal fusion path are respectively input into the corresponding detection head to obtain a detection result, wherein each detection head can detect one of the multiple tasks. In this way, the detection accuracy of multi-task targets is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and specifically provides a multi-task target detection method, electronic equipment, medium and vehicle. Background Art

[0002] At present, multi-task detection in traditional vehicle autonomous driving is usually implemented using a neural network model. However, when multiple tasks share a neural network, the detection effect will be significantly reduced. Some existing technologies divide a backbone network independently for each task, which results in low detection efficiency for multi-task targets and detection accuracy that is difficult to meet actual needs.

[0003] Accordingly, the field needs a new multi-task target detection solution to solve the above problems. Summary of the invention

[0004] In order to overcome the above defects, the present invention is proposed to solve or at least partially solve the technical problem that the existing multi-task detection method has low detection accuracy or detection effect. The present invention provides a multi-task target detection method, electronic equipment, medium and vehicle.

[0005] In a first aspect, the present invention provides a multi-task target detection method, comprising: acquiring an image captured by a vehicle-mounted sensor; inputting the image into a multi-scale feature extraction network to extract multi-scale features; inputting the multi-scale features into a multi-scale feature fusion network to obtain fused features, wherein the multi-scale feature fusion network includes multiple optimal fusion paths, each optimal fusion path corresponds to one of the multiple tasks; inputting the fused features output from each optimal fusion path into a corresponding detection head to obtain a detection result, wherein each detection head can detect one of the multiple tasks.

[0006] In one embodiment, it also includes: constructing a target detection network consisting of a multi-scale feature extraction network, a feature pyramid unit and a plurality of detection heads, wherein the feature pyramid unit includes N feature pyramid networks, each feature pyramid network includes M feature extraction layers, wherein the connection from the mth layer to the m+1th layer of the nth feature pyramid network is to perform an upsampling operation on the features, and the connection between the mth layer of the nth feature pyramid network and the mth layer of the n+1th feature pyramid network is to perform a convolution operation with a convolution kernel of 1*1 on the features, wherein N≥2, M is consistent with the number of feature extraction layers in the multi-scale feature extraction network, 1≤n≤N-1, 1≤m≤M-1; training the target detection network, obtaining the multiple optimal fusion paths from the paths between all feature extraction layers of the N feature pyramid networks, wherein the multiple optimal fusion paths constitute the multi-scale feature fusion network.

[0007] In one embodiment, the target detection network is trained to obtain the multiple optimal fusion paths from the paths between all feature extraction layers of the N feature pyramid networks, including: inputting a training set into the target detection network, traversing the paths between all feature extraction layers of the N feature pyramid networks, and obtaining parameters of the target detection network; and inputting a verification set into the target detection network for verification to obtain the multiple optimal fusion paths.

[0008] In one embodiment, the step of inputting the training set into the target detection network, traversing the paths between all feature extraction layers of the N feature pyramid networks, and obtaining the parameters of the target detection network includes: selecting one path from the possible paths corresponding to each of the K tasks to form K paths for single training, where K is the total number of tasks of the multiple tasks; and traversing the K paths composed of all paths in the possible paths corresponding to each task and repeating the single training.

[0009] In one embodiment, the step of inputting the verification set into the target detection network for verification to obtain the multiple optimal fusion paths includes: taking the path corresponding to the minimum distance between the test result obtained according to the verification set and the true label as the optimal fusion path.

[0010] In one embodiment, M=4.

[0011] In one embodiment, the multi-task target detection includes at least two of object detection, drivable area segmentation and lane detection.

[0012] In a second aspect, an electronic device is provided, comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, and the program codes are suitable for being loaded and run by the processor to execute any of the multi-task target detection methods described above.

[0013] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute any of the multi-task target detection methods described above.

[0014] In a fourth aspect, a vehicle is provided, characterized in that it includes the aforementioned electronic device.

[0015] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0016] The present invention provides a multi-task target detection method, firstly, an image collected by a vehicle-mounted sensor is obtained, then the image is input into a multi-scale feature extraction network to extract multi-scale features, then the multi-scale features are input into a multi-scale feature fusion network to obtain fusion features, and finally the fusion features output from each optimal fusion path are respectively input into a corresponding detection head to obtain a detection result. In this way, accurate detection of multi-task targets is achieved through the optimal fusion path corresponding to each task in the multi-task, which improves the detection accuracy and the detection efficiency at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which:

[0018] Figure 1 is a schematic flow chart of main steps of a multi-task target detection method according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of the structure of a target detection network according to an embodiment of the present invention;

[0020] Figure 3 is a path corresponding to a feature pyramid network according to an embodiment of the present invention;

[0021] Figure 4a is a path corresponding to a feature pyramid network according to an embodiment of the present invention;

[0022] Figure 4b is a path corresponding to a feature pyramid network according to an embodiment of the present invention;

[0023] Figure 4c is a path corresponding to a feature pyramid network according to an embodiment of the present invention;

[0024] Figure 4d is a path corresponding to a feature pyramid network according to an embodiment of the present invention;

[0025] Figure 5 are three optimal fusion paths corresponding to three types of tasks according to an embodiment of the present invention;

[0026] Figure 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0028] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program codes, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0029] At present, multi-task detection in traditional vehicle autonomous driving is usually implemented using a neural network model, but when the detection of multiple tasks shares a neural network, the detection effect will be significantly reduced. Some of the prior art is to independently divide a backbone network for each task, which results in low detection efficiency of multi-task targets and difficulty in meeting actual needs in detection accuracy. To this end, the present application proposes a multi-task target detection method, electronic equipment, medium and vehicle, firstly obtaining an image collected by an on-board sensor, then inputting the image into a multi-scale feature extraction network to extract multi-scale features, then inputting the multi-scale features into a multi-scale feature fusion network to obtain fusion features, and finally inputting the fusion features output from each optimal fusion path into the corresponding detection head to obtain the detection result. In this way, accurate detection of multi-task targets is achieved through the optimal fusion path corresponding to each task in the multi-task, which improves detection accuracy and detection efficiency.

[0030] See attached Figure 1 , Figure 1 1 is a flowchart of the main steps of a multi-task target detection method according to an embodiment of the present invention.

[0031] In a specific embodiment, the multi-task target detection in the present application includes at least two of object detection, drivable area segmentation and lane detection. Among them, object detection can be detection of obstacles such as vehicles, mainly marking the location and category of obstacles or vehicles in the form of detection frames. Lane line detection is to detect key points related to lane lines, and then connect all key points to obtain lane lines.

[0032] like Figure 1 As shown, the multi-task target detection method in the embodiment of the present invention mainly includes the following steps S101 to S104.

[0033] Step S101: Acquire an image collected by a vehicle-mounted sensor.

[0034] The vehicle-mounted sensor may specifically be a vehicle-mounted camera, Lidar, Radar, etc.

[0035] Step S102: Input the image into a multi-scale feature extraction network to extract multi-scale features. The multi-scale feature extraction network can extract features of different scales from different layers of the network. Specifically, the multi-scale feature extraction network here can be a neural network such as Resnet, VGG, etc., which can extract multi-scale features of the input image, but is not limited to this, and can also be other neural networks that can extract multi-scale features of the input image.

[0036] Step S103: inputting the multi-scale features into a multi-scale feature fusion network to obtain fused features, wherein the multi-scale feature fusion network includes multiple optimal fusion paths, and each optimal fusion path corresponds to one of the multiple tasks.

[0037] Specifically, the multi-scale feature fusion network is obtained by training the constructed object detection network.

[0038] In a specific embodiment, the object detection network is composed of a multi-scale feature extraction network, a feature pyramid unit, and multiple detection heads.

[0039] The feature pyramid unit includes N feature pyramid networks (FPN), each feature pyramid network includes M feature extraction layers, wherein the connection from the mth layer to the m+1th layer of the nth feature pyramid network is to upsample the features, and the connection from the mth layer of the nth feature pyramid network to the mth layer of the n+1th feature pyramid network is to perform a convolution operation with a convolution kernel of 1*1 on the features, wherein N≥2, M is consistent with the number of feature extraction layers in the multi-scale feature extraction network, 1≤n≤N-1, 1≤m≤M-1.

[0040] The number of detection heads is the same as the type of tasks, and each detection head can be, for example, a region candidate network (RPN), YOLO, SSD, etc., which is not limited in this application. For example, for the three tasks of object detection, drivable area segmentation, and lane detection, three detection heads are set. After training, each detection head outputs a result corresponding to one of the three tasks.

[0041] exist Figure 2 In the example shown, the feature pyramid unit includes 5 feature pyramid networks (5 columns in the figure), each of which has 4 feature extraction layers (4 rows in the figure, nodes represented by circles). Those skilled in the art will appreciate that the number of feature extraction layers in each feature pyramid network depends on the specific number of extraction layers in the previous multi-scale feature extraction network.

[0042] In addition, those skilled in the art can understand that the connection between any two feature extraction layers in each feature pyramid network (indicated by the vertical arrow) is an operation of upsampling the features output by the previous layer (for example, 2x upsampling), and the connection between adjacent feature pyramid networks (indicated by the horizontal arrow) is a convolution operation with a convolution kernel of 1*1 on the features output by the previous layer.

[0043] It should be noted that Figure 2 The feature extraction layers of the feature pyramid illustrated in the example are arranged in columns, but the present invention is not limited thereto and may also be arranged in rows.

[0044] After building the target detection network, the target detection network is trained to obtain multiple optimal fusion paths from the paths between all feature extraction layers of the feature pyramid network of the target detection network, thereby forming a multi-scale feature fusion network composed of multiple optimal fusion paths to extract fusion features.

[0045] For example, in Figure 2 In , for the features input from the first feature extraction layer of the first feature pyramid network and output from the fourth feature extraction layer of the first feature pyramid network, there is one possible path, namely Figure 3 The path shown.

[0046] For example, in Figure 2 In , there are four possible paths for the features input from the first feature extraction layer of the first feature pyramid network and output from the fourth feature extraction layer of the second feature pyramid network, such as Figures 4a-4d shown.

[0047] The first path is the path from the first feature extraction layer of the first feature pyramid network to the first feature extraction layer to the fourth feature extraction layer of the second feature pyramid network ( Figure 4a).

[0048] The second path is the path from the first feature extraction layer to the second feature extraction layer of the first feature pyramid network and from the second feature extraction layer to the fourth feature extraction layer of the second pyramid network ( Figure 4b ).

[0049] The third path is the path from the first feature extraction layer to the third feature extraction layer of the first feature pyramid and from the third feature extraction layer to the fourth feature extraction layer of the second feature pyramid ( Figure 4c ).

[0050] The fourth path is the path from the first feature extraction layer to the fourth feature extraction layer of the first feature pyramid and the fourth feature extraction layer of the second feature pyramid ( Figure 4d ).

[0051] During training, the training data in this application can be divided into a training set and a validation set according to a preset ratio. For example, the preset ratio here can be 8:2 or 7:3, and there is no limitation on this. Among them, the training set is used to train the target detection network, specifically to traverse the paths between all feature extraction layers of N feature pyramid networks to optimize the parameters of the target detection network. The validation set is mainly used to verify the trained target detection network, so as to extract multiple optimal fusion paths corresponding to multiple tasks.

[0052] In a specific implementation, a training set is input into a target detection network, and paths between all feature extraction layers of N feature pyramid networks are traversed to obtain parameters of the target detection network, including: selecting one path from possible paths corresponding to each task in K tasks (corresponding to K detection heads) to form K paths for single training, where K is the total number of tasks of the multi-task; and traversing the K paths composed of all paths in the possible paths corresponding to each task and repeating the single training.

[0053] In the process of training the target detection network through the training set to optimize the parameters of the target detection network, for any one of the K tasks, specifically, a path is randomly selected from all possible paths corresponding to the task, so that K paths are selected corresponding to the K tasks, and a single training is performed based on the K paths. After the training, one path is continuously selected from the possible paths corresponding to each of the K tasks to form K paths for a single training, and the K paths composed of all paths in the possible paths corresponding to each task are traversed to repeat the single training until all paths corresponding to each task are trained.

[0054] For example, there are three types of tasks, such as object detection, drivable area segmentation, and lane detection tasks. Assuming that there are five possible paths for each task, the total number of paths to be traversed is 5. 3 .

[0055] In the specific code implementation, three for loop statements can be used. For example, the first task selects the first path, the second task selects the second path, and traverses the 5 paths of the third task. Then, the first task selects the first path, the second task selects the second path, and traverses the 5 paths of the third task until the second task is traversed. Then the first task selects the second path, the second task selects a path, and traverses the third task, and so on. This continues until all paths are trained.

[0056] Next, the validation set is input into the target detection network for validation, and multiple optimal fusion paths are obtained. Thus, the multiple optimal fusion paths constitute a multi-scale feature fusion network used in actual multi-task target detection. Those skilled in the art will understand that the so-called optimal fusion path is the optimal combination of a series of operations on the input features for the corresponding task (for example, several 1*1 convolutions and several upsamplings, etc.).

[0057] In a specific example, the path corresponding to the minimum distance between the test result obtained from the validation set and the true label is used as the best fusion path. The true label here is the label corresponding to the validation set.

[0058] Specifically, when the best fusion path is obtained through the validation set, the validation set can be input into the target detection network, each path corresponds to a test result, each test result is compared with the true label, and finally the path corresponding to the minimum distance between the test result obtained by the validation set and the true label is taken as the best fusion path.

[0059] like Figure 5 As shown in the figure, three optimal fusion paths are obtained for the three types of tasks, that is, for task 1, such as the object detection task, the best path is to follow the route shown by the solid line in the figure; for task 2, such as the drivable area segmentation task, the best path is to follow the route shown by the dotted line in the figure; and for task 3, such as the lane line detection task, the best path is to follow the route shown by the dotted line in the figure, thus obtaining the multi-scale feature fusion network used for the target detection of these three tasks.

[0060] The above focuses on describing the process of training the feature pyramid unit. Those skilled in the art can understand that when using the training set to train the entire object detection network, the multi-scale feature extraction network and the detection head are also trained, which will not be repeated here.

[0061] By inputting the training set into the target detection network, traversing the paths between all feature extraction layers of N feature pyramid networks, obtaining the parameters of the target detection network, and inputting the verification set into the target detection network for verification, multiple optimal fusion paths are obtained. The multiple optimal fusion paths form a multi-scale feature fusion network to extract fusion features, which solves the technical problem in the prior art that the detection effect will be significantly reduced when the detection of multiple tasks shares a neural network (for example, only one feature pyramid network), and at the same time improves the detection efficiency.

[0062] By constructing a target detection network, the target detection network includes a multi-scale feature extraction network, a feature pyramid unit, and multiple detection heads, and by training the target detection network, multiple optimal fusion paths are obtained from the paths between all feature extraction layers of N feature pyramid networks, providing basic support for the subsequent detection of multi-task targets. In addition, compared with the prior art of directly using a multi-task detection network containing a single feature pyramid network for feature extraction, it is beneficial to improve the detection accuracy and efficiency of multi-task targets.

[0063] Step S104: Input the fusion features output from each optimal fusion path into the corresponding detection head to obtain the detection results.

[0064] like Figure 5 As shown, the fused features are output from the three best fusion paths respectively, and enter the task 1 detection head (e.g., object detection task detection head), the task 2 detection head (e.g., drivable area segmentation task detection head), and the task 3 detection head (lane detection task detection head) respectively, so that the object detection task detection head outputs the object detection result (category and coordinate information), the drivable area segmentation task detection head outputs the drivable area detection result (category and coordinate information), and the lane detection task detection head outputs the lane detection result (category and coordinate information).

[0065] Based on the above steps S101 to S104, firstly, the image collected by the vehicle sensor is obtained, then the image is input into the multi-scale feature extraction network to extract the multi-scale features, then the multi-scale features are input into the multi-scale feature fusion network to obtain the fusion features, and finally the fusion features output from each optimal fusion path are respectively input into the corresponding detection head to obtain the detection results. In this way, when there is multi-task target detection, the optimal fusion path corresponding to each task can be used to extract the detection result corresponding to the task, thereby improving the detection accuracy of each task. In addition, compared with the simultaneous detection of multiple tasks through a single feature pyramid network, the simultaneous detection of multiple tasks by multiple optimal fusion paths is conducive to improving the detection efficiency.

[0066] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0067] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0068] Furthermore, the present invention also provides an electronic device. In an electronic device embodiment according to the present invention, Figure 6 As shown, the electronic device includes a processor 30 and a storage device 31, the storage device can be configured to store a program for executing the multi-task target detection method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the multi-task target detection method of the above method embodiment. For ease of explanation, only the part related to the embodiment of the present invention is shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention.

[0069] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the multi-task target detection method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned multi-task target detection method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0070] Furthermore, the present invention also provides a vehicle, which includes the aforementioned electronic device.

[0071] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A multi-task target detection method, characterized in that: include: Acquire images collected by vehicle-mounted sensors; Inputting the image into a multi-scale feature extraction network to extract multi-scale features; Inputting the multi-scale features into a multi-scale feature fusion network to obtain fused features, wherein the multi-scale feature fusion network includes multiple optimal fusion paths, each optimal fusion path corresponds to one of the multiple tasks; Inputting the fusion features output from each optimal fusion path into the corresponding detection head respectively to obtain a detection result, wherein each detection head can detect one task among the multiple tasks; Also includes: Constructing an object detection network consisting of a multi-scale feature extraction network, a feature pyramid unit, and a plurality of detection heads, wherein the feature pyramid unit includes N feature pyramid networks, each feature pyramid network includes M feature extraction layers, where N ≥ 2, and M is consistent with the number of feature extraction layers in the multi-scale feature extraction network; The object detection network is trained to obtain the multiple optimal fusion paths from the paths between all feature extraction layers of the N feature pyramid networks, wherein the multiple optimal fusion paths constitute the multi-scale feature fusion network.

2. The multi-task target detection method according to claim 1, characterized in that: The connection from the mth layer to the m+1th layer of the nth feature pyramid network is to upsample the features, and the connection from the mth layer of the nth feature pyramid network to the mth layer of the n+1th feature pyramid network is to perform a convolution operation with a convolution kernel of 1*1 on the features, where 1≤n≤N-1, 1≤m≤M-1.

3. The multi-task target detection method according to claim 2, characterized in that: The training of the object detection network to obtain the multiple optimal fusion paths from the paths between all feature extraction layers of the N feature pyramid networks includes: Inputting the training set into the target detection network, traversing the paths between all feature extraction layers of the N feature pyramid networks, and obtaining the parameters of the target detection network; The verification set is input into the target detection network for verification to obtain the multiple optimal fusion paths.

4. The multi-task target detection method according to claim 3, characterized in that: The step of inputting the training set into the target detection network, traversing the paths between all feature extraction layers of the N feature pyramid networks, and obtaining the parameters of the target detection network includes: Select one path from the possible paths corresponding to each task in the K tasks to form K paths for single training, where K is the total number of tasks in the multi-task; The single training is repeated by traversing K paths consisting of all possible paths corresponding to each task.

5. The multi-task target detection method according to claim 3 or 4, characterized in that: The step of inputting the verification set into the target detection network for verification to obtain the multiple optimal fusion paths includes: The path corresponding to the minimum distance between the test result obtained according to the verification set and the true label is taken as the optimal fusion path.

6. The multi-task target detection method according to claim 2, characterized in that: The M=4.

7. The multi-task target detection method according to claim 1, characterized in that: The multi-task target detection includes at least two of object detection, drivable area segmentation and lane detection.

8. An electronic device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the multi-task target detection method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the multi-task target detection method according to any one of claims 1 to 7.

10. A vehicle, characterized in that: The electronic device comprising the electronic device described in claim 8.

Images