Training method, device and terminal equipment of multi-task detection model

Abstract

The present application is applicable to the field of image processing technology and provides a training method, apparatus, terminal device, and computer-readable storage medium for a multi-task detection model, which is applied to a preset multi-task detection model. The multi-task detection model includes a feature extraction network and a multi-task detection network. The method includes: obtaining global feature information and local feature information of a training image through the feature extraction network; inputting the global feature information into the multi-task detection network to obtain a multi-task detection result; obtaining a semantic segmentation result of the training image based on the global feature information and the local feature information; and training the multi-task detection model based on the multi-task detection result and the semantic segmentation result. Through the above method, the computational complexity of multi-task detection can be reduced while improving the accuracy of multi-task detection.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a training method, apparatus, terminal device, and computer-readable storage medium for a multi-task detection model. Background Art

[0002] With the continuous development of artificial intelligence in recent years, deep neural networks have been widely used in the field of computer vision. For example, in the field of autonomous driving, object detection tasks based on deep neural networks can be used to detect objects such as obstacles and signs during driving, and lane detection tasks based on deep neural networks can be used to detect lane lines on the road.

[0003] As described in the example above, multiple detection tasks are required within the same application scenario. Prior art approaches can utilize a multi-task detection network to perform these tasks. Simply inputting a single image to be detected into the multi-task detection network generates detection results for each of the multiple detection tasks. However, existing multi-task detection networks execute each detection task independently, resulting in high computational complexity and an inability to leverage the correlation features between different detection targets, leading to low detection accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, terminal device, and computer-readable storage medium for training a multi-task detection model, which can reduce the computational complexity of multi-task detection while improving the accuracy of multi-task detection.

[0005] In a first aspect, an embodiment of the present application provides a method for training a multi-task detection model, which is applied to a preset multi-task detection model, wherein the multi-task detection model includes a feature extraction network and a multi-task detection network, and the method includes:

[0006] Acquiring global feature information and local feature information of the training image through the feature extraction network;

[0007] Inputting the global feature information into the multi-task detection network to obtain a multi-task detection result;

[0008] Obtaining a semantic segmentation result of the training image according to the global feature information and the local feature information;

[0009] The multi-task detection model is trained according to the multi-task detection result and the semantic segmentation result.

[0010] In the embodiment of the present application, multi-task detection is performed using global feature information, which is equivalent to multiple detection tasks sharing global feature information. Through this method, multiple detection tasks do not need to repeatedly extract feature information, which reduces the amount of calculation, and due to the sharing of feature information, the detection accuracy can be effectively improved. Local feature information is added during semantic segmentation, which is equivalent to considering the detailed features in the image, and then the semantic segmentation results are combined with the multi-task detection results to jointly train the multi-task detection model, which can effectively improve the trained multi-task detection model's ability to recognize different target features and its ability to associate different target features. Through the above method, while reducing the amount of calculation of the multi-task detection model, the detection accuracy of the multi-task detection model is effectively improved.

[0011] In a possible implementation manner of the first aspect, obtaining a semantic segmentation result of the training image according to the global feature information and the local feature information includes:

[0012] Integrating the global feature information and the local feature information to obtain integrated feature information;

[0013] A semantic segmentation result of the training image is obtained according to the integrated feature information.

[0014] In a possible implementation manner of the first aspect, integrating the global feature information and the local feature information to obtain integrated feature information includes:

[0015] Performing upsampling processing on the global feature information to obtain first processed information;

[0016] performing convolution processing on the first processed information to obtain second processed information;

[0017] The second processed information and the local feature information are subjected to information splicing processing to obtain the integrated feature information.

[0018] In a possible implementation of the first aspect, the integrated feature information includes a probability value of each pixel on the training image belonging to each semantic category; the semantic segmentation result includes the semantic category to which each pixel in the training image belongs;

[0019] The obtaining of a semantic segmentation result of the training image according to the integrated feature information includes:

[0020] For each pixel in the training image, obtaining a first target value corresponding to the pixel, where the first target value is the maximum value of the probability values of the pixel belonging to each semantic category;

[0021] The semantic category corresponding to the first target value is determined as the semantic category to which the pixel point belongs.

[0022] In a possible implementation of the first aspect, the multi-task detection network includes a vehicle detection subnetwork, and the multi-task detection result includes a vehicle detection box;

[0023] Inputting the global feature information into the multi-task detection network to obtain a multi-task detection result includes:

[0024] Inputting the global feature information into the vehicle detection subnetwork to obtain multiple sets of detection box information, wherein each set of detection box information includes a center position, a probability value corresponding to the center position, a length value, a width value, and a center offset;

[0025] Generate a vehicle detection frame corresponding to each set of target frame information, wherein the target frame information is the detection frame information corresponding to the second target value, and the second target value is the probability value corresponding to the center position of the multiple sets of detection frame information that meets the first preset threshold.

[0026] In a possible implementation of the first aspect, the multi-task detection network includes a lane line detection subnetwork, and the multi-task detection result includes a lane line point set;

[0027] Inputting the global feature information into the multi-task detection network to obtain a multi-task detection result includes:

[0028] Inputting the global feature information into the lane detection subnetwork to obtain a category matrix corresponding to each group of target pixels, wherein a group of target pixels is a row of pixels in the training image, the number of groups of target pixels is less than the number of rows of pixels in the training image, and the category matrix includes a probability value for each pixel in the target pixels belonging to each lane category;

[0029] Determining a category vector corresponding to each group of target pixels according to the category matrix corresponding to each group of target pixels, wherein the category vector includes the lane line category corresponding to the maximum probability value of each pixel point in the target pixels in the category matrix;

[0030] The lane line point set is generated by using pixel points corresponding to a preset category in each of the category vectors.

[0031] In a possible implementation of the first aspect, the multi-task detection network includes a first detection subnetwork and a second detection subnetwork, and the multi-task detection result includes a first detection result output by the first detection subnetwork and a second detection result output by the second detection subnetwork;

[0032] The training of the multi-task detection model according to the multi-task detection result and the semantic segmentation result includes:

[0033] Calculating a first loss value of the first detection result according to a first loss function;

[0034] Calculating a second loss value of the second detection result according to a second loss function;

[0035] Calculating a third loss value of the semantic segmentation result according to a third loss function;

[0036] The multi-task detection model is trained according to the first loss value, the second loss value, and the third loss value.

[0037] In a second aspect, an embodiment of the present application provides a multi-task detection model training device, which is applied to a preset multi-task detection model, wherein the multi-task detection model includes a feature extraction network and a multi-task detection network, and the device includes:

[0038] A feature extraction unit, configured to obtain global feature information and local feature information of a training image through the feature extraction network;

[0039] an object detection unit, configured to input the global feature information into the multi-task detection network to obtain a multi-task detection result;

[0040] A semantic segmentation unit, configured to obtain a semantic segmentation result of the training image based on the global feature information and the local feature information;

[0041] A model training unit is used to train the multi-task detection model according to the multi-task detection result and the semantic segmentation result.

[0042] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a training method for a multi-task detection model as described in any one of the first aspects above when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. An embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the training method of the multi-task detection model as described in any one of the first aspects above.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the training method of the multi-task detection model described in any one of the above-mentioned first aspects.

[0045] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 is a schematic diagram of a multi-task detection model provided in an embodiment of the present application;

[0048] Figure 2 Schematic diagram of a multi-task detection model training method provided in an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of the model training process provided in an embodiment of the present application;

[0050] Figure 4 This is a structural block diagram of a training device for a multi-task detection model provided in an embodiment of the present application;

[0051] Figure 5 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0053] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0054] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized.

[0056] See also Figure 1 , is a schematic diagram of a multi-task detection model provided in an embodiment of the present application. Figure 1 As shown, the multi-task detection model includes a feature extraction network 11 and a multi-task detection network 12. The multi-task detection network is used to perform multiple detection tasks. The output of the feature extraction network is connected to the input of the multi-task detection network.

[0057] For example, in the automatic line patrol application scenario of autonomous driving, the detection tasks may include vehicle detection and lane line detection, that is, ensuring that the vehicle drives along the lane line. In this case, the multi-task detection network in the multi-task detection model may include a vehicle detection subnetwork and a lane line detection subnetwork. The image to be detected is input into the feature extraction network of the multi-task detection model. The feature extraction network outputs the global feature information of the image to be detected, and the global feature information is respectively input into the vehicle detection subnetwork and the lane line detection subnetwork in the multi-task detection network; the vehicle detection subnetwork detects the target vehicle in the image to be detected based on the global feature information and outputs the detection frame of the target vehicle; the lane line detection subnetwork detects the lane line in the image to be detected based on the global feature information and outputs a lane line point set. The vehicle controller controls the target vehicle to drive along the lane line based on the detection frame of the target vehicle and the lane line point set output by the multi-task detection model.

[0058] In the automatic obstacle avoidance application scenario of autonomous driving, detection tasks can include vehicle detection and obstacle detection, specifically ensuring that the vehicle avoids obstacles. In this case, the multi-task detection network in the multi-task detection model can include a vehicle detection subnetwork and an obstacle detection subnetwork. The image to be detected is input into the feature extraction network of the multi-task detection model. The feature extraction network outputs global feature information of the image to be detected, and this global feature information is input into the vehicle detection subnetwork and obstacle detection subnetwork of the multi-task detection network, respectively. The vehicle detection subnetwork detects the target vehicle in the image to be detected based on the global feature information and outputs a detection frame of the target vehicle. The obstacle detection subnetwork detects obstacles in the image to be detected based on the global feature information and outputs a detection frame of the obstacle. The vehicle controller controls the target vehicle to avoid obstacles based on the detection frames of the target vehicle and obstacle detected by the multi-task detection model.

[0059] It should be noted that the above is only an example of a multi-task detection model and does not specifically limit its structure. In practical applications, a multi-task detection model can be used to process a single detection task or multiple detection tasks. When processing multiple detection tasks, the multi-task detection network can include detection sub-networks corresponding to each of the multiple detection tasks.

[0060] In the application embodiment, global feature information is used for multi-task detection, which is equivalent to multiple detection tasks sharing global feature information. Through this method, multiple detection tasks do not need to repeatedly extract feature information, which reduces the amount of calculation, and due to the sharing of feature information, the detection accuracy can be effectively improved.

[0061] In the process of training a multi-task detection model, the output results of each detection task can be comprehensively considered. This can enhance the multi-task detection model's ability to recognize different detection targets and its ability to associate different target features.

[0062] Typically, in an application scenario, in addition to the features of the target itself, other image features, such as background features and color features, can also provide useful information for target detection. However, the above training method only considers the output results of each detection task, that is, only the feature information of the detected target itself, and ignores other useful information besides the target features.

[0063] The present invention provides a method for training a multi-task detection model. In this method, features other than the detection target feature information in the image are used to assist in training the model to improve the detection accuracy of the model. Figure 2 , is a flowchart of a method for training a multi-task detection model provided in an embodiment of the present application. As an example and not a limitation, the method may include the following steps:

[0064] S201, obtaining global feature information and local feature information of a training image through a feature extraction network.

[0065] In an embodiment of the present application, the feature extraction network may include multiple feature extraction layers. The output of the feature extraction network is global feature information, which may include color features, texture features, and shape features of the image, emphasizing the overall attributes of the image. The correlation between features is high, but the detailed features of the image are lost. The outputs of certain intermediate layers in the feature extraction network are obtained and used as local feature information. Local feature information focuses on the local details of the image and can effectively make up for the image details lost in the global feature information.

[0066] Existing backbone networks such as VGG, ResNet, and MobileNet can be used as feature extraction networks. The choice of feature extraction network can be determined by balancing accuracy with computational speed. If computational accuracy is a priority, a backbone network like ResNet can be chosen, as it uses a deeper network to extract more image features. If computational speed is a priority, a backbone network like MobileNet can be chosen. These lightweight backbone networks are more suitable for embedded devices, improving computational speed.

[0067] S202: Input the global feature information into the multi-task detection network to obtain a multi-task detection result.

[0068] When processing a single detection task, the multi-task detection network can include the detection sub-network corresponding to the single detection task. When processing multiple detection tasks, the multi-task detection network can include the detection sub-networks corresponding to each of the multiple detection tasks. Of course, the multi-task detection network can include multiple detection sub-networks, but in an application, the output results of only a few of the required detection sub-networks can be obtained.

[0069] For example, a multi-task detection network may include a vehicle detection subnetwork, an obstacle detection subnetwork, a lane detection subnetwork, and a pedestrian detection subnetwork. In the automatic line patrol application scenario, the output results of the vehicle detection subnetwork and the lane detection subnetwork can be obtained. In the automatic obstacle avoidance application scenario, the output results of the vehicle detection subnetwork and the obstacle detection subnetwork can be obtained. In the automatic avoidance application scenario, the output results of the vehicle detection subnetwork and the pedestrian detection subnetwork can be obtained.

[0070] like Figure 1 In the automatic line patrol application scenario described in the embodiment, the multi-task detection network includes a vehicle detection sub-network and a lane line detection sub-network. Accordingly, the output results of the multi-task detection network include a vehicle detection box and a lane line point set.

[0071] Optionally, the process of performing a vehicle detection task may include: pre-setting anchor points and anchor frames; obtaining multiple candidate detection frames of the target vehicle in the training image with reference to the anchor points and anchor frames; and then filtering out the target detection frame from the multiple candidate detection frames using a non-maximum suppression method.

[0072] This detection method relies on pre-set anchor points and anchor boxes for accuracy, making it less reliable. Furthermore, it requires filtering the detection boxes using a non-maximum suppression algorithm, which requires significant computation and slows down the algorithm.

[0073] In order to ensure detection accuracy while improving calculation speed, in one embodiment, the process of performing the vehicle detection task may include:

[0074] The global feature information is input into the vehicle detection subnetwork to obtain multiple sets of detection frame information, where each set of detection frame information includes a center position, a probability value corresponding to the center position, a length value, a width value, and a center offset; a vehicle detection frame corresponding to each set of target frame information is generated, where the target frame information is the detection frame information corresponding to the second target value, and the second target value is the probability value corresponding to the center position that meets the first preset threshold in the multiple sets of detection frame information.

[0075] See also Figure 3 , is a schematic diagram of the model training process provided in the embodiment of this application. Figure 3 As shown, the vehicle detection subnetwork may include three branches: heat map detection, length and width detection, and regression detection. The heat map detection branch is used to obtain a heat map of the training image, which includes multiple center positions on the training image that may belong to the center of the detection frame and the probability value corresponding to each center position. The length and width detection is used to obtain the length and width values of the detection frame of the target vehicle in the training image. The regression detection is used to obtain the center offset of the detection frame of the target vehicle in the training image. The output results of the three branches are interrelated, that is, a center position output by the heat map detection branch corresponds to a set of length and width values output by the length and width detection branch, and corresponds to a center offset output by the regression detection branch.

[0076] N center positions whose probability values are greater than a first preset threshold and a set of detection frame information corresponding to each of the N center positions may be selected, and detection frames corresponding to each of the N sets of detection frame information may be generated.

[0077] For any target frame information, the actual center position of the detection frame is determined based on the center position and center offset in the target frame information, and the position and shape of the detection frame are determined based on the actual center position and length and width values. Accordingly, the detection score of the generated detection frame is the probability value of the center position in the target frame information.

[0078] Typically, the lane line detection process involves classifying each pixel in the training image to determine whether each pixel belongs to a lane line and which lane line it belongs to. This method is computationally intensive. To reduce the computational complexity, in one embodiment, the process of performing the lane line detection task may include:

[0079] The global feature information is input into the lane line detection subnetwork to obtain the category matrix corresponding to each group of target pixels. The category vector corresponding to each group of target pixels is determined based on the category matrix corresponding to each group of target pixels. The pixel points corresponding to the preset category in each category vector are used to generate a lane line point set.

[0080] A group of target pixels is a row of pixels in the training image, the number of target pixel groups is less than the number of pixel rows in the training image, the category matrix includes the probability value of each target pixel belonging to each lane line category, and the category vector includes the lane line category corresponding to the maximum probability value of each target pixel in the category matrix.

[0081] The above method is equivalent to classifying based on the row direction of the image. Assume that the training image is an H×W image, where H is the number of rows of pixels in the training image and W is the number of columns of pixels in the training image. H rows of pixels can be sampled from the training image, where h is less than H. This way, only the classification problem of h rows needs to be processed, simplifying the original H×W classification problem to h classification problems, greatly reducing the computational effort.

[0082] Optionally, in these h classification problems, each classification problem is W-dimensional. W pixels can be sampled again from the W-dimensional problem, so that each classification problem is simplified to w-dimensional, further reducing the amount of computation.

[0083] For example, assume the training image is a 10×10 image (H=W=10), and the lane marking categories are I, II, and III. Category I indicates the left lane marking, Category II indicates the right lane marking, and Category III indicates no lane marking. Sample three rows of pixels (h=3), and within each row, sample one pixel every other pixel (w=5).

[0084] The category matrix corresponding to each row of pixels is obtained through the lane line detection subnetwork. The category matrix corresponding to the first row of pixels is The first row of the matrix represents the probability value of each pixel belonging to the left lane line, the second row represents the probability value of each pixel belonging to the right lane line, and the third row represents the probability value of each pixel not belonging to the lane line. Each column from left to right represents the pixel h1. w1 Corresponding probability value, pixel h1 w2 Corresponding probability value, pixel h1 w3 The corresponding probability value, h1w4 Corresponding probability value, pixel h1 w5 The corresponding probability value, hi wj Represents the pixel at row i and column j in the sampled pixels. To observe the maximum probability value of each pixel in the category matrix, we can follow the direction of the columns in the matrix. w1 The maximum probability value in the category matrix is 0.8 (corresponding to category III), and the pixel h1 w2 The maximum probability value in the category matrix is 0.8 (corresponding to category I), and the pixel h1 w3 The maximum probability value in the category matrix is 0.8 (corresponding to category III), and the pixel h1 w4 The maximum probability value in the category matrix is 0.75 (corresponding to category II), and the pixel h1 w5 The maximum probability value in the category matrix is 0.8 (corresponding to category III), so the category vector obtained according to the category matrix is [III I III II III].

[0085] Similarly, assuming that the category matrix corresponding to the second row of pixels is The category vector obtained according to the category matrix is [III I III II III]. Assume that the category matrix corresponding to the second row of pixels is The category vector obtained according to the category matrix is [III I III IIIII]. The pixel points of category I in the above three category vectors are used to generate the point set of the left lane line, that is, the point set of the left lane line includes pixel h1 w2 Pixel h2 w2 and pixel h3 w2 ; Generate the point set of the right lane line from the pixel points of category II in the above three category vectors, that is, the point set of the right lane line includes pixel h1 w4 , pixel h2 w4 and pixel h3 w4 The other pixels represent the background.

[0086] It should be noted that the above is only an example of lane detection and does not impose specific restrictions on the number of sampled rows h and columns w. Of course, larger h and w indicate denser sampling, requiring more data processing, and higher detection accuracy. Smaller h and w indicate sparser sampling, requiring less data processing, and lower detection accuracy. The values of h and w can be determined based on actual needs.

[0087] Lane line detection uses the global feature information of the image. That is, when detecting the position of the lane line in a certain row, the receptive field is the size of the entire image. Therefore, no complex information transmission mechanism is required to achieve good results, effectively solving the problem of inaccurate lane line detection due to a small receptive field.

[0088] S203: Obtain a semantic segmentation result of the training image according to the global feature information and the local feature information.

[0089] In one embodiment, S203 may include the following steps:

[0090] The global feature information and the local feature information are integrated to obtain integrated feature information; and the semantic segmentation result of the training image is obtained based on the integrated feature information.

[0091] The semantic segmentation process uses both global feature information and local feature information, which is equivalent to considering both the overall attributes of the image such as color, texture and shape, as well as the detailed attributes of the image, which is conducive to improving the accuracy of the semantic segmentation results.

[0092] Optional, such as Figure 3 As shown, the integration process may include:

[0093] The global feature information is upsampled to obtain first processed information; the first processed information is convolved to obtain second processed information; the second processed information and the local feature information are concatenated to obtain integrated feature information.

[0094] Furthermore, after the information concatenation process, convolution and upsampling operations can be performed to obtain a set of multidimensional images. The number of channels in this multidimensional image corresponds to the number of semantic categories. Accordingly, this set of multidimensional images is the integrated feature information, which includes the probability value of each pixel in the training image belonging to each semantic category.

[0095] Optionally, the semantic segmentation process may include:

[0096] For each pixel in the training image, a first target value corresponding to the pixel is obtained, where the first target value is the maximum value of the probability values of the pixel belonging to each semantic category; and the semantic category corresponding to the first target value is determined as the semantic category to which the pixel belongs.

[0097] For example, assume that there are four semantic categories: A, B, C, and D. A represents background, B represents vehicle, C represents lane, and D represents road surface but not lane. mn(The pixel at row m and column n) has probability values corresponding to the four categories of 0.1, 0.2, 0.65, and 0.05. Among them, the largest probability value of 0.65 corresponds to semantic category C, indicating that pixel C mn The semantic category of is C, which belongs to the lane line.

[0098] According to the above method, the semantic category to which each pixel in the training image belongs can be obtained. Accordingly, the semantic segmentation result includes the semantic category to which each pixel in the training image belongs.

[0099] S204: training a multi-task detection model based on the multi-task detection results and the semantic segmentation results.

[0100] Taking the multi-task detection model performing two detection tasks as an example, in one embodiment, the multi-task detection network includes a first detection sub-network and a second detection sub-network. Accordingly, the multi-task detection result includes a first detection result output by the first detection sub-network and a second detection result output by the second detection sub-network. Optionally, the training process in S204 includes:

[0101] A first loss value of the first detection result is calculated according to the first loss function; a second loss value of the second detection result is calculated according to the second loss function; a third loss value of the semantic segmentation result is calculated according to the third loss function; and a multi-task detection model is trained according to the first loss value, the second loss value and the third loss value.

[0102] like Figure 1 In the automatic line patrol application scenario described in the embodiment, the multi-task detection network includes a vehicle detection sub-network and a lane line detection sub-network. Accordingly, the output results of the multi-task detection network include a vehicle detection frame and a lane line point set. The training process of the model is: obtain a training image set, which includes multiple training images, and input each training image into the multi-task detection model to obtain the vehicle detection frame and lane line point set of each training image. For each training image, the first loss value corresponding to the vehicle detection frame is calculated according to the first loss function; the second loss value corresponding to the lane line point set is calculated according to the second loss function; the third loss value of the semantic segmentation result is calculated according to the third loss function; the first loss value, the second loss value and the third loss value are weighted and summed to obtain the total loss value; the total loss value is fed back to the feature extraction network of the multi-task detection model to update the network parameters of the feature extraction network. After that, the next training image is input into the multi-task detection model, and the updated feature extraction network is used for feature extraction.

[0103] The first loss function, the second loss function, and the third loss function can use the same loss function, or different loss functions can be selected based on the characteristics of each sub-network. Commonly used loss functions include cross entropy loss function and logarithmic loss function.

[0104] In another embodiment, a gradient descent method may be used to train the multi-task detection model based on the multi-task detection results and the semantic segmentation results.

[0105] In the embodiments of the present application, local feature information is incorporated into semantic segmentation, which is equivalent to taking into account the detailed features in the image. The semantic segmentation results are then combined with the multi-task detection results to jointly train a multi-task detection model, which can effectively improve the trained multi-task detection model's ability to identify and associate different target features. Through this method, while reducing the computational complexity of the multi-task detection model, the detection accuracy of the multi-task detection model is effectively improved.

[0106] It should be noted that the method of the embodiment of the present application only uses semantic segmentation as an auxiliary in the training phase, and the semantic segmentation task will be deleted in the detection phase. In this way, the detection accuracy of the trained multi-task detection model is guaranteed, and the running speed of the detection phase is not affected.

[0107] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0108] Corresponding to the training method of the multi-task detection model described in the above embodiment, Figure 4 This is a structural block diagram of the training device for the multi-task detection model provided in an embodiment of the present application. For the sake of convenience, only the parts related to the embodiment of the present application are shown.

[0109] Reference Figure 4 , the device comprises:

[0110] The feature extraction unit 41 is used to obtain global feature information and local feature information of the training image through the feature extraction network.

[0111] The target detection unit 42 is configured to input the global feature information into the multi-task detection network to obtain a multi-task detection result.

[0112] The semantic segmentation unit 43 is configured to obtain a semantic segmentation result of the training image according to the global feature information and the local feature information.

[0113] The model training unit 44 is configured to train the multi-task detection model according to the multi-task detection result and the semantic segmentation result.

[0114] Optionally, the semantic segmentation unit 43 is further configured to:

[0115] The global feature information and the local feature information are integrated to obtain integrated feature information; and a semantic segmentation result of the training image is obtained according to the integrated feature information.

[0116] Optionally, the semantic segmentation unit 43 is further configured to:

[0117] The global feature information is upsampled to obtain first processed information; the first processed information is convolved to obtain second processed information; and the second processed information and the local feature information are concatenated to obtain the integrated feature information.

[0118] Optionally, the integrated feature information includes a probability value of each pixel point on the training image belonging to each semantic category; and the semantic segmentation result includes the semantic category to which each pixel point in the training image belongs.

[0119] Accordingly, the semantic segmentation unit 43 is further configured to:

[0120] For each pixel point in the training image, a first target value corresponding to the pixel point is obtained, where the first target value is the maximum value among the probability values of the pixel point belonging to each semantic category; and the semantic category corresponding to the first target value is determined as the semantic category to which the pixel point belongs.

[0121] Optionally, the multi-task detection network includes a vehicle detection subnetwork, and the multi-task detection result includes a vehicle detection box.

[0122] Accordingly, the target detection unit 42 is further configured to:

[0123] Inputting the global feature information into the vehicle detection subnetwork to obtain multiple sets of detection box information, wherein each set of detection box information includes a center position, a probability value corresponding to the center position, a length value, a width value, and a center offset;

[0124] Generate a vehicle detection frame corresponding to each set of target frame information, wherein the target frame information is the detection frame information corresponding to the second target value, and the second target value is the probability value corresponding to the center position of the multiple sets of detection frame information that meets the first preset threshold.

[0125] Optionally, the multi-task detection network includes a lane line detection subnetwork, and the multi-task detection result includes a lane line point set.

[0126] Accordingly, the target detection unit 42 is further configured to:

[0127] Inputting the global feature information into the lane detection subnetwork to obtain a category matrix corresponding to each group of target pixels, wherein a group of target pixels is a row of pixels in the training image, the number of groups of target pixels is less than the number of rows of pixels in the training image, and the category matrix includes a probability value for each pixel in the target pixels belonging to each lane category;

[0128] Determining a category vector corresponding to each group of target pixels according to the category matrix corresponding to each group of target pixels, wherein the category vector includes the lane line category corresponding to the maximum probability value of each pixel point in the target pixels in the category matrix;

[0129] The lane line point set is generated by using pixel points corresponding to a preset category in each of the category vectors.

[0130] Optionally, the multi-task detection network includes a first detection sub-network and a second detection sub-network, and the multi-task detection result includes a first detection result output by the first detection sub-network and a second detection result output by the second detection sub-network.

[0131] Accordingly, the model training unit 44 is further configured to:

[0132] Calculate a first loss value of the first detection result according to a first loss function; calculate a second loss value of the second detection result according to a second loss function; calculate a third loss value of the semantic segmentation result according to a third loss function; and train the multi-task detection model according to the first loss value, the second loss value, and the third loss value.

[0133] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0134] in addition, Figure 4 The training device of the multi-task detection model shown can be a software unit, a hardware unit, or a combination of software and hardware units built into an existing terminal device, or can be integrated into the terminal device as an independent pendant, or can exist as an independent terminal device.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0136] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Figure 5 As shown, the terminal device 5 of this embodiment includes: at least one processor 50 ( Figure 5 Only one is shown in the figure) a processor, a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50. When the processor 50 executes the computer program 52, the steps in the embodiment of the training method of any of the above-mentioned multi-task detection models are implemented.

[0137] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 5 It is only an example of the terminal device 5 and does not constitute a limitation on the terminal device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0138] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0139] In some embodiments, the memory 51 may be an internal storage unit of the terminal device 5, such as a hard disk or memory of the terminal device 5. In other embodiments, the memory 51 may also be an external storage device of the terminal device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 5. Furthermore, the memory 51 may also include both an internal storage unit of the terminal device 5 and an external storage device. The memory 51 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been output or is about to be output.

[0140] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0141] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable medium cannot be an electric carrier signal or a telecommunication signal.

[0143] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0144] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0147] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A training method for a multi-task detection model, characterized in that: Applied to a preset multi-task detection model, the multi-task detection model includes a feature extraction network and a multi-task detection network, the method includes: Acquiring global feature information and local feature information of the training image through the feature extraction network; Inputting the global feature information into the multi-task detection network to obtain a multi-task detection result; wherein the multi-task detection network includes a vehicle detection subnetwork and a lane line detection subnetwork, and the multi-task detection result includes a vehicle detection box and a lane line point set; wherein inputting the global feature information into the multi-task detection network to obtain the multi-task detection result includes: inputting the global feature information into the lane line detection subnetwork to obtain a category matrix corresponding to each group of target pixels, wherein a group of target pixels is a row of pixels in the training image, the number of groups of target pixels is less than the number of rows of pixels in the training image, and the category matrix includes a probability value of each pixel in the target pixels belonging to each lane line category; determining a category vector corresponding to each group of target pixels according to the category matrix corresponding to each group of target pixels, wherein the category vector includes the lane line category corresponding to the maximum probability value of each pixel in the target pixel in the category matrix; generating the lane line point set from the pixels corresponding to the preset category in each category vector; Obtaining a semantic segmentation result of the training image according to the global feature information and the local feature information; The multi-task detection model is trained according to the multi-task detection results and the semantic segmentation results; wherein the trained task detection model is used to detect the vehicle detection frame and lane line point set in the image to be processed.

2. The training method of the multi-task detection model according to claim 1, characterized in that The obtaining of a semantic segmentation result of the training image according to the global feature information and the local feature information includes: Integrating the global feature information and the local feature information to obtain integrated feature information; A semantic segmentation result of the training image is obtained according to the integrated feature information.

3. The training method of the multi-task detection model according to claim 2, characterized in that: The integrating the global feature information and the local feature information to obtain integrated feature information includes: Performing upsampling processing on the global feature information to obtain first processed information; performing convolution processing on the first processed information to obtain second processed information; The second processed information and the local feature information are subjected to information splicing processing to obtain the integrated feature information.

4. The training method of the multi-task detection model according to claim 2, wherein: The integrated feature information includes a probability value of each pixel point on the training image belonging to each semantic category; the semantic segmentation result includes the semantic category to which each pixel point in the training image belongs; The obtaining of a semantic segmentation result of the training image according to the integrated feature information includes: For each pixel in the training image, obtaining a first target value corresponding to the pixel, where the first target value is the maximum value of the probability values of the pixel belonging to each semantic category; The semantic category corresponding to the first target value is determined as the semantic category to which the pixel point belongs.

5. The training method of the multi-task detection model according to claim 1, wherein: Inputting the global feature information into the multi-task detection network to obtain a multi-task detection result includes: Inputting the global feature information into the vehicle detection subnetwork to obtain multiple sets of detection box information, wherein each set of detection box information includes a center position, a probability value corresponding to the center position, a length value, a width value, and a center offset; Generate a vehicle detection frame corresponding to each set of target frame information, wherein the target frame information is the detection frame information corresponding to the second target value, and the second target value is the probability value corresponding to the center position of the multiple sets of detection frame information that meets the first preset threshold.

6. The method for training a multi-task detection model according to claim 1, wherein: The multi-task detection network includes a first detection sub-network and a second detection sub-network, and the multi-task detection result includes a first detection result output by the first detection sub-network and a second detection result output by the second detection sub-network; The training of the multi-task detection model according to the multi-task detection result and the semantic segmentation result includes: Calculating a first loss value of the first detection result according to a first loss function; Calculating a second loss value of the second detection result according to a second loss function; Calculating a third loss value of the semantic segmentation result according to a third loss function; The multi-task detection model is trained according to the first loss value, the second loss value, and the third loss value.

7. A training device for a multi-task detection model, characterized in that: Applied to a preset multi-task detection model, the multi-task detection model includes a feature extraction network and a multi-task detection network, and the device includes: A feature extraction unit, configured to obtain global feature information and local feature information of a training image through the feature extraction network; A target detection unit is configured to input the global feature information into the multi-task detection network to obtain a multi-task detection result; wherein the multi-task detection network includes a vehicle detection subnetwork and a lane line detection subnetwork, and the multi-task detection result includes a vehicle detection box and a lane line point set; the target detection unit is further configured to input the global feature information into the lane line detection subnetwork to obtain a category matrix corresponding to each group of target pixels, wherein a group of target pixels is a row of pixels in the training image, the number of groups of target pixels is less than the number of rows of pixels in the training image, and the category matrix includes a probability value of each pixel in the target pixels belonging to each lane line category; a category vector corresponding to each group of target pixels is determined based on the category matrix corresponding to each group of target pixels, wherein the category vector includes the lane line category corresponding to the maximum probability value of each pixel in the target pixels in the category matrix; and the lane line point set is generated by pixel points corresponding to the preset category in each category vector; A semantic segmentation unit, configured to obtain a semantic segmentation result of the training image based on the global feature information and the local feature information; A model training unit is used to train the multi-task detection model based on the multi-task detection results and the semantic segmentation results, wherein the trained task detection model is used to detect vehicle detection frames and lane line point sets in the image to be processed.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images