Multi-task detection model training method and device, equipment and storage medium

Abstract

The invention relates to a multi-task detection model training method and device, equipment and a storage medium. The method comprises the steps of performing polling training on an initial multi-task detection model based on a defect sample data set of each training task; inputting the target defect sample data set of the target training task into the backbone network for feature extraction to obtain target data features; predicting the target data features through a detection sub-network corresponding to the target training task to obtain a target detection result; determining training loss based on the target detection result and the corresponding standard detection result, and determining target training loss based on the training loss of each training task; determining a convergence rate based on the target detection result, and determining a learning rate based on the convergence rate of each training task; and training the initial multi-task detection model based on the target training loss and the learning rate to obtain a target multi-task detection model. By adopting the method and the device, the defect detection accuracy of the multi-task detection model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of defect detection, in particular to a multi-task detection model training method and device, equipment and a storage medium. BACKGROUND

[0002] Glass point defects usually include single-point large type or diffuse small point type. Different forms of point detection cannot be realized by using the same technical means. If different deep learning models are used to detect different types of glass point defects, the detection cost and time are large.

[0003] In the traditional technology, two detection heads are started in the same deep learning task, one for classification and the other for segmentation. However, the difficulty, defect type and defect quantity of the two tasks are different, and the convergence speed of different tasks is different in the process of training the deep learning model, which causes the different detection heads in the deep learning model to fail to converge effectively, resulting in inaccurate defect detection. SUMMARY

[0004] Therefore, it is necessary to provide a multi-task detection model training method, device, computer equipment, computer readable storage medium and computer program product to improve the accuracy of defect detection of the multi-task detection model.

[0005] In a first aspect, the present application provides a multi-task detection model training method, comprising:

[0006] Obtain defect sample data sets of a plurality of training tasks, and perform polling training on an initial multi-task detection model based on the defect sample data sets of each training task; the initial multi-task detection model comprises a backbone network and a plurality of head networks, and the head network comprises a detection sub-network corresponding to a training task;

[0007] For training of a target training task, input a target defect sample data set corresponding to the target training task into the backbone network to extract features, and obtain target data features corresponding to the target training task;

[0008] Predict the target data features through a detection sub-network corresponding to the target training task to obtain a target detection result;

[0009] Determine a training loss of the target training task based on the target detection result and a corresponding standard detection result, and determine a target training loss based on the training losses of each training task;

[0010] Determine the convergence speed of the target training task based on the target detection result, and determine the learning rate of the initial multi-task detection model based on the convergence speeds of each training task;

[0011] train the initial multi-task detection model based on the target training loss and the learning rate until a training condition is reached to obtain a target multi-task detection model for defect detection.

[0012] In a second aspect, the present application provides a training device of a multi-task detection model, comprising:

[0013] a data acquisition module configured to acquire defect sample data sets of a plurality of training tasks;

[0014] a polling training module configured to perform polling training on an initial multi-task detection model based on the defect sample data sets of the training tasks; the initial multi-task detection model comprises a backbone network and a plurality of head networks, and the head networks comprise detection sub-networks corresponding to the training tasks;

[0015] a feature extraction module configured to, for a target training task, input a target defect sample data set corresponding to the target training task into the backbone network to perform feature extraction, and obtain target data features corresponding to the target training task;

[0016] a result prediction module configured to predict the target data features through a detection sub-network corresponding to the target training task to obtain a target detection result;

[0017] a loss determination module configured to determine a training loss of the target training task based on the target detection result and a corresponding standard detection result, and determine a target training loss based on the training losses of the training tasks;

[0018] a learning rate determination module configured to determine a convergence speed of the target training task based on the target detection result, and determine a learning rate of the initial multi-task detection model based on the convergence speeds of the training tasks;

[0019] a model training module configured to train the initial multi-task detection model based on the target training loss and the learning rate until a training condition is reached to obtain a target multi-task detection model for defect detection.

[0020] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method when executing the computer program.

[0021] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method.

[0022] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in the above method.

[0023] The training method, device, computer equipment, computer readable storage medium and computer program product of the multi-task detection model provided by the embodiments of the present application can poll the initial multi-task detection model based on the defect sample data sets of each training task. In the training process of one of the target training tasks, the shared backbone network extracts features from the target defect sample data set, and the extracted target data features are predicted by the detection sub-network corresponding to the target training task to obtain a target detection result. The training loss and the convergence speed of the target training task are determined based on the target detection result. Then, the target training loss is determined based on the training loss of each training task, and the model learning rate is determined based on the convergence speed of each training task. Thus, the initial multi-task detection model is trained according to the target training loss and the model learning rate until the training condition is reached, and the target multi-task detection model is obtained. Because the learning rate of the model is adaptively adjusted for different training tasks in the training process of the model, the learning rate can match different training tasks, the convergence speeds of different training tasks remain consistent, the effective and stable convergence of different training tasks in the model is realized, and the robustness and accuracy of the multi-task detection model in defect detection are improved. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 An application environment diagram of a multi-task detection model training method provided by the embodiments of the present application;

[0025] Figure 2 A flowchart of a multi-task detection model training method provided by the embodiments of the present application;

[0026] Figure 3 A flowchart of another multi-task detection model training method provided by the embodiments of the present application;

[0027] Figure 4 A structural block diagram of a multi-task detection model training device provided by the embodiments of the present application;

[0028] Figure 5 An internal structure diagram of a computer equipment provided by the embodiments of the present application;

[0029] Figure 6 Another internal structure diagram of a computer equipment provided by the embodiments of the present application;

[0030] Figure 7 An internal structure diagram of a computer readable storage medium provided by the embodiments of the present application. DETAILED DESCRIPTION

[0031] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0032] The training method of the multi-task detection model provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 . The terminal 102 communicates with the server 104 through a communication network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, unmanned aerial vehicles, aircrafts, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0033] As shown in Figure 2 , the embodiments of the present application provide a training method of a multi-task detection model. The method is applied to the terminal 102 or the server 104 in Figure 1 for example. It can be understood that the computer device can include at least one of the terminal and the server. The method includes the following steps:

[0034] S202, obtaining a plurality of defect sample data sets of training tasks, and performing polling training on an initial multi-task detection model based on the defect sample data sets of the training tasks; wherein the initial multi-task detection model includes a backbone network and a plurality of head networks, and the head network includes a detection sub-network corresponding to a training task.

[0035] The different training tasks correspond to different defect detection logics, and the obtained detection results are also different. For example, the training tasks include a classification task and a segmentation task. The classification task is used to identify the global class in the image (whole image level), and each image corresponds to a label. The segmentation task is used to identify the class of each pixel in the image (pixel level), and each pixel is labeled. The defect sample dataset is used to train the model. The defect sample datasets for different training tasks are also different. For example, the defect sample dataset for the classification task needs to label the defect class of the whole sample image, and the defect sample dataset for the segmentation task needs to label the defect class of each pixel in the sample image. The defect sample dataset for each training task includes multiple training sample images, and each training sample image can include a label of a corresponding level defect class. The number of defect sample datasets for each training task is not limited herein. It is easy to understand that the more the number of defect sample datasets for each training task, the more beneficial it is to the training of the model accuracy.

[0036] The initial multi-task detection model is trained based on the defect sample dataset of each training task, that is, the defect sample dataset of each training task is trained in turn. The defect sample dataset of each training task can be trained once in turn, or the defect sample dataset of each training task can be continuously trained multiple times. For example, the training tasks include A task and B task. The defect sample dataset based on the A task can be trained once, and then the defect sample dataset based on the B task can be trained once. Similarly, the defect sample dataset based on the A task is repeatedly trained. Alternatively, the defect sample dataset based on the A task can be continuously trained twice, and then the defect sample dataset based on the B task can be continuously trained twice. Similarly, the defect sample dataset based on the A task is repeatedly trained.

[0037] The backbone network is used to extract features from the defect sample dataset. For example, the backbone network can extract pyramid features, that is, candidate data features of different scales. During training, different scales of candidate data features can be selected based on different training tasks to obtain data features corresponding to the respective training tasks. The head network is used to predict features and output detection results. Each training task corresponds to a detection subnetwork, that is, the defect sample dataset based on the training task is used to train the corresponding detection subnetwork to obtain the trained detection subnetwork.

[0038] S204, for the target training task, the target defect sample data set corresponding to the target training task is input into the backbone network for feature extraction, and the target data feature corresponding to the target training task is obtained.

[0039] In the case of extracting candidate data features of each scale from the target defect sample data set through the backbone network, the candidate data features of the corresponding scale extracted by the backbone network can be selected according to the type of the target training task to obtain the target data feature of the target training task. Wherein, each scale can be a pre-set scale, for example, the backbone network can extract candidate data features of scale 1, scale 2, scale 3, etc. in turn. The number of scales and the specific scale size extracted by the backbone network can be set according to the actual application scenario.

[0040] Exemplarily, if the target training task is a classification task, candidate data features of a target scale can be selected from candidate data features of each scale as target data features of the classification task. If the target task is a segmentation task, at least one candidate data feature can be selected from candidate data features of each scale to obtain target data features of the segmentation task. The target data features of other training tasks can be determined from candidate data features of each scale in the same way.

[0041] S206, the target data feature is predicted by the detection sub-network corresponding to the target training task to obtain the target detection result.

[0042] Among them, the detection sub-network corresponding to different training tasks can be different. For example, the detection sub-network corresponding to the classification task is a classification network, and the detection sub-network corresponding to the segmentation task is a segmentation network. If the target data feature is predicted by the classification network corresponding to the classification task, the target detection result corresponding to the classification task is obtained. If the target data feature is predicted by the segmentation network corresponding to the segmentation task, the target detection result corresponding to the segmentation task is obtained.

[0043] S208, based on the target detection result and the corresponding standard detection result, the training loss of the target training task is determined, and based on the training loss of each training task, the target training loss is determined.

[0044] Among them, the standard detection result is used to represent the real defect detection result corresponding to the target defect sample data set. The training loss of the target training task can be determined based on the difference between the target detection result and the corresponding standard detection result. For example, the cross entropy or mean square error of the target detection result and the corresponding standard detection result can be determined as the training loss of the target training task.

[0045] It is easy to understand that, since each training task is trained based on the defect sample data set of the training task by polling, when training based on the target defect sample data set of the target training task, the detection sub-networks corresponding to other training tasks can be frozen, that is, the corresponding parameters will not be updated. Therefore, the training loss of other training tasks except the target training task can be calculated based on the training loss obtained by the last training of other training tasks. Since the current training round is the training round of the target training task, the current training round can also be understood as the last training round of the target training task. In other words, the target training loss can be determined based on the training loss obtained by the last training of each training task. For example, the training loss obtained by the last training of each training task is fused to obtain the target training loss.

[0046] S210, determine the convergence speed of the target training task based on the target detection result, and determine the learning rate of the initial multi-task detection model based on the convergence speed of each training task.

[0047] The convergence speed is used to represent the rate of decrease of model error with the decrease of training number in the model training process. The convergence speeds of different training tasks are usually different. Illustratively, the detection accuracy can be determined based on the target detection result, and the convergence speed of the target training task can be determined based on the detection accuracy. The learning rate is used to represent the step size when the model parameters are updated each time.

[0048] Similar to the training loss, the convergence speed is also generated in the process of training the training task. The convergence speed of each training task can be determined based on the detection result obtained by the last training of each training task.

[0049] S212, training the initial multi-task detection model based on the target training loss and the learning rate until the training condition is reached to obtain the target multi-task detection model for defect detection.

[0050] After obtaining the target training loss and the learning rate, the model parameters are adjusted according to the target training loss and the corresponding learning rate until the training condition is reached to obtain the target multi-task detection model. The training condition refers to the condition under which the model stops training, for example, the training condition can be that the difference between the detection result output by each detection sub-network and the true detection result is less than a difference threshold, or the training number of each detection sub-network reaches a preset number. The training condition can be set according to the actual application scenario.

[0051] The target multi-task detection model can detect defects in the to-be-detected image based on each detection sub-network to obtain the detection result corresponding to each detection sub-network.

[0052] It can be seen that in the embodiment of the present application, the initial multi-task detection model is trained by polling based on the defect sample data sets of each training task. In the training process for one of the target training tasks, the shared backbone network extracts features from the target defect sample data set, and the extracted target data features are predicted through the detection sub-network corresponding to the target training task to obtain a target detection result. The training loss and the convergence speed of the target training task are determined based on the target detection result. Then, the target training loss is determined based on the training loss of each training task, and the model learning rate is determined based on the convergence speed of each training task. The initial multi-task detection model is trained according to the target training loss and the model learning rate until the training condition is reached, and the target multi-task detection model is obtained. Because the learning rate of the model is adaptively adjusted for different training tasks during the training process of the model, the learning rate can match different training tasks, the convergence speeds of different training tasks remain consistent, the effective and stable convergence of different training tasks in the model is realized, and the robustness and accuracy of the multi-task detection model for defect detection are improved.

[0053] In some embodiments, the step of inputting the target defect sample data set corresponding to the target training task into the backbone network for feature extraction to obtain target data features corresponding to the target training task in S204 includes:

[0054] The target defect sample data set corresponding to the target training task is input into the backbone network for feature extraction to obtain a scale feature set containing features of each scale;

[0055] The target scale feature corresponding to the target training task is determined from the scale feature set, and the target data feature corresponding to the target training task is determined based on the target scale feature.

[0056] The scale feature set refers to a set composed of candidate data features of each scale. The scale of the candidate data features in the scale feature set can be a preset scale, such as preset scale 1, preset scale 2, preset scale 3, preset scale 4, etc. Each scale can be set according to the actual application scenario. The backbone network can extract pyramid features of each scale from the target defect sample data set. The target scale feature refers to the candidate data feature corresponding to the target training task. The target scale features corresponding to different training tasks are generally different.

[0057] Exemplarily, if the target training task is a classification task, a candidate data feature with a smaller scale can be selected from the scale feature set as a target scale feature, and the target scale feature can be taken as a target data feature of the classification task. Alternatively, at least one candidate data feature with a smaller scale can be selected from the scale feature set as a target scale feature, and the target scale features can be fused to obtain a target data feature of the classification task. If the target training task is a segmentation task, at least one candidate data feature with a larger scale can be selected from the scale feature set as a target scale feature, and the target scale features can be fused to obtain a target data feature corresponding to the segmentation task.

[0058] It can be seen that, in the embodiment, the scale feature set composed of candidate data features of different scales extracted by the backbone network is determined, the target scale feature corresponding to the target training task is determined from the scale feature set, and the target data feature corresponding to the corresponding training task is determined based on the target scale feature, so that the data features matched with different training tasks can be extracted based on the same backbone network, information redundancy caused by extracting too many detailed features for a training task with less feature requirement in a multi-head network, or poor training effect caused by extracting too few detailed features for a training task with more feature requirement can be avoided, and the pertinence of feature extraction is improved.

[0059] In some embodiments, the target training task is a segmentation task, and determining the target data feature corresponding to the target training task based on the target scale feature includes:

[0060] The plurality of target scale features are fused to obtain a target data feature corresponding to the segmentation task.

[0061] The target scale features include a plurality of target scale features. The plurality of target scale features can be candidate data features with larger scales in the scale feature set.

[0062] In an exemplary embodiment, the plurality of target scale features can be added after being adjusted to the same scale to obtain a target data feature corresponding to the segmentation task.

[0063] In an exemplary embodiment, the plurality of target scale features can be spliced on corresponding channels to obtain a target data feature corresponding to the segmentation task. For example, for a plurality of target scale feature maps, each target scale feature map can be spliced in an R (red) channel, a G (green) channel, and a B (blue) channel to obtain a target data feature. It is easily understood that the channels in the actual application scenario are not limited to R / G / B channels.

[0064] In an exemplary embodiment, a weight can be assigned to each target scale feature, and the plurality of target scale features can be fused by weighting to obtain a target data feature corresponding to the segmentation task.

[0065] It can be seen that, in the embodiment, by fusing the multiple target scale features, the target data features corresponding to the segmentation task are obtained, and the accuracy of the target data features of the segmentation task can be improved.

[0066] In some embodiments, based on the training loss of each training task, the target training loss is determined, including:

[0067] The training loss obtained by each training task corresponding to the last training is weighted and fused to obtain the target training loss corresponding to the current training round.

[0068] Since the defect sample dataset of each training task is used to poll the corresponding detection subnetwork in the multi-task detection model, that is, each detection subnetwork is trained by the corresponding defect sample dataset in turn. The target training loss for the current training round can be obtained by weighted fusion of the training loss obtained by each training task corresponding to the last training.

[0069] For example, if the training task includes a classification task and a segmentation task. The classification task and the segmentation task are trained in turn, that is, if the classification task is trained in the current training round, the segmentation task is trained in the next training round, and then the classification task is trained, and so on. If the classification task is trained in the current training round, the training loss of the classification task in the current training round can be obtained, and the training loss of the classification task in the current training round is weighted and fused with the training loss of the segmentation task corresponding to the last training round to obtain the target training loss of the current training round. Or, if the segmentation task is trained in the current training round, the training loss of the segmentation task in the current training round can be obtained, and the training loss of the segmentation task in the current training round is weighted and fused with the training loss of the classification task corresponding to the last training round to obtain the target training loss of the current training round.

[0070] It can be seen that, in the embodiment, by weighted fusion of the training loss obtained by each training task corresponding to the last training, the target loss corresponding to the current training round is obtained, the training loss of each training task can be considered comprehensively, the model is trained, and the accuracy of the model training loss can be improved.

[0071] In some embodiments, based on the target detection result, the convergence speed of the target training task is determined, including:

[0072] Based on the target detection result, the detection accuracy is determined;

[0073] Based on the detection accuracy obtained in the current training round and the detection accuracy corresponding to the target training task obtained in the last training round, the convergence speed of the target training task in the current training round is determined.

[0074] The detection accuracy rate refers to a proportion of accurate detection results in the target detection results. The detection accuracy rate can be obtained by counting the target detection results. The detection accuracy rate corresponding to the target training task obtained in the last training round refers to the target training task in the current training round, and the detection accuracy rate corresponding to the target training task obtained in the last training.

[0075] Exemplarily, the difference between the detection accuracy rate obtained in the current training round and the detection accuracy rate corresponding to the target training task obtained in the last training round is taken as the convergence speed of the target training task in the current training round. Alternatively, the ratio of the difference between the detection accuracy rate obtained in the current training round and the detection accuracy rate corresponding to the target training task obtained in the last training round to the detection accuracy rate corresponding to the target training task obtained in the last training round is taken as the convergence speed of the target training task in the current training round.

[0076] In an actual application scenario, if the target training task is a classification task and the target detection result is a classification result, the proportion of accurate classification results in the classification results is directly counted as the detection accuracy rate of the classification result. If the target training task is a segmentation task and the target detection result is a segmentation result, the segmentation result includes the position of the prediction box, and the detection accuracy rate can be determined according to the Intersection over Union (IOU) between the prediction box and the real box. For example, the prediction box with an IOU higher than a threshold value can be taken as an accurate segmentation result, so as to count the proportion of the accurate segmentation result as the detection accuracy rate of the segmentation task.

[0077] It can be seen that, in the embodiment, the convergence speed of the target training task in the current training round is determined by the detection accuracy rates corresponding to the two consecutive training rounds of the target training task, so that the convergence speed of the target training task in the current training round can be quickly determined.

[0078] In some embodiments, the learning rate of the initial multi-task detection model is determined based on the convergence speeds of the training tasks, including:

[0079] The convergence speeds corresponding to the last training of the training tasks are normalized to obtain normalized convergence speeds;

[0080] The learning rate of the initial multi-task detection model in the next training round is adjusted according to the size relationship between the normalized convergence speeds of the training tasks.

[0081] The normalization processing can be normalization based on a preset value, and the preset value can be set based on the scale of the training loss. For example, the convergence speed corresponding to the last training of each training task is divided by a set value to obtain the normalized convergence speed corresponding to each training task.

[0082] For example, if the training task includes a classification task and a segmentation task, if the current training round trains the classification task, if the normalized convergence speed corresponding to the classification task is smaller than the normalized convergence speed corresponding to the segmentation task, the learning rate of the initial multi-task detection model in the next training round is adjusted to be larger. If the normalized convergence speed corresponding to the classification task is larger than the normalized convergence speed corresponding to the segmentation task, the learning rate of the initial multi-task detection model in the next training round is adjusted to be smaller. Conversely, if the current training round trains the segmentation task, if the normalized convergence speed corresponding to the segmentation task is smaller than the normalized convergence speed corresponding to the classification task, the learning rate of the initial multi-task detection model in the next training round is adjusted to be larger. If the normalized convergence speed corresponding to the segmentation task is larger than the normalized convergence speed corresponding to the classification task, the learning rate of the initial multi-task detection model in the next training round is adjusted to be smaller. In other words, if the convergence speed corresponding to the training task trained in the current training round is larger, the learning rate of the initial multi-task detection model in the next training round is adjusted to be smaller.

[0083] For example, if the training task includes a classification task, a segmentation task and a C task, the normalized convergence speeds corresponding to the classification task, the segmentation task and the C task can be obtained, and the training task corresponding to the smallest convergence speed can be ignored. For example, if the convergence speed of the C task is the smallest, the learning rate of the initial multi-task detection model in the next training round is adjusted according to the normalized convergence speeds corresponding to the classification task and the segmentation task, that is, the model learning rate in the next training round is adjusted in the manner described in the above example.

[0084] It can be seen that, in the embodiment, by adjusting the learning rate of the initial multi-task detection model in the next training round according to the size relationship between the normalized convergence speeds corresponding to the last training of each training task, the learning rate in the next training round can be adjusted to be higher when the convergence speed of the current training task is slower, and the learning rate in the next training round can be adjusted to be lower when the convergence speed of the current training task is faster. The learning rate is dynamically adjusted based on the convergence speed of each training task, so that the convergence speeds of different training tasks are consistent, and the problem that the model cannot converge effectively due to the large difference in convergence speed caused by the nature and difficulty of different training tasks is prevented.

[0085] In some embodiments, the above method further includes:

[0086] obtaining a to-be-detected image corresponding to a to-be-detected object;

[0087] inputting the to-be-detected image into the target multi-task detection model for processing to obtain a sub-detection result corresponding to each detection sub-network;

[0088] comparing each sub-detection result with a corresponding acceptance condition to obtain a comparison result;

[0089] determining a defect detection result according to the comparison result.

[0090] The object to be detected refers to an object that needs to be subjected to defect detection. The object to be detected is, for example, an industrial product such as a glass piece, a plastic piece, or a metal piece. The detection image is obtained by capturing the object to be detected. The acceptance condition is a condition that meets the requirements and is set in advance. For example, the acceptance condition includes that the number of defects is less than a preset number, the size of the defect is less than a preset size, and the like. The acceptance conditions corresponding to different detection tasks can be the same or different. The acceptance condition can be set according to the actual application scenario. The comparison result is used to represent whether the sub-detection result meets the corresponding acceptance condition. The defect detection result is used to represent whether the defect detection is passed. The defect detection result includes detection pass and detection fail.

[0091] For example, the detection image is input into the target multi-task detection model, and each detection subnetwork in the target multi-task detection model performs defect detection on the detection image to obtain a sub-detection result corresponding to each detection subnetwork. The defect detection result is determined as detection pass in a case where a preset number of sub-detection results in the comparison result meet the corresponding acceptance condition. The preset number can be 1, 2, 3, or the number of detection subnetworks, and the preset number can be set according to the actual application scenario.

[0092] In an example embodiment, if one sub-detection result in the comparison result meets the corresponding acceptance condition, the defect detection result is determined as detection pass. For example, if the detection subnetwork includes a classification network and a segmentation network, the classification network performs defect detection on the detection image to obtain a sub-detection result as a classification detection result, and the segmentation network performs defect detection on the detection image to obtain a sub-detection result as a segmentation detection result. Then, the classification detection result is compared with the corresponding acceptance condition, and the segmentation detection result is compared with the corresponding acceptance condition. If the classification detection result or the segmentation detection result meets the corresponding acceptance condition, the defect detection result is determined as detection pass. If no sub-detection result in the comparison result meets the corresponding acceptance condition, the defect detection result is determined as detection fail.

[0093] In an example embodiment, if each sub-detection result in the comparison result meets the corresponding acceptance condition, the defect detection result is determined as detection pass. If one sub-detection result in the comparison result does not meet the corresponding acceptance condition, the defect detection result is determined as detection fail.

[0094] It can be seen that, in the embodiment, the target multi-task detection model is used to detect the to-be-detected image of the to-be-detected object, and the sub-detection results corresponding to each detection sub-network are obtained. Since the convergence speeds of each detection sub-network are consistent, the model can effectively converge, the accuracy of the obtained sub-detection results is high, the result of the corresponding acceptance condition judgment based on each sub-detection result is more accurate, and therefore the accuracy of the defect detection result is improved.

[0095] In some embodiments, after obtaining the defect sample data sets of multiple training tasks, if the number of the initial defect sample data set of a training task is relatively small compared with other training tasks, the initial defect sample data set of the training task can be subjected to data enhancement processing to obtain an enhanced defect sample data set, so that the corresponding training task is trained based on the enhanced defect sample data set. For example, data enhancement processing can be achieved by changing the color, defect position or random mask area of the image in the initial defect sample data set. Alternatively, in some application scenarios, the initial defect sample data set can be subjected to data enhancement to obtain an enhanced defect sample data set when the initial defect sample data set is used up and the corresponding training task needs to be trained. Each time of training polls one training task, which can prevent the situation that the number of defect sample data sets of different training tasks is too different, resulting in poor model convergence effect.

[0096] The traditional multi-task model is described by taking classification and segmentation as examples. The classification problem is relatively simple, and the training data (defect sample data set) is easier to obtain. The segmentation problem is relatively complex, and the training data is more difficult to obtain. Moreover, the classification model can converge to a good effect in a relatively short time, while the segmentation model needs several times as much time to converge to the same effect. For example, as shown in FIG. 1, the initial multi-task detection model provided in the embodiment includes a shared feature extraction backbone, a classification head, a segmentation head, a loss (training loss) fusion module and a learning rate constraint model. Figure 3

[0097] The initial multi-task detection model polls the classification task and the segmentation task, that is, the training of the classification task and the segmentation task is performed alternately. The classification task is trained based on the classification data set to the classification head, and the segmentation task is trained based on the segmentation data set to the segmentation head. In the case where the training data set is insufficient, the initial training data set is enhanced and reused.

[0098] ​In the training of each training round, the training data is subjected to feature extraction by the backbone, and according to the different training tasks, the features corresponding to the training tasks are extracted into the corresponding task head to obtain the corresponding detection result. For example, if the classification task is targeted, low-dimensional global features are extracted as target data features of the classification data set, the low-dimensional global features are predicted by the classification head to obtain the classification detection result (target detection result), and the classification loss is calculated based on the classification detection result. If the segmentation task is targeted, multiple-dimensional features are extracted for fusion to obtain cross-dimension fusion features as target data features of the segmentation data set, the cross-dimension fusion features are predicted by the segmentation head to obtain the segmentation detection result, and the segmentation loss is calculated based on the segmentation detection result. The classification loss and the segmentation loss are normalized and fused to obtain the target training loss. The training loss of the current training round is calculated based on the detection result of the current training round and the real detection result, and is normalized to a pre-set scale. It should be noted that in the current training round, one task head is trained and another task head is frozen, that is, the other task head will not update the parameters. The target training loss is obtained by fusing the training loss of the current training round and the training loss of the last training round.

[0099] Based on the detection result of the current training round, the convergence speed of the current training task is calculated. According to the convergence speed of the current training task and the convergence speed of another training task, the learning rate of the next round of training task is dynamically adjusted. That is, if the convergence speed of the current training task is faster, the learning rate of the next round of training task is lowered, and if the convergence speed of the current training task is slower, the learning rate of the next round of training task is increased. The above steps are iterated, and the initial multi-task detection model is trained based on the target training loss and the learning rate until the model converges or reaches a pre-set stopping condition, and the target multi-task detection model is obtained.

[0100] It can be seen that, by adding the task convergence rate recovery constraint, the embodiment realizes the effect of balanced convergence of multi-task, prevents the convergence speed difference caused by the nature and difficulty of different tasks from being too large, and causes the whole model to be unable to effectively converge. By keeping the convergence speed of each training task consistent, the effective convergence of the model can be realized, thereby improving the accuracy of the model in defect detection.

[0101] It should be understood that although each step in the flowchart involved in each of the above embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0102] Based on the same inventive concept, the embodiments of the present application also provide a multi-task detection model training device. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-task detection model training device embodiments provided below can refer to the limitations of the multi-task detection model training method in the above text, which will not be repeated here.

[0103] As shown in Figure 4 The embodiments of the present application provide a multi-task detection model training device 400, which comprises:

[0104] The data acquisition module 402 is configured to acquire defect sample data sets of a plurality of training tasks.

[0105] The polling training module 404 is configured to perform polling training on an initial multi-task detection model based on the defect sample data sets of the training tasks. The initial multi-task detection model comprises a backbone network and a plurality of head networks, and the head network comprises a detection subnetwork corresponding to a training task.

[0106] The feature extraction module 406 is configured to, for training of a target training task, input a target defect sample data set corresponding to the target training task into the backbone network for feature extraction, to obtain target data features corresponding to the target training task.

[0107] The result prediction module 408 is configured to predict the target data features through a detection subnetwork corresponding to the target training task, to obtain a target detection result.

[0108] The loss determination module 410 is configured to determine a training loss of the target training task based on the target detection result and a corresponding standard detection result, and determine a target training loss based on the training losses of the training tasks.

[0109] The learning rate determination module 412 is configured to determine the convergence speed of the target training task based on the target detection result, and determine the learning rate of the initial multi-task detection model based on the convergence speeds of the training tasks.

[0110] The model training module 414 is configured to train the initial multi-task detection model based on the target training loss and the learning rate until a training condition is reached, to obtain a target multi-task detection model for defect detection.

[0111] In some embodiments, in the aspect of inputting the target defect sample data set corresponding to the target training task into the backbone network for feature extraction to obtain the target data feature corresponding to the target training task, the feature extraction module 406 is specifically configured to:

[0112] input the target defect sample data set corresponding to the target training task into the backbone network for feature extraction to obtain a scale feature set containing scale features of different scales;

[0113] determine the target scale feature corresponding to the target training task from the scale feature set, and determine the target data feature corresponding to the target training task based on the target scale feature.

[0114] In some embodiments, the target training task is a segmentation task, and in the aspect of determining the target data feature corresponding to the target training task based on the target scale feature, the feature extraction module 406 is specifically configured to:

[0115] fuse the plurality of target scale features to obtain the target data feature corresponding to the segmentation task.

[0116] In some embodiments, in the aspect of determining the target training loss based on the training losses of the training tasks, the loss determination module 410 is specifically configured to:

[0117] weight and fuse the training losses obtained in the last training corresponding to the training tasks to obtain the target training loss corresponding to the current training round.

[0118] In some embodiments, in the aspect of determining the convergence speed of the target training task based on the target detection result, the learning rate determination module 412 is specifically configured to:

[0119] determine the detection accuracy based on the target detection result;

[0120] determine the convergence speed of the target training task in the current training round based on the detection accuracy obtained in the current training round and the detection accuracy corresponding to the target training task obtained in the last training round.

[0121] In some embodiments, in the aspect of determining the learning rate of the initial multi-task detection model based on the convergence speeds of the training tasks, the learning rate determination module 412 is specifically configured to:

[0122] The convergence speed corresponding to the last training of each training task is normalized to obtain a normalized convergence speed;

[0123] According to the size relationship between the normalized convergence speeds of each training task, the learning rate of the initial multi-task detection model in the next training round is adjusted.

[0124] In some embodiments, the above device further comprises a defect detection module for: acquiring a to-be-detected image corresponding to a to-be-detected object; inputting the to-be-detected image into the target multi-task detection model for processing to obtain a sub-detection result corresponding to each detection sub-network; comparing each sub-detection result with the corresponding acceptance condition to obtain a comparison result; and determining a defect detection result according to the comparison result.

[0125] Each module in the above multi-task detection model training device can be realized by software, hardware, and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0126] In some embodiments, a computer device, which can be a server, is provided, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store training-related data of a multi-task detection model. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the steps in the above multi-task detection model training method.

[0127] In some embodiments, a computer device, which can be a terminal, is provided, and an internal structure diagram thereof can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through the system bus, the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the external terminal in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to realize the steps in the training method of the multi-task detection model. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0128] Those skilled in the art can understand that, Figure 5 or Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0129] In some embodiments, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps in the above method embodiments.

[0130] In some embodiments, as Figure 7 A block diagram of the internal structure of a computer readable storage medium is provided, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in the above method embodiments.

[0131] In some embodiments, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to realize the steps in the above method embodiments.

[0132] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0133] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of each method can be included. Any reference to a memory, database or other medium used in the embodiments provided by the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0134] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0135] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for training a multitask detection model, the method comprising: The method comprises: obtaining a plurality of defect sample data sets of training tasks, and performing polling training on an initial multi-task detection model based on the defect sample data set of each training task; the initial multi-task detection model comprises a backbone network and a plurality of head networks, and the head network comprises a detection subnetwork corresponding to a training task; for training of a target training task, a target defect sample data set corresponding to the target training task is input into the backbone network for feature extraction, to obtain a target data feature corresponding to the target training task; a target detection result is obtained by predicting the target data feature through a detection subnetwork corresponding to the target training task; based on the target detection result and a corresponding standard detection result, a training loss of the target training task is determined, and a target training loss is determined based on the training loss of each training task; based on the target detection result, a convergence speed of the target training task is determined, and a learning rate of the initial multi-task detection model is determined based on the convergence speed of each training task; based on the target training loss and the learning rate, the initial multi-task detection model is trained until a training condition is reached, to obtain a target multi-task detection model for defect detection.

2. The method of claim 1, wherein, The method further comprises: inputting the target defect sample data set corresponding to the target training task into the backbone network for feature extraction to obtain a scale feature set containing each scale; determining a target scale feature corresponding to the target training task from the scale feature set, and determining a target data feature corresponding to the target training task based on the target scale feature.

3. The method of claim 2, wherein, The target training task is a segmentation task, and the target data feature corresponding to the target training task is determined based on the target scale feature, comprising: fuse a plurality of target scale features to obtain a target data feature corresponding to the segmentation task.

4. The method of claim 1, wherein, The target training loss is determined based on the training loss of each training task, comprising: weighting and fusing the training loss obtained in the last training of each training task to obtain a target training loss corresponding to the current training round.

5. The method of claim 1, wherein, The convergence speed of the target training task is determined based on the target detection result, comprising: determining a detection accuracy based on the target detection result; based on the detection accuracy obtained in the current training round and the detection accuracy corresponding to the target training task obtained in the last training round, determining the convergence speed of the target training task in the current training round.

6. The method of claim 1, wherein, The learning rate of the initial multi-task detection model is determined based on the convergence speed of each training task, comprising: normalizing the convergence speed corresponding to each training task in the last training to obtain a normalized convergence speed; adjusting the learning rate of the initial multi-task detection model in the next training round according to the size relationship between the normalized convergence speeds of each training task.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: Obtaining a to-be-detected image corresponding to a to-be-detected object; The to-be-detected image is input into the target multi-task detection model for processing to obtain a sub-detection result corresponding to each detection sub-network; Each sub-detection result is compared with a corresponding acceptance condition to obtain a comparison result; According to the comparison result, a defect detection result is determined. 8.A device for training a multitask detection model, comprising: Comprise: A data acquisition module for acquiring defect sample data sets of multiple training tasks; A polling training module for polling training an initial multi-task detection model based on the defect sample data sets of each training task; The initial multi-task detection model comprises a backbone network and a plurality of head networks, and the head networks comprise detection sub-networks corresponding to the training tasks; A feature extraction module for extracting features of a target defect sample data set corresponding to a target training task by inputting the target defect sample data set into the backbone network, to obtain target data features corresponding to the target training task; A result prediction module for predicting the target data features by a detection sub-network corresponding to the target training task to obtain a target detection result; A loss determination module for determining a training loss of the target training task based on the target detection result and a corresponding standard detection result, and determining a target training loss based on the training losses of each training task; A learning rate determination module for determining a convergence speed of the target training task based on the target detection result, and determining a learning rate of the initial multi-task detection model based on the convergence speeds of each training task; A model training module for training the initial multi-task detection model based on the target training loss and the learning rate until a training condition is reached to obtain a target multi-task detection model for defect detection. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 7.

Images