A model training method and related equipment

Abstract

The embodiment of the present application provides a model training method and related devices, which can be used in the fields of artificial intelligence, computer vision, etc., for image detection. The method includes: extracting feature information in the target image through the feature extraction layer of the first network and the feature extraction layer of the second network respectively; and further extracting features about the target object in the feature information through Gaussian masks respectively, to obtain first local features and second local features; then determining feature loss through the first local features and the second local features; and predicting first classification prediction values ​​and second classification prediction values ​​based on the same set of region proposals through the first network and the second network, and then obtaining classification loss according to the first classification prediction values ​​and the second classification prediction values; then training the second network according to the classification loss and feature loss to obtain the target network. The use of the embodiment of the present application can make the prediction speed of the target network used to detect images faster and more accurate.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a model training method and related equipment. Background Art

[0002] Object detection refers to the classification and location of target objects in an image, such as Figure 1 As shown, the umbrella 101 in the image can be classified and located through target detection, and the person 102 in the image can also be classified and located. The application of target detection in images is very wide, such as autonomous driving, safe cities, and mobile terminals, so there are high requirements for detection accuracy and speed. Target detection in images is usually achieved through neural networks. However, although the detection accuracy of large neural networks is high, the speed is very slow; the detection speed of small neural networks is fast, but the accuracy is very low.

[0003] How to train a neural network with faster detection speed and more accurate detection results is a technical problem that technicians in this field are studying. Summary of the invention

[0004] The embodiments of the present application disclose a model training method and related equipment, which can be applied to the fields of artificial intelligence, computer vision, etc. for image detection. The method and related equipment can improve the prediction efficiency and accuracy of the network.

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0006] Extracting first feature information in the target image through a feature extraction layer of the first network;

[0007] Extracting second feature information in the target image through a feature extraction layer of a second network, wherein both the first network and the second network are classification networks, and the depth of the first network is greater than the depth of the second network;

[0008] Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature;

[0009] Extracting features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature;

[0010] Determine a feature loss by using the first local feature and the second local feature;

[0011] The second network is trained according to the feature loss to obtain a target network.

[0012] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0013] In combination with the first aspect, in a first possible implementation manner of the first aspect, the method further includes:

[0014] Generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of the first network;

[0015] Generating, by the classification layer of the second network, a second classification prediction value of the target region proposal in the region proposal set;

[0016] Determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0017] The step of training the second network according to the feature loss to obtain a target network includes:

[0018] The second network is trained according to the feature loss and the classification loss to obtain a target network.

[0019] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0020] In combination with the first aspect, or the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, training the second network according to the feature loss to obtain the target network includes:

[0021] training the second network according to the feature loss;

[0022] The trained second network is trained through a third network to obtain a target network, wherein the depth of the third network is greater than the depth of the first network.

[0023] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0024] In a second aspect, an embodiment of the present application provides a model training method, the method comprising:

[0025] Training the second network based on the first network to obtain an intermediate network;

[0026] The intermediate network is trained based on the third network to obtain a target network, wherein the first network, the second network and the third network are all classification networks, and the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network.

[0027] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0028] In combination with the second aspect, in a first possible implementation manner of the second aspect, training the second network based on the first network includes:

[0029] Extracting first feature information in the target image through a feature extraction layer of the first network;

[0030] Extracting second feature information in the target image through a feature extraction layer of a second network;

[0031] Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature;

[0032] Extracting features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature;

[0033] Determine a feature loss by using the first local feature and the second local feature;

[0034] The second network is trained according to the feature loss to obtain the intermediate network.

[0035] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0036] In combination with the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the method further includes:

[0037] Generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of a first network;

[0038] Generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of a second network;

[0039] Determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0040] The step of training the second network according to the feature loss to obtain the intermediate network includes:

[0041] The second network is trained according to the feature loss and the classification loss to obtain the intermediate network.

[0042] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0043] In combination with the first aspect, or the second aspect, or any possible implementation manner of the first aspect, or any possible implementation manner of the second aspect, in another possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0044] In combination with the first aspect, or the second aspect, or any possible implementation of the first aspect, or any possible implementation of the second aspect, in another possible implementation, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0045] In combination with the first aspect, or the second aspect, or any possible implementation of the first aspect, or any possible implementation of the second aspect, in another possible implementation, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal in the region proposal set that belongs to the target object.

[0046] In combination with the first aspect, or the second aspect, or any possible implementation of the first aspect, or any possible implementation of the second aspect, in another possible implementation, the classification loss L cls Satisfies the following relationship:

[0047]

[0048] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0049] In combination with the first aspect, or the second aspect, or any possible implementation of the first aspect, or any possible implementation of the second aspect, in yet another possible implementation, the method further includes:

[0050] Determining a regression loss and an RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image;

[0051] The second network is trained according to the feature loss and the classification loss to obtain a target network, including:

[0052] The second network is trained according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

[0053] In combination with the first aspect, or the second aspect, or any possible implementation manner of the first aspect, or any possible implementation manner of the second aspect, in another possible implementation manner, after training the second network according to the feature loss to obtain the target network, the method further includes:

[0054] The target network is sent to a model-using device, where the target network is used to predict content in an image.

[0055] In a third aspect, an embodiment of the present application provides an image detection method, the method comprising:

[0056] Acquire a target network, wherein the target network is a network obtained by training the second network through the first network, the parameters used for training the second network through the first network include feature loss, the feature loss is determined according to the first local feature and the second local feature, the first local feature is a feature about the target object extracted from the first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from the second feature information through a Gaussian mask, the first feature information is feature information in the target image extracted through the feature extraction layer of the first network, the second feature information is feature information in the target image extracted through the feature extraction layer of the second network, the first network and the second network are both classification networks, and the depth of the first network is greater than the depth of the second network;

[0057] The target network identifies the content in the image.

[0058] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0059] In combination with the third aspect, in a first possible implementation manner of the third aspect, the parameters used to train the second network also include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is the classification prediction value of the target region proposal in the region proposal set generated by the classification layer of the first network, and the second classification prediction value is the classification prediction value of the target region proposal in the region proposal set generated by the classification layer of the second network.

[0060] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0061] In combination with the third aspect, or the first possible implementation of the third aspect, in a second possible implementation, the target network is specifically a network obtained by training the second network through the first network and further training the trained network through a third network, wherein the depth of the third network is greater than the depth of the first network.

[0062] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0063] In a fourth aspect, an embodiment of the present application provides an image detection method, the method comprising:

[0064] Acquire a target network, wherein the target network is a network obtained by training the second network through multiple network iterations, the multiple networks are all classification networks, the multiple networks include at least a first network and a third network, the third network is used to train the intermediate network after the first network trains the second network to obtain the intermediate network, wherein the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network;

[0065] The target network identifies the content in the image.

[0066] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0067] In combination with the fourth aspect, in a first possible implementation manner of the fourth aspect, the parameters used by the first network when training the second network include feature loss, wherein the feature loss is determined based on a first local feature and a second local feature, the first local feature is a feature about the target object extracted from the first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from the second feature information through a Gaussian mask, the first feature information is feature information in the target image extracted through the feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through the feature extraction layer of the second network.

[0068] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0069] In combination with the first possible implementation manner of the fourth aspect, in a second possible implementation manner of the fourth aspect, the parameters used by the first network when training the second network include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value is a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

[0070] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0071] In combination with the third aspect, or the fourth aspect, or any possible implementation manner of the third aspect, or any possible implementation manner of the fourth aspect, in another possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0072] In combination with the third aspect, or the fourth aspect, or any possible implementation of the third aspect, or any possible implementation of the fourth aspect, in another possible implementation, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0073] In combination with the third aspect, or the fourth aspect, or any possible implementation of the third aspect, or any possible implementation of the fourth aspect, in another possible implementation, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal in the region proposal set that belongs to the target object.

[0074] In combination with the third aspect, or the fourth aspect, or any possible implementation of the third aspect, or any possible implementation of the fourth aspect, in another possible implementation, the classification loss L cls Satisfies the following relationship:

[0075]

[0076] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0077] In combination with the third aspect, or the fourth aspect, or any possible implementation of the third aspect, or any possible implementation of the fourth aspect, in another possible implementation, the parameters used to train the second network also include the regression loss and RPN loss of the second network, wherein the regression loss and RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the second network predicting the region proposal in the target image.

[0078] In combination with the third aspect, or the fourth aspect, or any possible implementation of the third aspect, or any possible implementation of the fourth aspect, in another possible implementation, acquiring the target network includes:

[0079] Receive a target network sent by a model training device, wherein the model training device is used to train to obtain the target network.

[0080] In a fifth aspect, an embodiment of the present application provides a model training device, the device comprising:

[0081] A feature extraction unit, configured to extract first feature information from a target image through a feature extraction layer of a first network;

[0082] The feature extraction unit is further used to extract second feature information in the target image through a feature extraction layer of a second network, wherein both the first network and the second network are classification networks, and the depth of the first network is greater than the depth of the second network;

[0083] A first optimization unit, configured to extract features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature;

[0084] A second optimization unit is used to extract features about the target object in the second feature information through a Gaussian mask to obtain a second local feature;

[0085] A first determining unit, configured to determine a feature loss by using the first local feature and the second local feature;

[0086] A weight adjustment unit is used to train the second network according to the feature loss to obtain a target network.

[0087] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0088] In combination with the fifth aspect, in a first possible implementation manner of the fifth aspect, the device further includes:

[0089] A first generating unit, configured to generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of the first network;

[0090] A second generating unit, configured to generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of the second network;

[0091] A second determining unit, configured to determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0092] The weight adjustment unit is specifically used to: train the second network according to the feature loss and the classification loss to obtain a target network.

[0093] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0094] In combination with the fifth aspect, or any possible implementation manner of the fifth aspect, in a second possible implementation manner of the fifth aspect, when training the second network according to the feature loss to obtain the target network, the weight adjustment unit is specifically used to:

[0095] training the second network according to the feature loss;

[0096] The trained second network is trained through a third network to obtain a target network, wherein the depth of the third network is greater than the depth of the first network.

[0097] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0098] In a sixth aspect, an embodiment of the present application provides a model training device, the device comprising:

[0099] A first training unit, configured to train a second network based on the first network to obtain an intermediate network;

[0100] The second training unit is used to train the intermediate network based on the third network to obtain a target network, wherein the first network, the second network and the third network are all classification networks, and the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network.

[0101] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0102] In combination with the sixth aspect, in a first possible implementation manner of the sixth aspect, training the second network based on the first network to obtain the intermediate network includes:

[0103] Extracting first feature information in the target image through a feature extraction layer of the first network;

[0104] Extracting second feature information in the target image through a feature extraction layer of a second network;

[0105] Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature;

[0106] Extracting features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature;

[0107] Determine a feature loss by using the first local feature and the second local feature;

[0108] The second network is trained according to the feature loss to obtain the intermediate network.

[0109] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0110] In combination with the sixth aspect, or the first possible implementation manner of the sixth aspect, in a second possible implementation manner of the sixth aspect, the apparatus further includes:

[0111] A first generating unit, configured to generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of a first network;

[0112] A second generating unit, configured to generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of a second network;

[0113] A second determining unit, configured to determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0114] The step of training the second network according to the feature loss to obtain the intermediate network is specifically as follows:

[0115] The second network is trained according to the feature loss and the classification loss to obtain the intermediate network.

[0116] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0117] In combination with the fifth aspect, or the sixth aspect, or any possible implementation manner of the fifth aspect, or any possible implementation manner of the sixth aspect, in another possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0118] In combination with the fifth aspect, or the sixth aspect, or any possible implementation of the fifth aspect, or any possible implementation of the sixth aspect, in another possible implementation, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0119] In combination with the fifth aspect, or the sixth aspect, or any possible implementation of the fifth aspect, or any possible implementation of the sixth aspect, in another possible implementation, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal in the region proposal set that belongs to the target object.

[0120] In combination with the fifth aspect, or the sixth aspect, or any possible implementation of the fifth aspect, or any possible implementation of the sixth aspect, in another possible implementation, the classification loss L cls Satisfies the following relationship:

[0121]

[0122] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0123] In combination with the fifth aspect, or the sixth aspect, or any possible implementation manner of the fifth aspect, or any possible implementation manner of the sixth aspect, in yet another possible implementation manner, the apparatus further includes:

[0124] A third determining unit is used to determine the regression loss and the RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image;

[0125] The weight adjustment unit is specifically used to: train the second network according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

[0126] In combination with the fifth aspect, or the sixth aspect, or any possible implementation of the fifth aspect, or any possible implementation of the sixth aspect, in yet another possible implementation, the method further includes:

[0127] A sending unit is used to send the target network to a model using device after the weight adjustment unit trains the second network according to the feature loss to obtain the target network, wherein the target network is used to predict the content in the image.

[0128] In a seventh aspect, an embodiment of the present application provides an image detection device, the device comprising:

[0129] an acquisition unit, configured to acquire a target network, wherein the target network is a network obtained by training the second network through the first network, the parameters used for training the second network through the first network include feature loss, the feature loss is determined according to a first local feature and a second local feature, the first local feature is a feature about a target object extracted from first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from second feature information through a Gaussian mask, the first feature information is feature information in a target image extracted through a feature extraction layer of the first network, the second feature information is feature information in the target image extracted through a feature extraction layer of the second network, the first network and the second network are both classification networks, and the depth of the first network is greater than the depth of the second network;

[0130] The recognition unit is used to recognize the content in the image through the target network.

[0131] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0132] In combination with the seventh aspect, in a first possible implementation manner of the seventh aspect, the parameters used to train the second network also include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is the classification prediction value of the target region proposal in the region proposal set generated by the classification layer of the first network, and the second classification prediction value is the classification prediction value of the target region proposal in the region proposal set generated by the classification layer of the second network.

[0133] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0134] In combination with the seventh aspect, or the first possible implementation of the seventh aspect, in a second possible implementation of the seventh aspect, the target network is specifically a network obtained by training the second network through the first network and further training the trained network through a third network, wherein the depth of the third network is greater than the depth of the first network.

[0135] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0136] In an eighth aspect, an embodiment of the present application provides an image detection device, the device comprising:

[0137] an acquisition unit, configured to acquire a target network, wherein the target network is a network obtained by training the second network through multiple network iterations, the multiple networks are all classification networks, the multiple networks include at least a first network and a third network, the third network is used to train the intermediate network after the first network trains the second network to obtain the intermediate network, wherein the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network;

[0138] The recognition unit is used to recognize the content in the image through the target network.

[0139] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0140] In combination with the eighth aspect, in a first possible implementation method of the eighth aspect, the parameters used by the first network when training the second network include feature loss, wherein the feature loss is determined based on a first local feature and a second local feature, the first local feature is a feature about the target object extracted from the first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from the second feature information through a Gaussian mask, the first feature information is feature information in the target image extracted through the feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through the feature extraction layer of the second network.

[0141] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0142] In combination with the first possible implementation manner of the eighth aspect, in a second possible implementation manner of the eighth aspect, the parameters used by the first network when training the second network include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value is a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

[0143] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0144] In combination with the seventh aspect, or the eighth aspect, or any possible implementation manner of the seventh aspect, or any possible implementation manner of the eighth aspect, in another possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0145] In combination with the seventh aspect, or the eighth aspect, or any possible implementation of the seventh aspect, or any possible implementation of the eighth aspect, in another possible implementation, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0146] In combination with the seventh aspect, or the eighth aspect, or any possible implementation of the seventh aspect, or any possible implementation of the eighth aspect, in another possible implementation, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal in the region proposal set that belongs to the target object.

[0147] In combination with the seventh aspect, or the eighth aspect, or any possible implementation of the seventh aspect, or any possible implementation of the eighth aspect, in another possible implementation, the classification loss L cls Satisfies the following relationship:

[0148]

[0149] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0150] In combination with the seventh aspect, or the eighth aspect, or any possible implementation of the seventh aspect, or any possible implementation of the eighth aspect, in another possible implementation: the parameters used to train the second network also include the regression loss and RPN loss of the second network, wherein the regression loss and RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the second network predicting the region proposal in the target image.

[0151] In combination with the seventh aspect, or the eighth aspect, or any possible implementation manner of the seventh aspect, or any possible implementation manner of the eighth aspect, in another possible implementation manner, the acquisition unit is specifically used to:

[0152] Receive a target network sent by a model training device, wherein the model training device is used to train to obtain the target network.

[0153] In the ninth aspect, an embodiment of the present application provides a model training device, which includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call the computer program to implement the method described in the above-mentioned first aspect, or the second aspect, or any possible implementation of the first aspect, or any possible implementation of the second aspect.

[0154] In the tenth aspect, an embodiment of the present application provides a model usage device, which includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call the computer program to implement the method described in the above-mentioned third aspect, or the fourth aspect, or any possible implementation of the third aspect, or any possible implementation of the fourth aspect.

[0155] In the eleventh aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program runs on a processor, it implements the method described in the above-mentioned first aspect, or the second aspect, or the third aspect, or the fourth aspect, or any possible implementation method of any one of the aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0156] The following is an introduction to the drawings used in the embodiments of the present application.

[0157] Figure 1 is a schematic diagram of an image detection scenario provided by an embodiment of the present application;

[0158] Figure 2 is a schematic diagram of a model distillation scenario provided in an embodiment of the present application;

[0159] Figure 3 It is a schematic diagram of another scenario of model distillation provided in an embodiment of the present application;

[0160] Figure 4 It is a schematic diagram of another scenario of model distillation provided in an embodiment of the present application;

[0161] Figure 5 It is a schematic diagram of another scenario of model distillation provided in an embodiment of the present application;

[0162] Figure 6 It is a schematic diagram of the architecture of a model training provided in an embodiment of the present application;

[0163] Figure 7 It is a schematic diagram of another scenario of model distillation provided in an embodiment of the present application;

[0164] Figure 8 is a schematic diagram of another image detection scenario provided by an embodiment of the present application;

[0165] Fig. 9 is a schematic diagram of another image detection scenario provided by an embodiment of the present application;

[0166] Fig.10 It is a flowchart of a model training method provided in an embodiment of the present application;

[0167] Fig.11 is a schematic diagram of the principle of a Gaussian mask provided in an embodiment of the present application;

[0168] Fig.12 It is a schematic diagram of another scenario of model distillation provided in an embodiment of the present application;

[0169] Fig.13 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0170] Fig.14 It is a structural schematic diagram of another model training device provided in an embodiment of the present application;

[0171] Fig.15 is a structural schematic diagram of an image detection device provided in an embodiment of the present application;

[0172] Fig.16 is a structural schematic diagram of another image detection device provided in an embodiment of the present application;

[0173] Fig.17 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0174] Fig.18 It is a structural schematic diagram of another model training device provided in an embodiment of the present application;

[0175] Fig.19 It is a structural schematic diagram of a model using device provided in an embodiment of the present application;

[0176] Fig. 20 It is a structural schematic diagram of another model using device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0177] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0178] In the fields of artificial intelligence and computer vision, image detection is usually involved, that is, identifying objects (or targets) in images. Image detection is usually achieved through neural networks. When faced with the technical problem of how to train a neural network with faster detection speed and more accurate detection results for target detection, model distillation can be used to solve the problem, such as Figure 2 As shown, specifically, the large neural network guides the training of the small neural network, or the small neural network simulates the performance of the large neural network, thereby giving the small neural network the more accurate performance of the large neural network's detection results, so that the small neural network has the advantages of both fast detection speed and more accurate detection results.

[0179] The following are some examples of how to use a large neural network to guide the training of a small neural network.

[0180] For example, Figure 3 As shown, the feature information of the image is extracted through the feature extraction layer of the large neural network, and then sent to the classification layer of the large neural network to generate classification soft labels; and the feature information of the image is extracted through the feature extraction layer of the small neural network, and then sent to the classification layer of the small neural network to generate classification soft labels; then the classification loss is determined by the classification soft labels generated by the two networks; and then the classification loss is used to guide the training of the classification layer of the small neural network. However, in this scheme, the guidance of the large neural network to the small neural network is not comprehensive enough (such as ignoring the guidance of the feature extraction layer), nor is it refined enough, so the effect after guidance is not ideal.

[0181] For example, Figure 4 As shown, the information of the feature extraction layer of the large neural network is passed to the small neural network, and then the small neural network with the information of the feature extraction layer of the large neural network is compressed to obtain a new thinner and deeper small neural network. Figure 4 W T is the model weight of the large neural network (teacher network), different layers can have different weights, for example, wait; Figure 4 W Sis the model weight of the small neural network (student network), different layers can have different weights, for example, However, this solution requires the construction of a new small neural network, and the number of layers of the newly constructed small neural network becomes more, which has a certain degree of complexity. In addition, in this solution, the feature extraction layer extracts features from the entire image, and there is a lot of background noise, which leads to irrational detection results.

[0182] For example, Figure 5 As shown in the figure, the area near the target area in the image is selected as the distillation area, and the small neural network is allowed to learn the feature extraction layer expression of the large neural network in the distillation area. However, there is still a lot of useless background noise in the distillation area, so the learned feature extraction layer expression is still not ideal; and the small neural network only learns the expression of the feature extraction layer, so the performance improvement is limited.

[0183] In view of the fact that there are still many limitations in guiding the training of small neural networks by large neural networks, the embodiments of the present application further provide relevant architectures, devices, and methods to further improve the effect of guiding the training of small neural networks by large neural networks.

[0184] Please parameter Figure 6 , Figure 6 It is a schematic diagram of the architecture of a model training provided in an embodiment of the present application, and the architecture includes a model training device 601 and one or more model using devices 602. The model training device 601 communicates with the model using device 602 via wired or wireless means, so that the model training device 601 can send the trained model (or network) for predicting the target object in the image to the model using device 602; accordingly, the model using device 602 predicts the target object in the image to be predicted by using the received model.

[0185] Optionally, the model-using device 602 can feed back the results predicted by the model to the above-mentioned model training device 601, so that the model training device 601 can further train the model based on the prediction results of the model-using device 602; the retrained model can be sent to the model-using device 602 to update the original model.

[0186] The model training device 601 may be a device with strong computing power, for example, a server, or a server cluster composed of multiple servers. The model training device 601 may include many neural networks, wherein a neural network with more layers can be called a large neural network relative to a neural network with fewer layers, and a neural network with fewer layers can be called a small neural network relative to a neural network with more layers, that is, the depth of the first network is greater than the depth of the second network.

[0187] like Figure 7 As shown, the model training device 601 includes a first network 701 and a second network 702. The first network 701 can be a neural network larger than the second network 702. The first network 701 and the second network 702 both include a feature extraction layer (also called a feature layer) and a classification layer (also called a classifier, or a classification head), wherein the feature extraction layer is used to extract feature information in an image, and the classification layer is used to classify target objects in an image based on the extracted feature information.

[0188] In the embodiment of the present application, the first network 701 can be used as a teacher network, and the second network 702 can be used as a student network. The first network 701 guides the training of the second network 702. This process can be called distillation. In the embodiment of the present application, the idea of ​​the first network 701 guiding the second network 702 includes the following three technical points:

[0189] 1. Feature information is extracted respectively through the feature extraction layer of the first network 701 and the feature extraction layer of the second network 702, and then the feature information about the target object in the feature information extracted by the feature extraction layers of the two networks is highlighted through a Gaussian mask; feature loss is determined through the feature information about the target object extracted by the first network and the feature information about the target object extracted by the second network, and then the feature loss is used to guide the training of the feature extraction layer of the second network 702.

[0190] 2. The first network 701 and the second network 702 select the same region proposal set. For example, by sharing a region proposal network (RPN), the first network 701 and the second network 702 both have a region proposal set. Therefore, the first network 701 and the second network 702 can generate soft labels based on the same region proposal, and then obtain a binary cross entropy loss (BCEloss) based on the soft labels generated by the first network 701 and the soft labels generated by the second network 702. Then, the binary cross entropy loss (BCEloss) is used to guide the training of the classification layer of the second network 702.

[0191] 3. Use progressive distillation to guide the training of the second network 702. For example, if the first network 701 is a neural network with 101 layers (res101) and the second network 702 is a neural network with 50 layers (res50), then based on the technical points of the first and / or second items, the second network 702 is trained through the first network 701 to obtain the target neural network (which can be marked as res101-50), and then the target neural network (res101-50) is further trained through the third neural network. The principle of training the target neural network through the third neural network is the same as the principle of training the second network 702 through the first network 701, and will not be repeated here. The third neural network here is a neural network larger than the first network 701. For example, the third neural network is a neural network with 152 layers (res152).

[0192] The implementation of the above items 1, 2 and 3 will be described in more detail in the subsequent method embodiments.

[0193] The model uses device 602, which is a device that needs to recognize (or detect) images, such as handheld devices (e.g., mobile phones, tablet computers, PDAs, etc.), vehicle-mounted devices (e.g., cars, bicycles, electric vehicles, airplanes, ships, etc.), wearable devices (e.g., smart watches (such as iWatch, etc.), smart bracelets, pedometers, etc.), smart home devices (e.g., refrigerators, televisions, air conditioners, electric meters, etc.), smart robots, workshop equipment, etc.

[0194] The following uses the model usage device 602 as a car and a mobile phone as examples for explanation.

[0195] The realization of unmanned or computer-driven cars is a very popular topic at present. With the development of the economy, the number of cars in the world is increasing, and road congestion and driving accidents have caused great losses to people’s property and social property. Human factors are the main factor causing traffic accidents. How to reduce human errors, intelligent obstacle avoidance and reasonable planning are important issues to improve driving safety. The emergence of autonomous driving has made all this possible. It can perceive the surrounding environment and navigate without human operation. At present, major companies around the world have begun to pay attention to and develop autonomous driving systems, such as Google, Tesla, Baidu, etc. Since camera equipment has the advantages of low price and easy use, building a perception system based on visual perception is the research and development direction of many companies. For example Figure 8As shown, it is a road scene captured by the on-board camera during the autonomous driving process. The visual perception system on the car acts as the human eye (i.e., computer vision) and automatically detects the image captured by the camera through a detection network (for example, the target network, the first network, the second network, the third network, the fourth network, etc. mentioned in the subsequent method embodiments. The detection network can also be called a classification network, or a detection model, or a detection module, etc.), thereby determining the objects and positions around the car (i.e., detecting the target objects in the image).

[0196] For example, a car uses a detection network to identify objects in an image. If a person is found in the image and is close to the car, the car can slow down or stop to avoid casualties. If other cars are found in the image, the car's speed can be appropriately controlled to avoid rear-end collisions. If an object is found in the image and is rapidly hitting the car, the car can be controlled to avoid the object by shifting or changing lanes.

[0197] For another example, a car can identify objects in an image through a detection network. If it finds traffic routes on the road (such as double yellow lines, single yellow lines, lane dividing lines, etc.), it can predict the car's driving status. If it predicts that the car may cross the line, the car can be controlled accordingly to avoid crossing the line. Of course, it can also identify lane dividing lines and their positions and use this information to decide how to change lanes. The control of other traffic routes is similar.

[0198] For example, a car uses a detection network to identify objects in an image, and then uses the identified object as a target to calculate the car's speed, acceleration, turning angle and other information.

[0199] Mobile phones were originally used as a communication tool to facilitate communication. With the development of the global economy and the improvement of people's quality of life, people's pursuit of mobile phone experience and performance has become higher and higher. In addition to entertainment, navigation, shopping, and taking pictures, detection and recognition functions have also received a lot of attention. At present, the recognition technology for detecting target objects in images has been applied in many mobile phone apps. Developers only need to call the authorized mobile SDK package for face detection, face key point detection, and face analysis to automatically identify the identity of faces in photos and videos (that is, detect target objects in images). Fig. 9As shown, it is a face picture collected by the mobile phone camera. The mobile phone locates and identifies the objects in the picture through a detection network (for example, the target network mentioned in the subsequent method embodiment, which can also be called a detection model, or a detection module, or a detection and recognition system, etc.), finds the face (and obtains the face position), and determines whether the face in the image is the face of a specific person (i.e., the computer perspective). If it is, the next step of operation can be performed, such as mobile payment, face login and other functions. The user-related information used in the embodiments of this application belongs to information obtained within the legal scope.

[0200] Devices (such as cars, mobile phones, etc.) perceive the surrounding environmental information through the computer perspective, and then perform corresponding intelligent control based on the environmental information. Intelligent control based on the computer perspective is a realization of artificial intelligence.

[0201] See also Fig.10 , Fig.10 is a flow chart of a model training method provided in an embodiment of the present application. The method can be based on Figure 6 The model training system shown is implemented, and the method includes but is not limited to the following steps:

[0202] Step S1001: The model training device extracts feature information in the target image through the feature extraction layer of the first network and the feature extraction layer of the second network respectively.

[0203] Specifically, the first network is used as a teacher network, and the second network is used as a student network. In the process of the first network guiding the training of the second network, that is, the model distillation process, the feature extraction layer of the first network and the feature extraction layer of the second network extract feature information for the same image. For the convenience of description, the same image can be called a target image.

[0204] Optionally, the layers of the first network are greater than the layers of the second network. For example, the first network may be a neural network with 101 layers (res101), and the second network may be a neural network with 50 layers (res50).

[0205] The feature information in the embodiments of the present application may be represented by a vector or other machine-recognizable manner.

[0206] Step S1002: The model training device highlights the features about the target object in the first feature information through a Gaussian mask.

[0207] The inventors of the present application have found that in the process of target object detection (i.e., recognition) based on neural networks, the gain in detection performance largely comes from the feature information extracted by the feature extraction layer (backbone layer), so the simulation of the feature extraction layer is an important part of model training. In the embodiment of the present application, the introduction of the Gaussian mask is actually a process of highlighting the features of the target object in the target image and suppressing the features of the background outside the target object; it is actually a process of highlighting the response to the target object and weakening the edge information. It should be noted that the use of the Gaussian mask can not only suppress the features of the background outside the rectangular frame where the target object is located (generally the smallest rectangular frame that can frame the target object), but also suppress the features of the background other than the target object in the rectangular frame, thereby maximizing the highlighting of the features of the target object.

[0208] like Fig.11 As shown in the figure, the Gaussian mask can highlight the person (i.e., the target object) in the target image and weaken the features of the background other than the person. This is reflected in the model data of the Gaussian mask in that the feature information corresponding to the target object forms a larger protrusion in the coordinates, and the height of the background information in the coordinates is close to zero, which is approximately on a plane with a height of zero.

[0209] For ease of understanding, the following is an example of the Gaussian mask definition for the target image, as shown in Formula 1-1:

[0210]

[0211] In formula 1-1, (x, y) is the coordinate of the pixel point in the target image, B is a positive region proposal of the target object in the target image, the geometric specifications of the positive region proposal B are w×h, the coordinates of the center point of the positive region proposal B are (x0, y0), σ x and σ y are the attenuation factors on the x-axis and y-axis respectively. Optional, for convenience, you can set σ x =σ y , this Gaussian mask is only valid for the target true value frame, and all background outside the frame is filtered out. When there are multiple positive region proposals about the target object, the pixel point (x, y) in the target object may have multiple Gaussian mask values ​​(corresponding to different positive region proposals). At this time, the maximum value of the multiple Gaussian mask values ​​can be selected as the Gaussian mask value M(x, y) of the pixel point (x, y). The Gaussian mask value M(x, y) can be expressed by formula 1-2:

[0212]

[0213] In formula 1-2, N p The number of positive region proposals for the target object in the target image, M1 (x, y) is the value of the Gaussian mask of the midpoint (x, y) of the first positive region proposal, M 2 (x,y) is the value of the Gaussian mask of the point (x,y) in the second positive region proposal. It is the Nth p The value of the Gaussian mask of the point (x, y) in the positive region proposal, and so on. From formula 1-2, we can see that the value of the Gaussian mask of a pixel point (x, y) in the target object takes the maximum value among multiple values.

[0214] For the sake of ease of description, the feature information in the target image extracted by the feature extraction layer of the first network is called the first feature information, and the feature about the target object in the first feature information highlighted by the Gaussian mask is called the first local feature.

[0215] Step S1003: The model training device highlights the features about the target object in the second feature information through a Gaussian mask.

[0216] For the sake of ease of description, the feature information in the target image extracted by the feature extraction layer of the second network is called second feature information, and the feature about the target object in the second feature information highlighted by the Gaussian mask is called second local feature.

[0217] Step S1004: The model training device determines feature loss through the first local feature and the second local feature.

[0218] In the embodiment of the present application, the first local feature is a feature of a target object in a target image obtained by the first network, and the second local feature is a feature of a target object in a target image obtained by the second network. The difference between the first local feature and the second local feature can reflect the difference between the feature extraction layer of the first network and the feature extraction layer of the second network. The feature loss (also called distillation loss) in the embodiment of the present application can reflect the difference between the second local feature and the first local feature.

[0219] Optionally, the first local feature can be expressed as The second local feature can be expressed as The feature loss L can be calculated by formula 1-3 b :

[0220]

[0221] In the embodiment of the present application, A is introduced to realize the normalization operation; the size of the target image is WxH, M ijRepresents the value of the Gaussian mask of a pixel in the target image. i can take values ​​from 1 to W, and j can take values ​​from 1 to H. The value of the Gaussian mask of any pixel (i, j) in the target image is M. ij It can be calculated by the above formula 1-1 and formula 1-2, which will not be repeated here. In addition, Represents the features of the pixel (i, j) extracted by the second network, Represents the features of the pixel point (i, j) extracted by the first network; C represents the number of channels of the feature map when the first network and the second network extract feature information in the target image.

[0222] Step S1005: The model training device generates a classification prediction value of the target region proposal in the region proposal set through the classification layer of the first network.

[0223] Specifically, a corresponding algorithm or strategy may be used to ensure that both the first network and the second network have the region proposal set. For example, the first network and the second network share an RPN so that both the first network and the second network have the region proposal set. For example, the shared RPN may include 2000 region proposals, and the region proposal set includes 512 region proposals from the 2000 region proposals. The same detector may be configured in the first network and the second network so that the first network and the second network can extract the same 512 region proposals, i.e., the region proposal set, from the 2000 shared region proposals. Fig.12 The figure shows a flow chart of enabling the first network and the second network to select the same region proposal by sharing the RPN and configuring the same detector. All the selected region proposals are collectively referred to as a region proposal set.

[0224] The RPN is an RPN shared by the first network and the second network, and may be shared by the second network to the first network, or shared by the first network to the second network, or shared in other ways.

[0225] In an optional solution, the target region proposal is all region proposals in the region proposal set, i.e., including positive region proposals of the target object and negative region proposals of the target object. Optionally, which ones in the region proposal set are positive region proposals and which ones are negative region proposals can be pre-marked manually or automatically marked by a machine; the usual classification criterion is that if the overlap between a region proposal and the rectangular box where the target object is located (usually the smallest rectangular box that frames the target object) exceeds a set reference threshold (for example, it can be set to 50%, or other values), then the region proposal can be classified as a positive region proposal of the target object, otherwise the region proposal is classified as a negative region proposal of the target object.

[0226] In yet another optional solution, the target region proposal is a positive region proposal belonging to the target object in the region proposal set.

[0227] In the embodiment of the present application, the classification prediction value of the target region proposal in the region proposal set generated by the classification layer of the first network can be referred to as the first classification prediction value to facilitate subsequent description.

[0228] Step S1006: The model training device generates a classification prediction value of the target region proposal in the region proposal set through the classification layer of the second network.

[0229] In the embodiment of the present application, the classification prediction value of the target region proposal in the region proposal set generated by the classification layer of the second network can be referred to as the second classification prediction value to facilitate subsequent description.

[0230] Both the first classification prediction value and the second classification prediction value are used to characterize the classification tendency or probability of the corresponding region proposal. For example, the classification prediction value of region proposal 1 generated by the classification label of the first network represents that the probability of the object in region proposal 1 being classified as a person is 0.8, the probability of being classified as a tree is 0.3, and the probability of being classified as a car is 0.1. It should be noted that the classification prediction values ​​obtained by the classification layers in different networks for the same region proposal classification may be different, because the model parameters of different networks are generally different, and their prediction capabilities generally differ.

[0231] Step S1007: The model training device determines the classification loss according to the first classification prediction value and the second classification prediction value.

[0232] Specifically, the embodiment of the present application selects the same set of region proposals so that the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value, and actually uses the first prediction value as a soft label to determine the classification loss of the classification layer of the second network. In this way, the loss of the second network relative to the first network can be maximized, so the model training effect is better.

[0233] In an optional solution, the classification loss satisfies the relationship shown in Formula 1-4:

[0234]

[0235] In formula 1-4, K is the total number of region proposals in the region proposal set, N pis the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0236] Step S1008: The model training device trains the second network according to the feature loss and the classification loss to obtain a target network.

[0237] The meaning of training the second network according to the feature loss and the classification loss to obtain the target network mentioned in the embodiment of the present application is: the target network is obtained by training the second network, and the parameters used in the training process include but are not limited to the feature loss and the classification loss, that is, other parameters besides these two items may also be used; and in the training process, only the feature loss and the classification loss obtained based on the first network may be used (that is, the target network is obtained by distilling the second network based on the first network), or not only the feature loss and the classification loss obtained based on the first network may be used, but also information obtained based on other networks (one or more) may be used (that is, the target network is obtained by distilling the second network based on the first network and other networks).

[0238] Optionally, in the process of training the second network according to the feature loss and the classification loss, the feature loss L b and the classification loss L cls Determine the total loss L, and then train the second network by the total loss. Optionally, a portion of the model parameters in the second network (e.g., the model parameters of the feature extraction layer) may be trained by feature loss, and another portion of the model parameters in the second network (e.g., the model parameters of the classification layer) may be trained by classification loss. Of course, feature loss and classification loss may also be used to train the second network in other ways.

[0239] For the case of training the second network through the total loss L, two optional cases of calculating the total loss are listed below.

[0240] Case 1: Calculate the total loss L using Formula 1-5.

[0241] L=δL b +L cls Formula 1-5

[0242] In formula 1-5, δ is a preset or pre-trained weight balancing factor.

[0243] Case 2: According to the true value label of the region proposal in the target image and the predicted value of the region proposal predicted by the second network in the target image, the regression loss and RPN loss of the second network are determined; that is, the second network is trained without relying on the first network to obtain the regression loss L reg and RPN loss L rpn , and then combined with the regression loss L reg , RPN loss L rpn , the above feature loss L b and classification loss L cls The total loss L is obtained. Optionally, the total loss L is calculated as shown in Formula 1-6.

[0244] L=δL b +L cls +L reg +L rpn Formula 1-6

[0245] In the embodiment of the present application, the order of determining the classification loss and the feature loss is not limited, and they can be executed simultaneously, or the process of determining the classification loss can be executed first, or the process of determining the feature loss can be executed first.

[0246] In an optional solution, the model training device trains the second network according to the feature loss and the classification loss, and the obtained network is not the final target network, but an intermediate network. The intermediate network will be trained again through other networks (such as a third network) in the future. This process can be regarded as using progressive distillation for the second network. The principle is as follows: after the intermediate network is obtained by distilling the second network based on the above-mentioned first network (that is, the feature loss and classification loss are obtained according to the first network and the second network, and then the second network is trained according to the feature loss and classification loss), the intermediate network can be further distilled again through a third network with more layers than the first network. The principle of distilling the intermediate network through the third network is the same as the principle of distilling the second network through the first network; subsequently, the newly trained network can be further distilled using networks with more layers until the distillation of the second network reaches the expected goal, thereby obtaining the target network.

[0247] For example, if the first network 701 is a neural network with 101 layers (res101) and the second network 702 is a neural network with 50 layers (res50), then based on the technical points of the first and / or second items, the second network 702 is distilled through the first network 701 to obtain an intermediate neural network (which can be marked as res101-50), and then the intermediate neural network (res101-50) is further distilled through the third neural network (i.e., the second network 702 is distilled through the first network 701 and the third network in sequence). The principle of distilling the intermediate neural network through the third neural network is the same as the principle of distilling the second network 702 through the first network 701, and will not be repeated here. The third neural network is a neural network larger than the first network 701, for example, a neural network with 152 layers (res152).

[0248] It should be noted that in Fig.10 In the method embodiment shown, one, two, or three of the three technical points listed above can be used. For example, the technical point of progressive distillation (the first of the three technical points mentioned above) can be used, but no special restrictions are made on how to extract features and how to use region proposals (i.e., no restrictions are made on whether to use the second and third of the three technical points mentioned above).

[0249] Step S1009: The model training device sends the target network to the model using device.

[0250] Step S1010: The model-using device receives the target network sent by the model training device.

[0251] Specifically, after the model uses the device to receive the target network, it predicts (or detects, or estimates) the content in the image (i.e., identifies the target in the image) through the target network. For example, it identifies whether there is a face in the image, and if so, where the face is located in the image; or it identifies whether there is a road obstacle in the image, and if so, where the obstacle is located in the image, etc. For specific usage scenarios, please refer to Figure 6 The architecture shown is an introduction to the model using device 602.

[0252] In order to verify the effect of the above embodiment, the inventors of the present application conducted verification on two standard detection data sets, namely, COCO2017 data set and BDD100k data set. The COCO data set contains 80 object categories, 110,000 training images and 5,000 verification images, and the BDD100k data set contains 10 categories, with a total of 100,000 images. Both data sets were evaluated using the coco evaluation criteria, namely, the category average accuracy (mAP).

[0253] Table 1 shows different distillation strategy solutions, in which the networks (or models) with layers of res18, res50, res101, and res152 have been pre-trained on the COCO dataset, and only the above embodiment needs to be used for distillation.

[0254] Table 1 Different distillation strategies

[0255] Strategy Number Teacher Network Student Network Distilled Student Network 1 res50 res18 res50-18 2 res101 res18 res101-18 3 res101 res50-18 res101-50-18 4 res152 res101-50-18 res152-101-50-18

[0256] In Table 1, res50-18 represents a network obtained by distilling a 18-layer network through a 50-layer network; res101-18 represents a network obtained by distilling a 18-layer network through a 101-layer network; res101-50-18 represents a network obtained by further distilling the network res101-18 obtained by distilling through a 101-layer network; res152-101-50-18 represents a network obtained by further distilling the network res101-50-18 obtained by distilling through a 152-layer network. Optionally, network res50 can be considered as the first network, network res18 can be considered as the second network, network 101 can be considered as the third network, and network res152 can be considered as the fourth network, which is a network with more layers than the third network. The network res152-101-50-18 obtained after the second network is distilled through the first network, the third network, and the fourth network in sequence can be sent to the above-mentioned model using device to detect targets in the image.

[0257] Table 2 shows the evaluation results of different networks on the COCO dataset, among which the detection accuracy of network Res50-18 has been significantly improved by 2.8 percentage points compared with the original network res18, and the detection accuracy of network res101-18 has been improved by 3.2 points compared with the accuracy of network res18. The network res101-50-18 obtained by the progressive distillation method has been further improved than the network re50-18 obtained by single distillation. It is worth mentioning that the detection accuracy of network res152-101-50-18 has been improved by 4.4 percentage points compared with network res18, and the mAP after distillation has reached 0.366, which has surpassed the accuracy of network res50 detection of 0.364. In other words, although network res18 has fewer network layers than network res50, and the original mAP differs a lot by 4.2 percentage points, the method of the embodiment of the present application performs progressive distillation on network res18, so that the distilled network res18 exceeds the performance of network res50.

[0258] Table 2 Performance evaluation results of different networks on the COCO dataset

[0259] network MAP AP50 AP75 Aps(small) Apm(Medium) Apl (large) res18 0.322 0.534 0.34 0.183 0.354 0.411 res50 0.364 0.581 0.393 0.216 0.398 0.468 res101 0.385 0.602 0.417 0.225 0.429 0.492 res152 0.409 0.622 0.448 0.241 0.453 0.532 res50-18 0.35 0.56 0.373 0.187 0.384 0.459 res101-18 0.354 0.563 0.38 0.186 0.387 0.473 res101-50-18 0.358 0.567 0.387 0.184 0.392 0.479 res152-101-50-18 0.366 0.574 0.396 0.184 0.399 0.5

[0260] In Table 2, MAP is the mean average precision, AP50 is the mean precision when the detection evaluation function (Intersection over Union, IOU) is greater than 0.5, AP75 is the mean precision when IOU is greater than 0.75, Aps is the mean precision of small objects, Apm is the mean precision of medium objects, and Apl is the mean precision of large objects.

[0261] As shown in Table 3. Network res50 and network res101 are used as teacher networks, network res18 is used as student network, +1 means only the first technical point mentioned above is used (i.e., Gaussian mask is used to highlight the target object), +2 means the second technical point mentioned above is used (i.e., the same set of region proposals is selected), and +3 means the third technical point mentioned above is used (i.e., progressive distillation). The student network res18 in network res18(+1) and network res18(+1+2) is not pre-trained on the COCO dataset, while the student network res18 in res18(+1+2+3) is pre-trained on COCO, which is conducive to narrowing the difference between it and the teacher network, which is equivalent to a progressive distillation solution. It can be seen that with continuous improvement, the effect of distillation is gradually improving, which also proves the effectiveness of the above three technical points.

[0262] Table 3 Experimental results of each technical point

[0263] res18 res18(+1) res18(+1+2) res18(+1+2+3) Teacher network res50 0.322 0.342 0.344 0.35 Teacher network res101 0.322 0.347 0.349 0.354

[0264] In order to verify the applicability of this application on different data, a comparative experiment was also conducted on the BDD100k dataset, and the results are shown in Table 4. There is a 2.1 percentage point mAP (accuracy) gap between the original network res18 (as a student network) and the network res50 (as a teacher network). After distillation using the method of the embodiment of this application, the distilled network res50-18 has an improvement of 1.5 percentage points in detection mAP (accuracy) over the original network res18, and is only 0.6 percentage points behind the teacher network res50, making up for nearly 75% of the mAP (accuracy) gap, and the effect is very obvious.

[0265] Table 4 Performance evaluation results on the BDD100k dataset

[0266] network mAP AP50 AP75 Aps Apm Apl res18 0.321 0.619 0.289 0.159 0.374 0.513 res50 0.342 0.644 0.314 0.171 0.399 0.546 res50-18 0.336 0.636 0.309 0.166 0.393 0.539

[0267] exist Fig.10 In the described method, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0268] Furthermore, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0269] Furthermore, after the second network is trained through the first network, further training the trained second network using a third network with more layers can stably improve the performance of the second network.

[0270] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0271] See also Fig.13 , Fig.13 It is a structural schematic diagram of a model training device 130 provided in an embodiment of the present application. The model training device 130 may be the model training device in the above method embodiment or a device in the model training device. The model training device 130 may include a feature extraction unit 1301, a first optimization unit 1302, a second optimization unit 1303, a first determination unit 1304 and a weight adjustment unit 1305, wherein each unit is described in detail as follows.

[0272] A feature extraction unit 1301 is used to extract first feature information in a target image through a feature extraction layer of a first network;

[0273] The feature extraction unit 1301 is further used to extract second feature information in the target image through a feature extraction layer of a second network, wherein both the first network and the second network are classification networks, and the depth of the first network is greater than the depth of the second network;

[0274] A first optimization unit 1302 is configured to extract features about the target object in the first feature information by using a Gaussian mask to obtain a first local feature;

[0275] A second optimization unit 1303 is configured to extract features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature;

[0276] A first determining unit 1304, configured to determine a feature loss by using the first local feature and the second local feature;

[0277] The weight adjustment unit 1305 is used to train the second network according to the feature loss to obtain a target network.

[0278] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0279] In a possible implementation manner, the device further includes:

[0280] A first generating unit, configured to generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of the first network;

[0281] A second generating unit, configured to generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of the second network;

[0282] A second determining unit, configured to determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0283] The weight adjustment unit is specifically used to: train the second network according to the feature loss and the classification loss to obtain a target network.

[0284] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0285] In a possible implementation manner, when training the second network according to the feature loss to obtain a target network, the weight adjustment unit is specifically used to:

[0286] training the second network according to the feature loss;

[0287] The trained second network is trained through a third network to obtain a target network, wherein the depth of the third network is greater than the depth of the first network.

[0288] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0289] In a possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0290] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0291] In one possible implementation, or any possible implementation of the sixth aspect, in another possible implementation, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal in the region proposal set that belongs to the target object.

[0292] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0293]

[0294] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0295] In a possible implementation manner, the device further includes:

[0296] A third determining unit is used to determine the regression loss and the RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image;

[0297] The weight adjustment unit is specifically used to: train the second network according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

[0298] In a possible implementation manner, the device further includes:

[0299] A sending unit is used to send the target network to a model using device after the weight adjustment unit trains the second network according to the feature loss to obtain the target network, wherein the target network is used to predict the content in the image.

[0300] It should be noted that the implementation and beneficial effects of each unit can also refer to Fig.10 The corresponding description of the method embodiment shown.

[0301] See also Fig.14 , Fig.14 It is a structural schematic diagram of a model training device 140 provided in an embodiment of the present application. The model training device 140 may be the model training device in the above method embodiment or a device in the model training device. The model training device 140 may include a first training unit 1401 and a second training unit 1402, wherein each unit is described in detail as follows.

[0302] A first training unit 1401 is used to train a second network based on the first network to obtain an intermediate network;

[0303] The second training unit 1402 is used to train the intermediate network based on the third network to obtain a target network, wherein the first network, the second network and the third network are all classification networks, and the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network.

[0304] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0305] In a possible implementation, training the second network based on the first network to obtain the intermediate network includes:

[0306] Extracting first feature information in the target image through a feature extraction layer of the first network;

[0307] Extracting second feature information in the target image through a feature extraction layer of a second network;

[0308] Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature;

[0309] Extracting features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature;

[0310] Determine a feature loss by using the first local feature and the second local feature;

[0311] The second network is trained according to the feature loss to obtain the intermediate network.

[0312] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0313] In a possible implementation manner, the device further includes:

[0314] A first generating unit, configured to generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of a first network;

[0315] A second generating unit, configured to generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of a second network;

[0316] A second determining unit, configured to determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0317] The step of training the second network according to the feature loss to obtain the intermediate network is specifically as follows:

[0318] The second network is trained according to the feature loss and the classification loss to obtain the intermediate network.

[0319] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0320] In a possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0321] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0322] In a possible implementation manner, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

[0323] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0324]

[0325] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0326] In a possible implementation manner, the device further includes:

[0327] A third determining unit is used to determine the regression loss and the RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image;

[0328] The weight adjustment unit is specifically used to: train the second network according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

[0329] In a possible implementation, the device further includes:

[0330] A sending unit is used to send the target network to a model using device after the weight adjustment unit trains the second network according to the feature loss to obtain the target network, wherein the target network is used to predict the content in the image.

[0331] It should be noted that the implementation and beneficial effects of each unit can also refer to Fig.10 The corresponding description of the method embodiment shown.

[0332] See also Fig.15 , Fig.15 It is a structural schematic diagram of an image detection device 150 provided in an embodiment of the present application. The image detection device 150 can be a model using device in the above method embodiment or a device in the model using device. The image detection device 150 may include an acquisition unit 1501 and a recognition unit 1502, wherein each unit is described in detail as follows.

[0333] An acquisition unit 1501 is used to acquire a target network, wherein the target network is a network obtained by training the second network through the first network, the parameters used for training the second network through the first network include feature loss, the feature loss is determined according to a first local feature and a second local feature, the first local feature is a feature about a target object extracted from first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from second feature information through a Gaussian mask, the first feature information is feature information in a target image extracted through a feature extraction layer of the first network, the second feature information is feature information in the target image extracted through a feature extraction layer of the second network, the first network and the second network are both classification networks, and the depth of the first network is greater than the depth of the second network;

[0334] The recognition unit 1502 is used to recognize the content in the image through the target network.

[0335] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0336] In a possible implementation, the parameters used to train the second network also include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value is a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

[0337] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0338] In a possible implementation, the target network is specifically a network obtained by training the second network through the first network and further training the trained network through a third network, wherein the depth of the third network is greater than the depth of the first network.

[0339] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0340] In a possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0341] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0342] In a possible implementation manner, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

[0343] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0344] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0345] In a possible implementation, the parameters used to train the second network also include a regression loss and an RPN loss of the second network, wherein the regression loss and the RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the region proposal predicted by the second network in the target image.

[0346] In a possible implementation manner, the acquiring unit is specifically configured to:

[0347] Receive a target network sent by a model training device, wherein the model training device is used to train to obtain the target network.

[0348] It should be noted that the implementation and beneficial effects of each unit can also refer to Fig.10The corresponding description of the method embodiment shown.

[0349] See also Fig.16 , Fig.16 It is a structural schematic diagram of an image detection device 160 provided in an embodiment of the present application. The image detection device 160 can be a model using device in the above method embodiment or a device in the model using device. The image detection device 160 may include an acquisition unit 1601 and a recognition unit 1602, wherein each unit is described in detail as follows.

[0350] An acquisition unit 1601 is used to acquire a target network, wherein the target network is a network obtained by training a second network through multiple network iterations, the multiple networks are all classification networks, the multiple networks include at least a first network and a third network, the third network is used to train the intermediate network after the first network trains the second network to obtain the intermediate network, wherein the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network;

[0351] The recognition unit 1602 is used to recognize the content in the image through the target network.

[0352] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0353] In a possible implementation, the parameters used by the first network when training the second network include feature loss, wherein the feature loss is determined based on a first local feature and a second local feature, the first local feature is a feature about a target object extracted from first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from second feature information through a Gaussian mask, the first feature information is feature information in a target image extracted through a feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through a feature extraction layer of the second network.

[0354] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0355] In a possible implementation, the parameters used by the first network when training the second network include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value is a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

[0356] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0357] In one possible implementation manner, or any possible implementation manner of the eighth aspect, in another possible implementation manner, the first network and the second network share a region proposal network (RPN) so that the first network and the second network both have the region proposal set.

[0358] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0359] In a possible implementation manner, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

[0360] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0361] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0362] In a possible implementation, the parameters used to train the second network also include a regression loss and an RPN loss of the second network, wherein the regression loss and the RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the region proposal predicted by the second network in the target image.

[0363] In a possible implementation manner, the acquiring unit is specifically configured to:

[0364] Receive a target network sent by a model training device, wherein the model training device is used to train to obtain the target network.

[0365] It should be noted that the implementation and beneficial effects of each unit can also refer to Fig.10 The corresponding description of the method embodiment shown.

[0366] See also Fig.17 , Fig.17 It is a structural diagram of a model training device 170 provided in an embodiment of the present application. The model training device 170 includes a processor 1701, a memory 1702 and a communication interface 1703. The processor 1701, the memory 1702 and the communication interface 1703 are interconnected via a bus.

[0367] The memory 1702 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and the memory 1702 is used for related computer programs and data. The communication interface 1703 is used to receive and send data.

[0368] The processor 1701 may be one or more central processing units (CPUs). When the processor 1701 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0369] The processor 1701 in the model training device 170 is used to read the computer program code stored in the memory 1702 and perform the following operations:

[0370] Extracting first feature information in the target image through a feature extraction layer of the first network;

[0371] Extracting second feature information in the target image through a feature extraction layer of a second network;

[0372] Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature;

[0373] Extracting features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature;

[0374] Determine a feature loss by using the first local feature and the second local feature;

[0375] The second network is trained according to the feature loss to obtain a target network.

[0376] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0377] In a possible implementation manner, the processor is further configured to:

[0378] Generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of the first network;

[0379] Generating, by the classification layer of the second network, a second classification prediction value of the target region proposal in the region proposal set;

[0380] Determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0381] The step of training the second network according to the feature loss to obtain a target network includes:

[0382] The second network is trained according to the feature loss and the classification loss to obtain a target network.

[0383] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0384] In a possible implementation manner, in terms of training the second network according to the feature loss to obtain the target network, the processor is specifically configured to:

[0385] training the second network according to the feature loss;

[0386] The trained second network is trained through a third network to obtain a target network, wherein the depth of the third network is greater than the depth of the first network.

[0387] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0388] In a possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0389] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0390] In a possible implementation manner, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

[0391] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0392] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0393] In a possible implementation manner, the processor is further configured to:

[0394] Determining a regression loss and an RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image;

[0395] Training the second network according to the feature loss and the classification loss to obtain a target network includes:

[0396] The second network is trained according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

[0397] In a possible implementation manner, after training the second network according to the feature loss to obtain a target network, the processor is further configured to:

[0398] The target network is sent to a model-using device via a communication interface 1703 , wherein the target network is used to predict content in an image.

[0399] It should be noted that the implementation of each operation can also refer to Fig.10 The corresponding description of the method embodiment shown.

[0400] See also Fig.18 , Fig.18 It is a structural diagram of a model training device 180 provided in an embodiment of the present application. The model training device 180 includes a processor 1801, a memory 1802 and a communication interface 1803. The processor 1801, the memory 1802 and the communication interface 1803 are interconnected via a bus.

[0401] The memory 1802 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and is used for related computer programs and data. The communication interface 1803 is used to receive and send data.

[0402] The processor 1801 may be one or more central processing units (CPUs). When the processor 1801 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0403] The processor 1801 in the model training device 180 is used to read the computer program code stored in the memory 1802 and perform the following operations:

[0404] The second network is trained based on the first network to obtain an intermediate network. In various embodiments of the present application, training the second network based on the first network is essentially distilling the second network through the first network, and training the second network trained by the first network based on the third network is essentially distilling the second network distilled by the first network through the third network, which are uniformly explained here.

[0405] The intermediate network is trained based on the third network to obtain a target network, wherein the first network, the second network and the third network are all classification networks, and the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network.

[0406] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0407] In a possible implementation manner, in terms of obtaining an intermediate network by training the second network based on the first network, the processor is specifically configured to:

[0408] Extracting first feature information in the target image through a feature extraction layer of the first network;

[0409] Extracting second feature information in the target image through a feature extraction layer of a second network;

[0410] Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature;

[0411] Extracting features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature;

[0412] Determine a feature loss by using the first local feature and the second local feature;

[0413] The second network is trained according to the feature loss to obtain the intermediate network.

[0414] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0415] In a possible implementation manner, the processor 1801 is further configured to:

[0416] Generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of a first network;

[0417] Generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of a second network;

[0418] Determine a classification loss according to the first classification prediction value and the second classification prediction value;

[0419] In terms of training the second network according to the feature loss to obtain the intermediate network, the processor is specifically configured to:

[0420] The second network is trained according to the feature loss and the classification loss to obtain the intermediate network.

[0421] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0422] In a possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0423] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0424] In a possible implementation manner, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

[0425] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0426]

[0427] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0428] In a possible implementation, the method further includes:

[0429] Determining a regression loss and an RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image;

[0430] Training the second network according to the feature loss and the classification loss to obtain a target network includes:

[0431] The second network is trained according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

[0432] In a possible implementation manner, after training the second network according to the feature loss to obtain a target network, the method further includes:

[0433] The target network is sent to a model-using device via a communication interface 1803 , wherein the target network is used to predict content in an image.

[0434] It should be noted that the implementation of each operation can also refer to Fig.10 The corresponding description of the method embodiment shown.

[0435] See also Fig.19 , Fig.19 It is a structural diagram of a model using device 190 provided in an embodiment of the present application. The model using device 190 may also be called an image detection device or other names. The model using device 190 includes a processor 1901, a memory 1902 and a communication interface 1903. The processor 1901, the memory 1902 and the communication interface 1903 are interconnected via a bus.

[0436] The memory 1902 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and the memory 1902 is used for related computer programs and data. The communication interface 1903 is used to receive and send data.

[0437] The processor 1901 may be one or more central processing units (CPUs). When the processor 1901 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0438] The model uses the processor 1901 in the device 190 to read the computer program code stored in the memory 1902 to perform the following operations:

[0439] Acquire a target network, wherein the target network is a network obtained by training the second network through the first network, the parameters used for training the second network through the first network include feature loss, the feature loss is determined according to the first local feature and the second local feature, the first local feature is a feature about the target object extracted from the first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from the second feature information through a Gaussian mask, the first feature information is feature information in the target image extracted through the feature extraction layer of the first network, the second feature information is feature information in the target image extracted through the feature extraction layer of the second network, the first network and the second network are both classification networks, and the depth of the first network is greater than the depth of the second network;

[0440] The target network identifies the content in the image.

[0441] In the above method, the local features of the target object in the feature information extracted by the first network are highlighted by the Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by the Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0442] In a possible implementation, the parameters used to train the second network also include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value is a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

[0443] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0444] In a possible implementation, the target network is specifically a network obtained by training the second network through the first network and further training the trained network through a third network, wherein the depth of the third network is greater than the depth of the first network.

[0445] In this possible implementation, after the second network is trained by the first network, the trained second network is further trained by using a third network with more layers, which can stably improve the performance of the second network.

[0446] In a possible implementation manner, the first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

[0447] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0448] In a possible implementation manner, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

[0449] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0450]

[0451] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0452] In a possible implementation, the parameters used to train the second network also include a regression loss and an RPN loss of the second network, wherein the regression loss and the RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the region proposal predicted by the second network in the target image.

[0453] In a possible implementation manner, in terms of acquiring the target network, the processor is specifically configured to:

[0454] The target network sent by the model training device is received through the communication interface 1903, wherein the model training device is used to train to obtain the target network.

[0455] It should be noted that the implementation of each operation can also refer to Fig.10 The corresponding description of the method embodiment shown.

[0456] See also Fig. 20 , Fig. 20 It is a structural diagram of a model using device 200 provided in an embodiment of the present application. The model using device 200 may also be called an image detection device or other names. The model using device 200 includes a processor 2001, a memory 2002 and a communication interface 2003. The processor 2001, the memory 2002 and the communication interface 2003 are interconnected via a bus.

[0457] The memory 2002 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and the memory 2002 is used for related computer programs and data. The communication interface 2003 is used to receive and send data.

[0458] The processor 2001 may be one or more central processing units (CPUs). When the processor 2001 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0459] The model uses the processor 2001 in the device 200 to read the computer program code stored in the memory 2002 to perform the following operations:

[0460] Acquire a target network, wherein the target network is a network obtained by training the second network through multiple network iterations, the multiple networks are all classification networks, the multiple networks include at least a first network and a third network, the third network is used to train the intermediate network after the first network trains the second network to obtain the intermediate network, wherein the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network;

[0461] The target network identifies the content in the image.

[0462] In the above method, after the second network is trained by the first network, the trained second network is further trained using the third network with more layers, which can stably improve the performance of the second network.

[0463] In a possible implementation, the parameters used by the first network when training the second network include feature loss, wherein the feature loss is determined based on a first local feature and a second local feature, the first local feature is a feature about a target object extracted from first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from second feature information through a Gaussian mask, the first feature information is feature information in a target image extracted through a feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through a feature extraction layer of the second network.

[0464] In this possible implementation, the local features of the target object in the feature information extracted by the first network are highlighted by a Gaussian mask, and the local features of the target object in the feature information extracted by the second network are highlighted, and then the feature loss is determined according to the local features of the target object in the two networks, and the second network is subsequently trained based on the feature loss. The background noise of the image (including the background noise outside the box of the target object and the background noise inside the box) is filtered out by a Gaussian mask, and the feature loss obtained on this basis can better reflect the difference between the second network and the first network. Therefore, training the second network based on the feature loss can make the second network's expression of features closer to the first network, and the model distillation effect is very good.

[0465] In a possible implementation, the parameters used by the first network when training the second network include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value is a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

[0466] In this possible implementation, by selecting the same set of region proposals, the classification layer of the first network and the classification layer of the second network generate classification prediction values ​​based on the same region proposals. When the region proposals are the same, the difference in the prediction values ​​generated by the two networks is generally due to the difference in the model parameters of the two networks. Therefore, the embodiment of the present application determines the classification loss for training the second network based on the difference between the first prediction value and the second prediction value; in this way, the loss of the second network relative to the first network can be maximized. Therefore, training the second model based on the classification loss can make the classification result of the second network closer to the first network, and the model distillation effect is very good.

[0467] In a possible implementation manner, the first network and the second network share a region proposal network RPN so that both the first network and the second network have the region proposal set.

[0468] In a possible implementation manner, the RPN is shared by the second network to the first network, or is shared by the first network to the second network.

[0469] In a possible implementation manner, the target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

[0470] In a possible implementation, the classification loss L cls Satisfies the following relationship:

[0471]

[0472] Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

[0473] In a possible implementation, the parameters used to train the second network also include a regression loss and an RPN loss of the second network, wherein the regression loss and the RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the region proposal predicted by the second network in the target image.

[0474] In a possible implementation manner, acquiring the target network includes:

[0475] Receive a target network sent by a model training device, wherein the model training device is used to train to obtain the target network.

[0476] It should be noted that the implementation of each operation can also refer to Fig.10 The corresponding description of the method embodiment shown.

[0477] The embodiment of the present application further provides a chip system, the chip system comprising at least one processor, a memory and an interface circuit, the memory, the transceiver and the at least one processor are interconnected through a line, the at least one memory stores a computer program; when the computer program is executed by the processor, Fig.10 The method flow shown is realized.

[0478] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a processor, Fig.10 The method flow shown is realized.

[0479] The present application also provides a computer program product. When the computer program product is run on a processor, Fig.10 The method flow shown is realized.

[0480] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program or hardware related to the computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store computer program codes.

Claims

1. A model training method, characterized in that: include: Extracting first feature information in the target image through a feature extraction layer of the first network; Extracting second feature information in the target image through a feature extraction layer of a second network, wherein both the first network and the second network are classification networks, and the depth of the first network is greater than the depth of the second network; Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature; Extracting features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature; Determine a feature loss by using the first local feature and the second local feature; Generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of the first network; Generating, by the classification layer of the second network, a second classification prediction value of the target region proposal in the region proposal set; Determine a classification loss according to the first classification prediction value and the second classification prediction value; The second network is trained according to the feature loss and the classification loss to obtain a target network.

2. The method according to claim 1, characterized in that The first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

3. The method according to claim 2, characterized in that The RPN is shared by the second network to the first network, or is shared by the first network to the second network.

4. The method according to any one of claims 1 to 3, characterized in that: The target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

5. The method according to any one of claims 1 to 3, characterized in that: The classification loss L cls Satisfies the following relationship: Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determining a regression loss and an RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image; The second network is trained according to the feature loss and the classification loss to obtain a target network, including: The second network is trained according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

7. The method according to any one of claims 1 to 3, characterized in that: After the second network is trained according to the feature loss and the classification loss to obtain the target network, the method further includes: The target network is sent to a model-using device, where the target network is used to predict content in an image.

8. A model training method, characterized in that: include: Extracting first feature information in the target image through a feature extraction layer of the first network; Extracting second feature information in the target image through a feature extraction layer of a second network; Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature; Extracting features about the target object in the second feature information through a Gaussian mask to obtain a second local feature; Determine a feature loss by using the first local feature and the second local feature; Training the second network according to the feature loss to obtain the intermediate network; The intermediate network is trained based on the third network to obtain a target network, wherein the first network, the second network and the third network are all classification networks, and the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network.

9. The method according to claim 8, characterized in that The method further comprises: Generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of a first network; Generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of a second network; Determine a classification loss according to the first classification prediction value and the second classification prediction value; The step of training the second network according to the feature loss to obtain the intermediate network includes: The second network is trained according to the feature loss and the classification loss to obtain the intermediate network.

10. An image detection method, characterized in that: include: Acquire a target network, wherein the target network is a network obtained by training the second network through the first network, and the parameters used for training the second network through the first network include feature loss and classification loss, the feature loss is determined according to a first local feature and a second local feature, the first local feature is a feature about the target object extracted from the first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from the second feature information through a Gaussian mask, the first feature information is feature information in the target image extracted through the feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through the feature extraction layer of the second network; the classification loss is determined according to a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated by the classification layer of the first network, the second classification prediction value is a classification prediction value of a target region proposal in the region proposal set generated by the classification layer of the second network, the first network and the second network are both classification networks, and the depth of the first network is greater than the depth of the second network; The target network identifies the content in the image.

11. The method according to claim 10, characterized in that The first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

12. The method according to claim 11, characterized in that The RPN is shared by the second network to the first network, or is shared by the first network to the second network.

13. The method according to any one of claims 10 to 12, characterized in that: The target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

14. The method according to any one of claims 10 to 12, characterized in that: The classification loss L cls Satisfies the following relationship: Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

15. The method according to any one of claims 10 to 12, characterized in that: The parameters used to train the second network also include the regression loss and RPN loss of the second network, wherein the regression loss and RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the region proposal predicted by the second network in the target image.

16. The method according to any one of claims 10 to 12, characterized in that: The target network is specifically a network obtained by training the second network through the first network and further training the trained network through the third network, wherein the depth of the third network is greater than the depth of the first network.

17. The method according to any one of claims 10 to 12, characterized in that: The acquiring of the target network comprises: Receive a target network sent by a model training device, wherein the model training device is used to train to obtain the target network.

18. An image detection method, characterized in that: include: Acquire a target network, wherein the target network is a network obtained by training a second network through multiple network iterations, the multiple networks are all classification networks, the multiple networks include at least a first network and a third network, the third network is used to train the intermediate network after the first network trains the second network to obtain the intermediate network, wherein the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network; the parameters used by the first network when training the second network include feature loss, wherein the feature loss is determined according to a first local feature and a second local feature, the first local feature is a feature about a target object extracted from first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from second feature information through a Gaussian mask, the first feature information is feature information in a target image extracted through a feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through a feature extraction layer of the second network; The target network identifies the content in the image.

19. The method according to claim 18, characterized in that The parameters used by the first network when training the second network include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value being a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value being a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

20. A model training device, characterized in that: include: A feature extraction unit, configured to extract first feature information from a target image through a feature extraction layer of a first network; The feature extraction unit is further used to extract second feature information in the target image through a feature extraction layer of a second network, wherein both the first network and the second network are classification networks, and the depth of the first network is greater than the depth of the second network; A first optimization unit, configured to extract features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature; A second optimization unit, configured to extract features about the target object in the second feature information by using a Gaussian mask to obtain a second local feature; A first determining unit, configured to determine a feature loss by using the first local feature and the second local feature; A first generating unit, configured to generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of the first network; A second generating unit, configured to generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of the second network; A second determining unit, configured to determine a classification loss according to the first classification prediction value and the second classification prediction value; A weight adjustment unit is used to train the second network according to the feature loss and the classification loss to obtain a target network.

21. The device according to claim 20, characterized in that The first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

22. The device according to claim 21, characterized in that The RPN is shared by the second network to the first network, or is shared by the first network to the second network.

23. The device according to any one of claims 20 to 22, characterized in that The target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

24. The device according to any one of claims 20 to 22, characterized in that The classification loss L cls Satisfies the following relationship: Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

25. The device according to any one of claims 20 to 22, characterized in that The device also includes: A third determining unit is used to determine the regression loss and the RPN loss of the second network according to the true value label of the region proposal in the target image and the predicted value predicted by the second network for the region proposal in the target image; The weight adjustment unit is specifically used to: train the second network according to the feature loss, the classification loss, the regression loss and the RPN loss to obtain a target network.

26. The device according to any one of claims 20 to 22, characterized in that Also includes: A sending unit is used to send the target network to a model using device after the weight adjustment unit trains the second network according to the feature loss to obtain the target network, wherein the target network is used to predict the content in the image.

27. A model training device, characterized in that: include: The first training unit is configured to train the second network based on the first network to obtain an intermediate network, wherein the training of the second network based on the first network to obtain the intermediate network specifically includes: Extracting first feature information in the target image through a feature extraction layer of the first network; Extracting second feature information in the target image through a feature extraction layer of a second network; Extracting features about the target object from the first feature information by using a Gaussian mask to obtain a first local feature; Extracting features about the target object in the second feature information through a Gaussian mask to obtain a second local feature; Determine a feature loss by using the first local feature and the second local feature; Training the second network according to the feature loss to obtain the intermediate network; The second training unit is used to train the intermediate network based on the third network to obtain a target network, wherein the first network, the second network and the third network are all classification networks, and the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network.

28. The device according to claim 27, characterized in that The device also includes: A first generating unit, configured to generate a first classification prediction value of a target region proposal in a region proposal set through a classification layer of a first network; A second generating unit, configured to generate a second classification prediction value of the target region proposal in the region proposal set through a classification layer of a second network; A second determining unit, configured to determine a classification loss according to the first classification prediction value and the second classification prediction value; The step of training the second network according to the feature loss to obtain the intermediate network is specifically as follows: The second network is trained according to the feature loss and the classification loss to obtain the intermediate network.

29. An image detection device, characterized in that: include: an acquisition unit, configured to acquire a target network, wherein the target network is a network obtained by training the second network through the first network, the parameters used for training the second network through the first network include feature loss and classification loss, the feature loss is determined according to a first local feature and a second local feature, the first local feature is a feature about a target object extracted from first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from second feature information through a Gaussian mask, the first feature information is feature information in a target image extracted through a feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through a feature extraction layer of the second network; the classification loss is determined according to a first classification prediction value and a second classification prediction value, the first classification prediction value is a classification prediction value of a target region proposal in a region proposal set generated through a classification layer of the first network, the second classification prediction value is a classification prediction value of a target region proposal in the region proposal set generated through a classification layer of the second network, the first network and the second network are both classification networks, and the depth of the first network is greater than the depth of the second network; The recognition unit is used to recognize the content in the image through the target network.

30. The device according to claim 29, characterized in that The first network and the second network share a region proposal network (RPN) so that both the first network and the second network have the region proposal set.

31. The device according to claim 30, characterized in that The RPN is shared by the second network to the first network, or is shared by the first network to the second network.

32. The device according to any one of claims 29 to 31, characterized in that The target region proposal is all region proposals in the region proposal set, or is a positive region proposal belonging to the target object in the region proposal set.

33. The device according to any one of claims 29 to 31, characterized in that The classification loss L cls Satisfies the following relationship: Where K is the total number of region proposals in the region proposal set, N p is the total number of positive region proposals belonging to the target object in the region proposal set, is the classification prediction value predicted by the classification layer of the second network for the mth region proposal in the region proposal set, y m is the true value label corresponding to the mth region proposal in the region proposal set, is a second classification prediction value predicted by the classification layer of the second network for the nth positive example region proposal belonging to the target object in the region proposal set, is a first classification prediction value predicted by the classification layer of the first network for the nth positive region proposal belonging to the target object in the region proposal set, Indicates based on and m The resulting cross entropy loss is Indicates based on and The binary cross entropy loss is obtained, and β is the preset weight balancing factor.

34. The device according to any one of claims 29 to 31, characterized in that: The parameters used to train the second network also include the regression loss and RPN loss of the second network, wherein the regression loss and RPN loss of the second network are determined based on the true value label of the region proposal in the target image and the predicted value of the region proposal predicted by the second network in the target image.

35. The device according to any one of claims 29 to 31, characterized in that The target network is specifically a network obtained by training the second network through the first network and further training the trained network through the third network, wherein the depth of the third network is greater than the depth of the first network.

36. The device according to any one of claims 29 to 31, characterized in that The acquisition unit is specifically used for: Receive a target network sent by a model training device, wherein the model training device is used to train to obtain the target network.

37. An image detection device, characterized in that: include: an acquisition unit, configured to acquire a target network, wherein the target network is a network obtained by training a second network through multiple network iterations, the multiple networks are all classification networks, the multiple networks include at least a first network and a third network, the third network is used to train the intermediate network after the first network trains the second network to obtain the intermediate network, wherein the depth of the third network is greater than the depth of the first network, and the depth of the first network is greater than the depth of the second network; the parameters used by the first network when training the second network include feature loss, wherein the feature loss is determined based on a first local feature and a second local feature, the first local feature is a feature about a target object extracted from first feature information through a Gaussian mask, the second local feature is a feature about the target object extracted from second feature information through a Gaussian mask, the first feature information is feature information in a target image extracted through a feature extraction layer of the first network, and the second feature information is feature information in the target image extracted through a feature extraction layer of the second network; The recognition unit is used to recognize the content in the image through the target network.

38. The device according to claim 37, characterized in that The parameters used by the first network when training the second network include classification loss, wherein the classification loss is determined based on a first classification prediction value and a second classification prediction value, the first classification prediction value being a classification prediction value of a target region proposal in a region proposal set generated by a classification layer of the first network, and the second classification prediction value being a classification prediction value of the target region proposal in the region proposal set generated by a classification layer of the second network.

39. A model training device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program to execute the method according to any one of claims 1 to 9.

40. A model using device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program to execute the method according to any one of claims 10 to 19.

41. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and when the computer program is executed on a processor, the method according to any one of claims 1 to 19 is implemented.

Images