Neural network training and category detection method and apparatus, terminal, and storage medium

Abstract

The application provides a neural network training and category detection method and device, a terminal and a computer readable storage medium. The neural network training method comprises: obtaining a first sample image containing a first target and a second sample image containing a second target; performing feature extraction on the first sample image and the second sample image based on a neural network to obtain a first feature data set corresponding to the first sample image and a second feature data set corresponding to the second sample image; performing feature fusion on an i-th first feature map and a j-th second feature map to obtain a first fusion feature map, and performing feature fusion on the j-th first feature map and the i-th second feature map to obtain a second fusion feature map; and iteratively training the neural network based on an error between the first fusion feature map and the second fusion feature map. The application fuses multi-scale features, and distills the first fusion feature map and the second fusion feature map from each other to avoid overfitting of model training.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method, apparatus, terminal, and computer-readable storage medium for neural network training and category detection. Background Technology

[0002] Convolutional Neural Networks (CNNs), a popular method in deep learning, obtain high-order semantic features of objects in an image by performing a series of convolution and pooling operations. Then, fully connected layers (FCs) are used to classify the features to achieve image classification.

[0003] Convolutional neural networks (CNNs) have demonstrated superior feature extraction capabilities compared to traditional image classification methods, leading to significant advancements in classification accuracy and high practical value. They are frequently used in applications such as intelligent transportation to classify vehicles, license plate types, and license plate origins. However, CNNs only consider extracting and fusing features from individual images. The quality of the image itself or external attacks can affect the robustness of the extracted features, potentially rendering them inaccurate or even misrepresenting the true characteristics of the actual category, thus impacting the application of the neural network. Furthermore, because the optimization is applied to the features of each individual image, the network is prone to overfitting during training, resulting in insufficient robustness. Summary of the Invention

[0004] The main technical problem solved by this invention is to provide a method, device, terminal, and computer-readable storage medium for neural network training and category detection, thereby solving the problem of overfitting in the prior art.

[0005] To solve the above-mentioned technical problems, the first technical solution adopted by the present invention is: to provide a neural network training method, the neural network training method comprising:

[0006] Obtain a first sample image containing the first target and a second sample image containing the second target;

[0007] Based on a neural network, features are extracted from the first sample image and the second sample image respectively to obtain a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image. The first feature dataset includes m first feature maps of different scales extracted by the neural network. The second feature dataset includes m second feature maps of different scales extracted by the neural network. The nth first feature map and the nth second feature map have the same scale. n ≤ m, and n and m are both positive integers.

[0008] The first fused feature map is obtained by fusing the i-th first feature map with the j-th second feature map, and the second fused feature map is obtained by fusing the j-th first feature map with the i-th second feature map; where i ≠ j, i ≤ m, j ≤ m, and i and j are positive integers;

[0009] The neural network is iteratively trained based on the error between the first fused feature map and the second fused feature map.

[0010] The first sample image is associated with a first annotation category of a first target, and the second sample image is associated with a second annotation category of a second target;

[0011] Training methods also include:

[0012] The first and second annotation categories are weighted and summed to obtain the generated labels corresponding to the first and second fused feature maps.

[0013] The first predicted category is obtained by performing category prediction on the first fused feature map using a neural network, and the second predicted category is obtained by performing category prediction on the second fused feature map.

[0014] The neural network is trained iteratively based on the error values ​​between the first predicted category and the generated label of the first fused feature map, and the error values ​​between the second predicted category and the generated label of the second fused feature map.

[0015] The neural network includes a feature extraction layer and at least two residual modules, the at least two residual modules including a first residual module and a second residual module, and the convolutional layer, the first residual module and the second residual module are cascaded in sequence;

[0016] Based on a neural network, features are extracted from the first sample image and the second sample image respectively, resulting in a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image, including:

[0017] Convolutional layers extract features from sample images to obtain image feature maps;

[0018] The first residual module performs downsampling and feature extraction on the image feature map to obtain the first feature map.

[0019] The second feature map is obtained by downsampling and feature extraction of the first feature map through the second residual module.

[0020] The residual module includes an input layer, a first convolutional layer, a second convolutional layer, a pooling layer, and a third convolutional layer. The input layer is connected to the first convolutional layer and the pooling layer, the second convolutional layer is connected to the first convolutional layer, and the third convolutional layer is connected to the pooling layer.

[0021] The first module performs downsampling and feature extraction on the image feature map to obtain the first feature map, which includes:

[0022] The input layer transmits the image feature maps to the first convolutional layer and the pooling layer, respectively;

[0023] The first and second convolutional layers extract features from the image feature map sequentially to obtain the target feature map;

[0024] The pooling layer adjusts the number of channels in the image feature map to obtain the pooled feature map;

[0025] The third convolutional layer downsamples the pooled feature map to obtain a preprocessed feature map.

[0026] The target feature map and the preprocessed feature map are fused to obtain the first feature map.

[0027] In this case, each residual module that is not connected to a convolutional layer in the neural network in at least two residual modules is connected to a feature extraction layer;

[0028] After the step of downsampling and extracting features from the first feature map using the second residual module to obtain the second feature map, the following steps are also included:

[0029] The feature extraction layer adjusts the number of channels and size of the second feature map.

[0030] Where 0 < j < i;

[0031] The first fused feature map is obtained by fusing the i-th first feature map with the j-th second feature map, including:

[0032] The first feature map is fused with the second feature map extracted by any residual module between the residual module that extracted the first feature map and the residual module that connects the feature extraction layer to obtain the fused feature map.

[0033] To solve the above-mentioned technical problems, the second technical solution adopted by the present invention is: to provide a target classification method, comprising:

[0034] Obtain the image to be processed; the image to be processed contains the target object;

[0035] A classification network model is used to detect the category of the image to be processed, thereby obtaining the category information of the target object. The classification network model is obtained through the neural network training method described above.

[0036] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is: to provide a 5-neural network training device, the training device comprising:

[0037] The acquisition module is used to acquire a first sample image containing a first target and a second sample image containing a second target;

[0038] The feature extraction module is used to extract features from the first sample image and the second sample image based on a neural network.

[0039] Image feature extraction is performed to obtain a first feature dataset corresponding to a first sample image and a second feature dataset corresponding to a second sample image. The first feature dataset includes m first feature maps of different scales extracted by a neural network. The second feature dataset includes m second feature maps of different scales extracted by a neural network. The nth first feature map and the nth second feature map have the same scale. n ≤ m, and n and m are both positive integers.

[0040] The feature fusion module is used to perform 5-feature fusion on the i-th first feature map and the j-th second feature map to obtain a first fused feature map, and to fuse the j-th first feature map with the i-th second feature map.

[0041] The second fused feature map is obtained by fusing features from the graph; where i ≠ j, i ≤ m, j ≤ m, and i and j are positive integers;

[0042] The training module is used to iteratively train the neural network based on the error between the first fused feature map and the second fused feature map.

[0043] To solve the above-mentioned technical problems, the fourth technical solution adopted by the present invention is: to provide a

[0044] Category detection device, the category detection device includes:

[0045] The image acquisition module is used to acquire the image to be processed; the image to be processed contains the target object.

[0046] The category detection module is used to perform category detection on the image to be processed using a classification network model.

[0047] The classification network model is obtained by testing and obtaining the category information of the target object through the neural network training method described above.

[0048] To solve the above-mentioned technical problems, the fifth technical solution adopted by the present invention is: to provide a terminal, the terminal including a memory, a processor and a computer program stored in the memory and running on the processor, the processor being used to execute program data to implement the steps in the neural network training method or the category detection method described above.

[0049] To solve the above-mentioned technical problems, the sixth technical solution adopted by the present invention is: to provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the above-mentioned neural network training method or the steps in the above-mentioned category detection method.

[0050] The beneficial effects of this invention are as follows: Unlike existing technologies, this invention provides a neural network training and category detection method, apparatus, terminal, and computer-readable storage medium. The neural network training method includes: acquiring a first sample image containing a first target and a second sample image containing a second target; performing feature extraction on the first and second sample images based on a neural network to obtain a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image; the first feature dataset includes m first feature maps of different scales extracted by the neural network; the second feature dataset includes m second feature maps of different scales extracted by the neural network; wherein the nth first feature map and the nth second feature map have the same scale; n≤m, and n and m are both positive integers; fusing the i-th first feature map with the j-th second feature map to obtain a first fused feature map, and fusing the j-th first feature map with the i-th second feature map to obtain a second fused feature map; wherein i≠j, i≤m, j≤m, and i and j are positive integers; and iteratively training the neural network based on the error between the first and second fused feature maps. This application fuses multi-scale features to obtain a first fused feature map and a second fused feature map. The neural network is trained by distilling the first fused feature map and the second fused feature map together, which improves the overall feature expression ability of the model, makes the neural network more robust, and can resist the attack of external interference factors on the training samples, thus avoiding overfitting during model training. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the neural network training method provided by the present invention;

[0053] Figure 2 This is a schematic block diagram of an embodiment of the neural network provided by the present invention;

[0054] Figure 3 This is a schematic block diagram of an embodiment of the residual module provided by the present invention;

[0055] Figure 4 This is a schematic diagram illustrating the process of obtaining the first and second fused feature maps.

[0056] Figure 5 This is a flowchart illustrating the category detection method provided by the present invention;

[0057] Figure 6 This is a schematic diagram of the framework of an embodiment of the neural network training device provided by the present invention;

[0058] Figure 7 This is a schematic diagram of the framework of an embodiment of the category detection device provided by the present invention;

[0059] Figure 8 This is a schematic diagram of the framework of an embodiment of the terminal provided by the present invention;

[0060] Figure 9 A schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. Detailed Implementation

[0061] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0062] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0063] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0064] To enable those skilled in the art to better understand the technical solution of the present invention, a neural network training method provided by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] Please see Figure 1 , Figure 1 This is a flowchart illustrating the neural network training method provided by the present invention. This embodiment provides a neural network training method, which includes the following steps.

[0066] S11: Obtain a first sample image containing the first target and a second sample image containing the second target.

[0067] Obtain the CIFAR100 public dataset, which contains 50,000 training images and 10,000 test images. Divide the 50,000 training images into 100 groups, with each group having the same number of samples. Input each group of training images simultaneously into the neural network. Each training group must contain at least 128 sample images. The sample images within each training group should contain different targets.

[0068] Specifically, any two sample images from a set of training data are used as the first sample image and the second sample image. The first sample image includes a first target; the second sample image includes a second target. The first target may be the same as the second target, or the first target may be different from the second target. In this embodiment, the first target and the second target are different.

[0069] S12: Based on the neural network, feature extraction is performed on the first sample image and the second sample image respectively to obtain the first feature dataset corresponding to the first sample image and the second feature dataset corresponding to the second sample image.

[0070] Please see Figure 2 , Figure 2 This is a schematic block diagram of an embodiment of the neural network provided by the present invention.

[0071] Specifically, the neural network includes a feature extraction layer and at least two residual modules. The at least two residual modules include at least a first residual module and a second residual module, and the feature extraction layer, the first residual module, and the second residual module are cascaded sequentially. In one specific embodiment, the at least two residual modules include a first residual module, a second residual module, a third residual module, and a fourth residual module cascaded sequentially, and the feature extraction layer is cascaded with the first residual module. In one specific embodiment, the neural network is a ResNet18 network. The 7x7 convolutional layer with a stride of 2 and the 3x3 max-pooling layer with a stride of 2 in the ResNet18 network are replaced with a single 3x3 convolutional layer with a stride of 1 to facilitate application to small-sized images. The feature extraction layer can be a 3x3 convolutional layer.

[0072] The first sample image and the second sample image are input into the neural network. The neural network extracts features from the first sample image and the second sample image respectively, and obtains feature maps of different layers corresponding to the first sample image and the second sample image respectively.

[0073] Specifically, the feature extraction layer in the neural network extracts features from the sample image to obtain an image feature map; the first residual module performs downsampling and feature extraction on the image feature map to obtain a first feature map; the second residual module performs downsampling and feature extraction on the first feature map to obtain a second feature map. If other residual modules follow the second residual module, downsampling and feature extraction are then performed on the feature map output by the previous residual module.

[0074] Please see Figure 3 , Figure 3 This is a schematic block diagram of an embodiment of the residual module provided by the present invention.

[0075] All residual modules contained in at least two residual modules can have the same or different structures. In this embodiment, all residual modules have the same structure. Specifically, the residual module includes an input layer, a first convolutional layer, a second convolutional layer, a pooling layer, and a third convolutional layer. The input layer is connected to both the first and pooling layers, the second convolutional layer is connected to the first convolutional layer, and the third convolutional layer is connected to the pooling layer. The first and second convolutional layers are both 3x3 convolutional layers, and the third convolutional layer is a 1x1 convolutional layer. The parameters of the first, second, pooling, and third convolutional layers can be set according to actual needs and are not limited here.

[0076] In this embodiment, the stride of the 3x3 convolutional layer in the first residual module is 1; the stride of the first convolutional layer in the other residual modules is 2, and the stride of the second convolutional layer is 1.

[0077] In one embodiment, the residual module further includes a first normalization layer and a second normalization layer. The first normalization layer is disposed between the first convolutional layer and the second convolutional layer, and the second normalization layer is disposed at the end of the second convolutional layer furthest from the first convolutional layer.

[0078] Specifically, the input layer transmits the image feature map to the first convolutional layer and the pooling layer respectively; the first and second convolutional layers sequentially extract features from the image feature map to obtain the target feature map. The first and second normalization layers normalize the features output by the upper convolutional layers, respectively. The pooling layer adjusts the number of channels in the image feature map to obtain a pooled feature map; the third convolutional layer downsamples the pooled feature map to obtain a preprocessed feature map; the target feature map and the preprocessed feature map are then fused to obtain the first feature map.

[0079] Specifically, the target feature map and the preprocessed feature map are fused, and then processed by an activation function before the first feature map is output.

[0080] To facilitate the fusion of feature maps of different scales, each residual module in at least two residual modules that is not connected to the feature extraction layer in the neural network is connected to a 1x1 convolutional layer. The convolutional layer adjusts the number of channels and size of the second feature map so that the number of channels and size of the first feature map are consistent with the number of channels and size of the feature map to be fused. In a preferred embodiment, the number of channels and size of the feature maps output by all residual modules are adjusted to a preset size.

[0081] By sequentially extracting features from the image feature maps of the sample images using at least two residual modules, feature maps of different layers output by each residual module are obtained.

[0082] Specifically, feature extraction is performed on the first sample image to obtain a first feature dataset, which includes m first feature maps of different scales extracted by modules of different layers in the neural network. Feature extraction is performed on the second sample image to obtain a second feature dataset, which includes m second feature maps of different scales extracted by modules of different layers in the neural network. The nth first feature map and the nth second feature map have the same scale; n ≤ m, and both n and m are positive integers.

[0083] S13: Perform feature fusion between the i-th first feature map and the j-th second feature map to obtain a first fused feature map, and perform feature fusion between the j-th first feature map and the i-th second feature map to obtain a second fused feature map.

[0084] Specifically, the i-th first feature map and the j-th second feature map are fused to obtain a first fused feature map, and the j-th first feature map and the i-th second feature map are fused to obtain a second fused feature map. Where i ≠ j, i ≤ m, j ≤ m, and i and j are positive integers.

[0085] In one embodiment, 0 < j < i. The first feature map is fused with the second feature map extracted by any residual module between the residual module that extracted the first feature map and the residual module connecting the feature extraction layer to obtain a fused feature map.

[0086] The first and second feature maps are fused according to a preset ratio to obtain a corresponding fused feature map. For example, the first and second feature maps are fused in a 4:6 ratio to obtain a first fused feature map corresponding to the first sample image and a second fused feature map corresponding to the second sample image. The preset ratio can be set according to the actual situation.

[0087] When at least two residual modules include a first residual module and a second residual module, the first feature dataset corresponding to the first sample image includes two first feature maps; the second feature dataset corresponding to the second sample image includes two second feature maps.

[0088] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the process of obtaining the first and second fused feature maps.

[0089] The first feature map output by the second residual module and the second feature map output by the first residual module are fused according to a preset ratio to obtain a first fused feature map; the second feature map output by the second residual module and the first feature map output by the first residual module are fused according to the same preset ratio to obtain a second fused feature map.

[0090] When at least two residual modules include a first residual module, a second residual module, a third residual module, and a fourth residual module, the first feature dataset corresponding to the first sample image includes four first feature maps, i.e., F a ={f a1 f a2 f a3 f a4 The second feature dataset corresponding to the second sample image includes four second feature maps, namely F. b ={f b1 f b2 f b3 f b4}

[0091] The first feature map output by the fourth residual module can be fused with the second feature maps output by the first, second, and third residual modules respectively according to a preset ratio, i.e., f a4 The fusion objects include {f b1 f b2 f b3 The first feature map output by the third residual module can be fused with the second feature maps output by the first and second residual modules respectively according to a preset ratio, i.e., f a3 The fusion objects include {f b1 f b2 The first feature map output by the second residual module can be fused with the second feature map output by the first residual module according to a preset ratio, i.e., f. a2 The fusion objects include {f b1 By fusing the first feature map from the first feature dataset with the second feature map from the second feature dataset, six first fused feature maps can be obtained.

[0092] The six second fusion feature maps corresponding to the second sample image can also be obtained by obtaining the first fusion feature map.

[0093] S14: Iteratively train the neural network based on the error between the first fused feature map and the second fused feature map.

[0094] In this embodiment, a first fused feature map and a second fused feature map are obtained by cross-feature fusion based on the first feature map and the second feature map output by the two layers of modules, and the classification model is trained based on the error value between the classification result obtained by the first fused feature map and the classification result obtained by the second fused feature map.

[0095] For example, cross-feature fusion can be performed based on the first and second feature maps output by the second and fourth residual modules, respectively. This involves fusing the first feature map output by the fourth residual module with the second feature map output by the second residual module according to a preset ratio to obtain a first fused feature map, and then fusing the second feature map output by the fourth residual module with the first feature map output by the second residual module according to the same preset ratio to obtain a second fused feature map. Alternatively, cross-feature fusion can be performed based on the first and second feature maps output by two other layers.

[0096] Since both the first fused feature map and the second fused feature map are obtained by fusing the feature maps corresponding to the first sample image and the second sample image according to the same preset ratio, the only difference is that the feature fusion is performed based on feature maps of different layers.

[0097] For example, the first fusion feature map corresponding to the first sample image is a feature fusion based on the deeper first feature map with richer semantic information and the shallower second feature map corresponding to the second sample image. The second fusion feature map corresponding to the second sample image is a feature fusion based on the deeper second feature map with richer semantic information and the shallower first feature map corresponding to the first sample image.

[0098] In one embodiment, the first fused feature map and the second fused feature map serve as supervisory information for each other; that is, the neural network is trained based on the error value between the first and second fused feature maps to obtain a classification model. This training method reduces classification output and lowers the time cost of classification prediction based on the first and second fused feature maps.

[0099] In another embodiment, the neural network further includes fully connected layers. A first predicted category for the first sample image is obtained by using the fully connected layers in the neural network to predict the category based on a first fusion feature map of the first sample image; a second predicted category for the second sample image is obtained by using the fully connected layers in the neural network to predict the category based on a second fusion feature map of the second sample image. The neural network is iteratively trained based on the side error value between the first and second predicted categories, and the trained model is used as the classification model.

[0100] By fusing features at different stages to obtain classification results and distilling them together, the overall feature representation ability of the model is improved by combining feature map distillation. This makes the feature maps extracted by the classification model and the classification results more robust, able to resist external interference factors from attacking the samples to a certain extent, and avoid overfitting during model training.

[0101] Furthermore, in order to achieve faster convergence and shorten training time, the neural network is trained based on the error values ​​between the true and predicted categories corresponding to the first and second sample images, respectively.

[0102] Specifically, the first sample image is associated with a first annotation category of a first target, and the second sample image is associated with a second annotation category of a second target. For example, the first annotation category could be cat, and the second annotation category could be dog.

[0103] Based on the preset ratios corresponding to the first and second fused feature images, the first and second annotation categories are fused according to the same preset ratio to obtain the generated labels for the first and second sample images, respectively. When the preset ratios corresponding to the first and second fused feature images are the same, the generated labels for the first and second sample images are the same; when the preset ratios corresponding to the first and second fused feature images are different, the generated labels for the first and second sample images are different. The generated labels for the first sample image are fused based on the preset ratios corresponding to the first and second fused feature images, and the generated labels for the second sample image are fused based on the preset ratios corresponding to the second fused feature images.

[0104] In this embodiment, if the preset ratio corresponding to the first fused feature image is the same as the preset ratio corresponding to the second fused feature image, then the first sample image and the second sample image correspond to the same generated label. The neural network is iteratively trained based on the error value between the first predicted category and the generated label of the first fused feature image, and the error value between the second predicted category and the generated label of the second fused feature image.

[0105] The loss value in the above embodiments can be calculated based on the MSE loss function, which further improves the feature representation ability of the network.

[0106] The trained classification model classifies the targets in the image to be processed, and outputs the degree of matching between the target and each predicted category, as well as the category with the highest probability.

[0107] The neural network training method provided in this embodiment can be used to train classification networks and object detection networks.

[0108] The neural network training method provided in this embodiment includes: acquiring a first sample image containing a first target and a second sample image containing a second target; performing feature extraction on the first sample image and the second sample image respectively based on the neural network to obtain a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image; the first feature dataset includes m first feature maps of different scales extracted by the neural network; the second feature dataset includes m second feature maps of different scales extracted by the neural network; wherein the nth first feature map and the nth second feature map have the same scale; n≤m, and n and m are both positive integers; fusing the i-th first feature map with the j-th second feature map to obtain a first fused feature map, and fusing the j-th first feature map with the i-th second feature map to obtain a second fused feature map; wherein i≠j, i≤m, j≤m, and i and j are positive integers; and iteratively training the neural network based on the error between the first fused feature map and the second fused feature map. This application fuses multi-scale features to obtain a first fused feature map and a second fused feature map. The neural network is trained by distilling the first fused feature map and the second fused feature map together, which improves the overall feature expression ability of the model, makes the neural network more robust, and can resist the attack of external interference factors on the training samples, thus avoiding overfitting during model training.

[0109] Please see Figure 5 , Figure 5 This is a flowchart illustrating the category detection method provided by the present invention. This embodiment provides a target classification method, which includes the following steps.

[0110] S21: Obtain the image to be processed; the image to be processed contains the target object.

[0111] Specifically, an image acquisition device is used to acquire an image of the target object, resulting in an image to be processed. The target object can be a pedestrian, vehicle, or similar object.

[0112] S22: Use a classification network model to perform category detection on the image to be processed to obtain the category information of the target object. The classification network model is obtained through the neural network training method described above.

[0113] Specifically, the neural network trained in the above steps is used as a classification network model. This model extracts features from the image to be processed, resulting in a feature map. Based on this feature map, the category of the target object is determined, and the category information is output.

[0114] The target classification method provided in this embodiment improves the accuracy of category detection results by using a classification network model for category detection.

[0115] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework of an embodiment of the neural network training device provided by the present invention. This embodiment provides a neural network training device 60, which includes an acquisition module 61, a feature extraction module 62, a feature fusion module 63, and a training module 64.

[0116] The acquisition module 61 is used to acquire a first sample image containing a first target and a second sample image containing a second target.

[0117] The feature extraction module 62 is used to extract features from the first sample image and the second sample image based on a neural network, respectively, to obtain a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image; the first feature dataset includes m first feature maps of different scales extracted by the neural network; the second feature dataset includes m second feature maps of different scales extracted by the neural network; wherein the nth first feature map and the nth second feature map have the same scale; n≤m, and n and m are both positive integers.

[0118] The feature fusion module 63 is used to fuse the i-th first feature map with the j-th second feature map to obtain a first fused feature map, and to fuse the j-th first feature map with the i-th second feature map to obtain a second fused feature map; where i ≠ j, i ≤ m, j ≤ m, and i and j are positive integers.

[0119] Training module 64 is used to iteratively train the neural network based on the error between the first fused feature map and the second fused feature map.

[0120] The classification network training device provided in this embodiment fuses multi-scale features to obtain a first fused feature map and a second fused feature map. The neural network is trained by mutually distilling the first fused feature map and the second fused feature map, which improves the overall feature expression ability of the model, makes the neural network more robust, and can resist the attack of external interference factors on the training samples, thus avoiding overfitting during model training.

[0121] Please see Figure 7, Figure 7 This is a schematic diagram of the framework of an embodiment of the category detection device provided by the present invention. This embodiment provides a category detection device 70, which includes an image acquisition module 71 and a category detection module 72.

[0122] The image acquisition module 71 is used to acquire the image to be processed; the image to be processed contains the target object.

[0123] The category detection module 72 is used to perform category detection on the image to be processed using a classification network model.

[0124] The classification network model is obtained by testing and obtaining the category information of the target object through the neural network training method described above.

[0125] The target classification device provided in this embodiment improves the accuracy of category detection results by using a classification network model for category detection.

[0126] Please see Figure 8 , Figure 8 This is a schematic diagram of the framework of an embodiment of the terminal provided by the present invention.

[0127] Terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement any of the above-described neural network training methods or target distributions.

[0128] The steps of the method embodiment. In a specific implementation scenario, terminal 80 may include, but is not limited to: microcomputer, server, etc. In addition, terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.

[0129] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described neural network training method or target classification method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0130] Please see Figure 9 , Figure 9 This is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described neural network training method or target classification method embodiments.

[0131] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0132] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A neural network training method, characterized in that, The training method includes: Obtain a first sample image containing a first target and a second sample image containing a second target; the first sample image is associated with a first annotation category of the first target, and the second sample image is associated with a second annotation category of the second target; Based on a neural network, features are extracted from the first sample image and the second sample image respectively to obtain a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image. The first feature dataset includes m first feature maps of different scales extracted by the neural network. The second feature dataset includes m second feature maps of different scales extracted by the neural network. The nth first feature map and the nth second feature map have the same scale. n ≤ m, and n and m are both positive integers. The first feature map of the i-th generation is fused with the second feature map of the j-th generation to obtain a first fused feature map, and the second feature map of the j-th generation is fused with the second feature map of the i-th generation to obtain a second fused feature map; where i ≠ j, i ≤ m, j ≤ m, and i and j are positive integers; The neural network is iteratively trained based on the error between the first fused feature map and the second fused feature map. The training method further includes: The first annotation category and the second annotation category are weighted and summed to obtain the generated labels corresponding to the first fused feature map and the second fused feature map; The first predicted category is obtained by performing category prediction on the first fused feature map through the neural network, and the second predicted category is obtained by performing category prediction on the second fused feature map. The neural network is iteratively trained based on the error value between the first predicted category of the first fused feature map and the generated label, and the error value between the second predicted category of the second fused feature map and the generated label.

2. The training method according to claim 1, characterized in that, The neural network includes a feature extraction layer and at least two residual modules, the at least two residual modules including a first residual module and a second residual module, and the feature extraction layer, the first residual module and the second residual module are cascaded in sequence; The step of extracting features from the first sample image and the second sample image based on a neural network to obtain a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image includes: The feature extraction layer extracts features from the sample image to obtain an image feature map; The first residual module performs downsampling and feature extraction on the image feature map to obtain the first feature map. The second residual module performs downsampling and feature extraction on the first feature map to obtain the second feature map.

3. The training method according to claim 2, characterized in that, The residual module includes an input layer, a first convolutional layer, a second convolutional layer, a pooling layer, and a third convolutional layer. The input layer is connected to the first convolutional layer and the pooling layer, the second convolutional layer is connected to the first convolutional layer, and the third convolutional layer is connected to the pooling layer. The step of downsampling and extracting features from the image feature map using the first residual module to obtain the first feature map includes: The input layer transmits the image feature map to the first convolutional layer and the pooling layer, respectively. The first convolutional layer and the second convolutional layer sequentially extract features from the image feature map to obtain the target feature map; The pooling layer adjusts the number of channels in the image feature map to obtain a pooled feature map; The third convolutional layer downsamples the pooled feature map to obtain a preprocessed feature map. The target feature map and the preprocessed feature map are fused to obtain the first feature map.

4. The training method according to claim 2, characterized in that, Each of the at least two residual modules that is not connected to the feature extraction layer in the neural network is connected to a convolutional layer; After the step of downsampling and extracting features from the first feature map using the second residual module to obtain the second feature map, the method further includes: The convolutional layer adjusts the number of channels and size of the second feature map.

5. The training method according to claim 2, characterized in that, 0 < j < i; The step of fusing the i-th first feature map with the j-th second feature map to obtain the first fused feature map includes: The first feature map is fused with the second feature map extracted by any of the residual modules between the residual module that extracted the first feature map and the residual module connected to the feature extraction layer to obtain a fused feature map.

6. A category detection method, characterized in that, include: Acquire the image to be processed; the image to be processed contains the target object; The classification network model is used to perform category detection on the image to be processed to obtain the category information of the target object. The classification network model is obtained by the method of any one of claims 1-5.

7. A neural network training device, characterized in that, The training device includes: The acquisition module is used to acquire a first sample image containing a first target and a second sample image containing a second target; the first sample image is associated with a first annotation category of the first target, and the second sample image is associated with a second annotation category of the second target; The feature extraction module is used to extract features from the first sample image and the second sample image based on a neural network, respectively, to obtain a first feature dataset corresponding to the first sample image and a second feature dataset corresponding to the second sample image; the first feature dataset includes m first feature maps of different scales extracted by the neural network; the second feature dataset includes m second feature maps of different scales extracted by the neural network; wherein the nth first feature map and the nth second feature map have the same scale; n≤m, and n and m are both positive integers; The feature fusion module is used to fuse the i-th first feature map with the j-th second feature map to obtain a first fused feature map, and to fuse the j-th first feature map with the i-th second feature map to obtain a second fused feature map; where i ≠ j, i ≤ m, j ≤ m, and i and j are positive integers; The training module is used to iteratively train the neural network based on the error between the first fused feature map and the second fused feature map; it is also used to perform a weighted summation of the first labeled category and the second labeled category to obtain the generated labels corresponding to the first fused feature map and the second fused feature map; to perform category prediction on the first fused feature map to obtain a first predicted category, and to perform category prediction on the second fused feature map to obtain a second predicted category; and to iteratively train the neural network based on the error value between the first predicted category of the first fused feature map and the generated label, and the error value between the second predicted category of the second fused feature map and the generated label.

8. A category detection device, characterized in that, The category detection device includes: An image acquisition module is used to acquire an image to be processed; the image to be processed contains a target object. The category detection module is used to perform category detection on the image to be processed using a classification network model to obtain the category information of the target object. The classification network model is obtained by the method of any one of claims 1-5.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor. The processor is used to execute program data to implement the steps in the neural network training method as described in any one of claims 1 to 5 or the steps in the category detection method as described in claim 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the neural network training method as described in any one of claims 1 to 5 or the steps of the category detection method as described in claim 6.

Images