Neural network training, target detection method and device and storage medium

Abstract

The present disclosure provides a neural network training, target detection method and device and storage medium, wherein the method includes: determining multiple feature maps of a sample image; wherein the multiple feature maps correspond to different resolutions; performing two-stage target detection on a first feature map group corresponding to a first resolution in the multiple feature maps through a first branch of the neural network to obtain a first detection result; performing single-stage target detection on a second feature map group corresponding to a second resolution in the multiple feature maps through a second branch of the neural network to obtain a second detection result; wherein the second resolution is less than or equal to the first resolution; supervising the second detection result based on the first detection result, and training at least the second branch.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning, and in particular to a neural network training, target detection method and device, and storage medium. Background Art

[0002] In recent years, with the development of deep learning and artificial intelligence technology, computer vision has been widely used in more and more fields. Among them, target detection is one of the most important tasks in computer vision and is also a key technology in applications such as object detection and face recognition.

[0003] The current target detection methods are mainly divided into single-stage target detection methods and two-stage target detection methods. Among them, the single-stage target detection method is faster, and the two-stage target detection method is more accurate. Summary of the invention

[0004] The present invention provides a neural network training, target detection method and device, and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a neural network training method is provided, the method comprising: determining multiple feature maps of a sample image; wherein the multiple feature maps correspond to different resolutions; performing two-stage target detection on a first feature map group corresponding to a first resolution in the multiple feature maps through a first branch of the neural network to obtain a first detection result; performing single-stage target detection on a second feature map group corresponding to a second resolution in the multiple feature maps through a second branch of the neural network to obtain a second detection result; wherein the second resolution is less than or equal to the first resolution; supervising the second detection result based on the first detection result, and training at least the second branch.

[0006] In some optional embodiments, the method also includes: determining an offset vector corresponding to an area where at least one target is located in the second feature map group; performing two-stage target detection on a first feature map group corresponding to a first resolution in the multiple feature maps through the first branch of the neural network to obtain a first detection result, including: performing two-stage target detection on the first feature map group according to the offset vector through the first branch to obtain a first detection result; performing single-stage target detection on a second feature map group corresponding to a second resolution in the multiple feature maps through the second branch of the neural network to obtain a second detection result, including: performing single-stage target detection on the second feature map group according to the offset vector through the second branch to obtain the second detection result.

[0007] In some optional embodiments, the first branch is used to perform two-stage target detection on the first feature map group according to the offset vector to obtain a first detection result, including: determining, through the first branch, the area where the at least one target is located in the first feature map group according to the offset vector; obtaining feature information of the area where the at least one target is located in the first feature map group; and performing classification processing and regression processing on the feature information of the area where the at least one target is located, respectively, and using the obtained first classification feature information and first regression feature information as the first detection result.

[0008] In some optional embodiments, the second branch is used to perform single-stage target detection on the second feature map group according to the offset vector to obtain the second detection result, including: determining, through the target convolution layer in the second branch, the convolution offsets corresponding to multiple regions on the second feature map group according to the offset vector; on the second feature map group, according to the convolution offsets corresponding to the multiple regions, extracting feature information of the multiple regions; performing classification processing and regression processing on the feature information of the multiple regions on the second feature map group, respectively, and using the obtained second classification feature information and second regression feature information as the second detection result.

[0009] In some optional embodiments, the target convolution layer is a deformable convolution layer.

[0010] In some optional embodiments, supervising the second detection result based on the first detection result and training at least the second branch includes: based on the first detection result, supervising the second detection result by knowledge distillation to determine a first loss function; and training at least the second branch according to the first loss function.

[0011] In some optional embodiments, based on the first detection result, the second detection result is supervised by knowledge distillation to determine a first loss function, including: determining the similarity between the second detection result and the first detection result; determining the first loss function according to the number of images of the sample image and the similarity.

[0012] In some optional embodiments, the method further includes: determining at least one second loss function based on the offset vector, at least one of the first detection result and the second detection result; determining a target loss function based on the first loss function and the at least one second loss function, and training the neural network.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided a target detection method, comprising: inputting an image to be detected into a neural network for target detection, and obtaining a detection result output by a second branch of the neural network, wherein the second branch obtains the detection result by performing a single-stage target detection on the image to be detected; wherein the neural network is trained using the method described in any one of the first aspects.

[0014] According to a third aspect of an embodiment of the present disclosure, a neural network training device is provided, comprising: a feature map determination module, used to determine multiple feature maps of a sample image; wherein the multiple feature maps correspond to different resolutions; a first detection module, used to perform two-stage target detection on a first feature map group corresponding to a first resolution in the multiple feature maps through a first branch of the neural network, to obtain a first detection result; a second detection module, used to perform single-stage target detection on a second feature map group corresponding to a second resolution in the multiple feature maps through a second branch of the neural network, to obtain a second detection result; wherein the second resolution is less than or equal to the first resolution; and a first training module, used to supervise the second detection result based on the first detection result, and to train at least the second branch.

[0015] In some optional embodiments, the device also includes: an offset vector determination module, which is used to determine an offset vector corresponding to an area where at least one target is located in the second feature map group; the first detection module includes: a first detection submodule, which is used to perform two-stage target detection on the first feature map group through the first branch according to the offset vector to obtain a first detection result; the second detection module includes: a second detection submodule, which is used to perform single-stage target detection on the second feature map group through the second branch according to the offset vector to obtain a second detection result.

[0016] In some optional embodiments, the first detection submodule is used to determine, through the first branch and according to the offset vector, the area where the at least one target is located in the first feature map group; obtain feature information of the area where the at least one target is located in the first feature map group; perform classification processing and regression processing on the feature information of the area where the at least one target is located, and use the obtained first classification feature information and first regression feature information as the first detection result.

[0017] In some optional embodiments, the second detection submodule is used to determine, through the target convolution layer in the second branch, the convolution offsets corresponding to multiple regions on the second feature map group according to the offset vector; on the second feature map group, extract feature information of the multiple regions according to the convolution offsets corresponding to the multiple regions; perform classification processing and regression processing on the feature information of the multiple regions on the second feature map group, respectively, and use the obtained second classification feature information and second regression feature information as the second detection result.

[0018] In some optional embodiments, the target convolution layer is a deformable convolution layer.

[0019] In some optional embodiments, the first training module includes: a loss function determination submodule, used to supervise the second detection result based on the first detection result by means of knowledge distillation to determine the first loss function; and a training submodule, used to train at least the second branch according to the first loss function.

[0020] In some optional embodiments, the loss function determination submodule is used to determine the similarity between the second detection result and the first detection result; and determine the first loss function according to the number of images of the sample images and the similarity.

[0021] In some optional embodiments, the device also includes: a loss function determination module, used to determine at least one second loss function based on the offset vector, at least one of the first detection result and the second detection result; a second training module, used to determine the target loss function based on the first loss function and the at least one second loss function, and train the neural network.

[0022] According to a fourth aspect of an embodiment of the present disclosure, a target detection device is provided, comprising: a target detection module, configured to input an image to be detected into a neural network for target detection, and obtain a detection result output by a second branch of the neural network, wherein the second branch obtains the detection result by performing a single-stage target detection on the image to be detected; wherein the neural network is trained using the method described in any one of the first aspects.

[0023] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the neural network training method described in any one of the first aspects or to execute the target detection method described in the second aspect.

[0024] According to a sixth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to implement the training method of the neural network described in any one of the first aspect or to implement the target detection method described in the second aspect.

[0025] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0026] In the disclosed embodiment, multiple feature maps of different resolutions corresponding to the sample image can be first determined, and then two-stage target detection is performed for the first feature map group with higher resolution through the first branch of the neural network, and single-stage target detection is performed for the second feature map group with lower resolution through the second branch of the neural network. Thus, the second detection result of the single-stage target detection is supervised according to the first detection result of the two-stage target detection, and at least the second branch of the neural network is trained, so that the trained neural network has faster reasoning efficiency and higher accuracy when performing target detection using the single-stage target detection method of the second branch.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] Figure 1 is a flow chart of a training method for a neural network according to an exemplary embodiment of the present disclosure;

[0030] Figure 2 is a flow chart of another training method for a neural network according to an exemplary embodiment of the present disclosure;

[0031] Figure 3 is a flow chart of another training method for a neural network according to an exemplary embodiment of the present disclosure;

[0032] Figure 4 is a flow chart of another training method for a neural network according to an exemplary embodiment of the present disclosure;

[0033] Figure 5 is a flow chart of another training method for a neural network according to an exemplary embodiment of the present disclosure;

[0034] Figure 6is a flow chart of another training method for a neural network according to an exemplary embodiment of the present disclosure;

[0035] Figure 7 is a flow chart of another training method for a neural network according to an exemplary embodiment of the present disclosure;

[0036] Figure 8 is a schematic diagram of a network structure of a neural network according to an exemplary embodiment of the present disclosure;

[0037] Fig.9A is a schematic structural diagram of a T-head module according to an exemplary embodiment of the present disclosure;

[0038] Fig. 9B is a structural schematic diagram of an S-head module according to an exemplary embodiment of the present disclosure;

[0039] Fig.10 is a schematic diagram of a network structure of a neural network for target detection according to an exemplary embodiment of the present disclosure;

[0040] Fig.11 is a block diagram of a training device for a neural network according to an exemplary embodiment of the present disclosure;

[0041] Fig.12 is a block diagram of a target detection device according to an exemplary embodiment of the present disclosure;

[0042] Fig.13 It is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0044] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] Before introducing the solution provided by the embodiment of the present disclosure, the single-stage target detection method and the two-stage target detection method are first introduced. Among them, the single-stage target detection method refers to taking the original image as input, and the result of target detection on the original image can be directly obtained through a complete neural network. The two-stage target detection method refers to taking the original image as input, and it is necessary to first determine at least one possible target area on the original image, and then obtain its feature information based on the spatial position of the possible target areas, and input it into the regional neural network (RNN) to calculate the target detection results of these areas.

[0047] It can be seen that when a single-stage target detection method is adopted, the structure of the corresponding neural network is relatively simple and the detection speed is also relatively fast. When a two-stage target detection method is adopted, the network structure of the corresponding neural network is relatively complex. Although the detection speed is not as fast as that of the single-stage target detection method, the detection accuracy is significantly improved.

[0048] Currently, in many practical application scenarios, due to the high requirements for device computing power and inference speed, the single-stage detection method is more widely used. However, compared with the two-stage target detection method, the single-stage target detection method has problems such as unbalanced detection categories, small number of regressions, and no clear feature position alignment operation.

[0049] At present, when improving the above problems, the improvements are all made to the structure and reasoning process of the neural network for single-stage target detection. The detection accuracy is still far behind that of two-stage target detection.

[0050] In order to solve this problem, the embodiments of the present disclosure provide a neural network training, target detection method and device, and storage medium. During the training process, the second detection result of the single-stage target detection is supervised based on the first detection result of the two-stage target detection, and the second branch of the single-stage target detection is trained. The single-stage target detection of the second branch can be used in the inference process, thereby improving the accuracy of target detection while ensuring the reasoning efficiency of target detection.

[0051] For example Figure 1 As shown, Figure 1 A neural network training method according to an exemplary embodiment includes the following steps:

[0052] In step 101, a plurality of feature maps of a sample image are determined.

[0053] In the embodiments of the present disclosure, a set of images with marked target areas and target classification information can be used as a sample image set, and at least one image in the sample image set can be used as a sample image. The target area on the sample image can be marked by a bounding box, and the target classification types include but are not limited to people, animals, vehicles, buildings, etc.

[0054] In the disclosed embodiment, for each sample image, a feature pyramid network (FPN) method may be used but is not limited to obtain multiple feature maps corresponding to the sample image, wherein the multiple feature maps correspond to different resolutions.

[0055] In step 102, a first branch of the neural network is used to perform two-stage target detection on a first feature map group corresponding to a first resolution among the multiple feature maps to obtain a first detection result.

[0056] In the embodiment of the present disclosure, a two-stage target detection can be performed through a first branch on a first feature map group with higher resolution among multiple feature maps, firstly determining a possible region where at least one target in the first feature map group may be located, and further, performing classification processing and regression processing based on feature information of the possible region where at least one target may be located, respectively, to obtain a first detection result.

[0057] In step 103, a second branch of the neural network is used to perform single-stage target detection on a second feature map group corresponding to a second resolution in the plurality of feature maps to obtain a second detection result.

[0058] In the disclosed embodiment, the second resolution is less than or equal to the first resolution, that is, a second feature map group with a lower resolution can be used to perform single-stage target detection through a second branch, and feature information of multiple regions is classified and regressed on the second feature map group to obtain a second detection result. In one example, the multiple regions include part or all of the regions of each feature map on the second feature map group.

[0059] In step 104, the second detection result is supervised based on the first detection result, and at least the second branch is trained.

[0060] In the above embodiment, multiple feature maps of different resolutions corresponding to the sample image can be determined first, and then two-stage target detection is performed for the first feature map group with higher resolution through the first branch of the neural network, and single-stage target detection is performed for the second feature map group with lower resolution through the second branch of the neural network. In this way, the second detection result of the single-stage target detection is supervised according to the first detection result of the two-stage target detection, and the training is performed through the second branch of the neural network, so that the trained neural network has faster reasoning efficiency and higher accuracy when performing target detection using the single-stage target detection method of the second branch.

[0061] In an alternative embodiment, for example Figure 2 As shown, the above method may also include:

[0062] In step 105, in the second feature map group, an offset vector corresponding to a region where at least one target is located is determined.

[0063] In the disclosed embodiment, based on the feature information of the second feature map group, the possible location of at least one target has been determined. Through at least one convolutional layer, the possible location of at least one target on the second feature map group can be determined, relative to the offset vector of the location of at least one target marked on the sample image. Among them, at least one convolutional layer can be used but not limited to a 1×1 convolutional layer.

[0064] Accordingly, step 102 may include:

[0065] Through the first branch, two-stage target detection is performed on the first feature map group according to the offset vector to obtain a first detection result.

[0066] In the above embodiment, when performing two-stage target detection, the offset vector can be combined to determine the area where at least one target is located. In this way, the area where at least one target is located is more accurately determined. Subsequently, target detection is performed in combination with feature information of the area where at least one target is located, thereby improving the accuracy of the first detection result.

[0067] Likewise, step 103 may include:

[0068] Through the first branch, single-stage target detection is performed on the second feature map group according to the offset vector to obtain the second detection result.

[0069] In the above embodiment, the convolution offsets corresponding to multiple regions can be determined in the second feature map group based on the offset vector, and then the feature information corresponding to the multiple regions can be extracted based on the convolution offsets corresponding to the multiple regions, thereby improving the accuracy of the second detection result and having high availability.

[0070] In an alternative embodiment, for example Figure 3 As shown, step 102 may include:

[0071] In step 102-1, through the first branch, according to the offset vector, the area where the at least one target is located is determined in the first feature map group.

[0072] In the embodiment of the present disclosure, the first branch can use a region prediction network (RPN) method in the first feature map group to determine the possible location of at least one target, and further, in combination with the offset vector, correct the possible location, so as to determine the location of at least one target after position correction in the first feature map group.

[0073] In step 102-2, in the first feature map group, feature information of the area where the at least one target is located is obtained.

[0074] In the embodiment of the present disclosure, after the area where at least one target is located is determined in the first feature map group, feature information of the area where these targets are located can be obtained.

[0075] In step 102-3, classification processing and regression processing are performed on the feature information of the area where the at least one target is located, and the obtained first classification feature information and first regression feature information are used as the first detection result.

[0076] In the embodiment of the present disclosure, two separate branches may be used to respectively perform classification processing and regression processing on the feature information of the area where at least one target is located.

[0077] The classification processing branch may include but is not limited to two 1024-dimensional fully connected (FC) layers connected in sequence, an FC layer for outputting classification features corresponding to each category (in the disclosed embodiment, an 81-dimensional FC layer may be used) and a normalization (softmax) layer, and the output result of the classification processing group is the first classification feature information. The first classification feature information may represent a probability value corresponding to a preset classification type for at least one target in the sample image. For example, the preset classification type includes N categories, and the first classification feature information includes the probability values ​​of at least one target on the sample image belonging to each of the N categories.

[0078] The regression processing branch may use, but is not limited to, four consecutive 3×3 convolutional layers, a 1024-dimensional FC layer, and a 4-dimensional FC layer to obtain first regression feature information. The first regression feature information represents the position regression features of at least one target on the sample image in different dimensions. The different dimensions include, but are not limited to, horizontal, vertical, stretch, offset, and other position dimensions.

[0079] In the above embodiment, the offset vector can be combined to perform two-stage target detection on the first feature map group through the first branch to obtain first classification feature information and first regression feature information, so that the second detection result can be subsequently supervised based on the first classification feature information and the first regression feature information, with high availability.

[0080] In an alternative embodiment, for example Figure 4 As shown, step 103 may include:

[0081] In step 103-1, the convolution offsets corresponding to the multiple regions on the second feature map group are determined according to the offset vector through the target convolution layer in the second branch.

[0082] In the disclosed embodiment, since the region prediction network determines the offset vectors of multiple regions, a small neural network formed by the target convolution layer can be used to calculate the convolution offsets corresponding to multiple regions on the second feature map group.

[0083] In step 103-2, on the second feature map group, feature information of the multiple regions is extracted according to the convolution offsets corresponding to the multiple regions.

[0084] In order to ensure the speed and accuracy of the neural network, the target convolution layer included in the second branch can use a deformable convolution layer, including but not limited to a 3×3 or 5×5 convolution layer. For example, if the computing power allows, a 5×5 convolution layer can be used. If the amount of calculation is to be reduced, the 5×5 convolution layer can be replaced with a 3×3 convolution layer.

[0085] In the disclosed embodiment, based on the convolution offset, multiple regions on the second feature map group can be positionally offset to extract more accurate feature information.

[0086] In step 103 - 3 , classification processing and regression processing are performed on the feature information of the multiple regions on the second feature map group, and the obtained second classification feature information and second regression feature information are used as the second detection result.

[0087] In the embodiment of the present disclosure, a classification processing branch and a regression processing branch may also be used to perform classification processing and regression processing on the feature information of multiple regions on the second feature map group respectively.

[0088] The classification processing branch at this time may include but is not limited to two sequentially connected 1×1 convolutional layers. The regression processing branch at this time may also include but is not limited to two sequentially connected 1×1 convolutional layers. Through the above two branches, the second classification feature information and the second regression feature information are obtained respectively, which are used as the second detection result.

[0089] In the above embodiment, the offset vector can be combined to perform single-stage target detection on the second feature map group through the second branch to obtain second classification feature information and second regression feature information, thereby improving the accuracy of the second detection result and further improving the accuracy of target detection by the trained neural network.

[0090] In an alternative embodiment, for example Figure 5 As shown, the above step 104 may include:

[0091] In step 104 - 1 , based on the first detection result, the second detection result is supervised by using knowledge distillation to determine a first loss function.

[0092] In the disclosed embodiments, knowledge distillation refers to obtaining a small model that is more suitable for reasoning based on a trained large model by transferring knowledge. Knowledge distillation is generally used on large models and small models with the same structure.

[0093] In the disclosed embodiment, the first detection result obtained by the two-stage target detection includes the first regression feature information and the first classification feature information of the area where at least one target is located, and the second detection result is to identify the second regression feature information and the second classification feature information of multiple areas on the second feature map group. Since knowledge distillation is generally used for two models with the same structure, and the structures of the single-stage detection and the two-stage detection are different, in order to be able to supervise the second detection result based on the first detection result, after obtaining the second detection result, the corresponding second regression feature information and second classification feature information can be extracted from the second feature map group according to the position correspondence for each first regression feature information and classification feature information. Thereby, the first loss function is determined based on the second detection result and the first detection result, and then supervised training is performed based on the first loss function.

[0094] In step 104 - 2 , at least the second branch is trained according to the first loss function.

[0095] In the embodiment of the present disclosure, the network parameters included in the second branch can be adjusted to minimize the first loss function, thereby achieving the purpose of training the second branch.

[0096] In the above embodiment, in view of the fact that the network structures corresponding to the two-stage target detection and the single-stage target detection are different, the purpose of knowledge distillation is achieved and the accuracy of the single-stage target detection is improved.

[0097] In an alternative embodiment, for example Figure 6 As shown, step 104-1 may include:

[0098] In step 201, the similarity between the second detection result and the first detection result is determined.

[0099] In the embodiment of the present disclosure, the cosine similarity between the second detection result and the first detection result can be determined, but is not limited to, including the cosine similarity between the first classification feature information and the second classification feature information, and the cosine similarity between the first regression feature information and the second regression feature information.

[0100] In step 202, the first loss function is determined according to the number of images of the sample images and the similarity.

[0101] In the embodiment of the present disclosure, the following formula 1 can be used to determine the first loss function L mimic :

[0102]

[0103] in, is the number of sample images, F i tr is the first regression feature information obtained by the two-stage target detection of the i-th sample image, F i tc is the first classification feature information obtained by the two-stage target detection of the i-th sample image, F i sr is the second regression feature information obtained by single-stage target detection of the i-th sample image, F i sc It is the second classification feature information obtained by single-stage target detection of the i-th sample image.

[0104] In the above embodiment, the first loss function can be determined according to the similarity between the second detection result and the first detection result and the number of sample images, so as to achieve the purpose of supervised training of the second detection result based on the first detection result by using knowledge distillation.

[0105] In an alternative embodiment, for example Figure 7 As shown, the above method may also include:

[0106] In step 106, at least one second loss function is determined according to at least one of the offset vector, the first detection result, and the second detection result.

[0107] In the embodiment of the present disclosure, the offset vector can be used as the second loss function L R , L R The smaller it is, the closer the location of at least one target in the feature map is to the true value of the location of at least one target marked in the sample image.

[0108] In addition, the second loss function L can be determined according to the first detection result and the region where at least one target marked in the sample image and the classification type information. T Similarly, the second loss function L can be determined based on the first detection result and the region where at least one target marked in the sample image and the classification type information. S .

[0109] In step 107, a target loss function is determined according to the first loss function and the at least one second loss function, and the neural network is trained.

[0110] In the embodiment of the present disclosure, the target loss function L may be determined using the following formula 2:

[0111] L=L mimic +L R +L T +L S Formula 2

[0112] By adjusting the network parameters of the neural network to minimize the objective function, the training of the neural network is completed.

[0113] In the above embodiment, the neural network can be trained according to the first loss function and at least one second loss function, so as to further improve the accuracy of the neural network in single-stage target detection and have high availability.

[0114] In an alternative embodiment, for example Figure 8 As shown, a network structure of a neural network is provided. The neural network includes a backbone network, an optimization module, an S-head module corresponding to the second branch for single-stage target detection, a regional feature aggregation module corresponding to the first branch, and a T-head module for two-stage target detection. Among them, the backbone network and the optimization module are shared by the S-head module and the T-head module.

[0115] In the disclosed embodiment, the backbone network may be, but is not limited to, a residual network (Resnet), a Google network (GoogLeNet), a computer vision group (Visual Geometry Group) network, etc. After the sample image is input into the backbone network, the backbone network uses the FPN method to obtain multiple feature maps corresponding to different resolutions.

[0116] Furthermore, a second feature map group with lower resolution among the multiple feature maps is used as an input value of the optimization module, wherein the optimization module may adopt but is not limited to a 1×1 convolutional layer, and the optimization module may output offset vectors corresponding to multiple regions on the second feature map group and the probability that the region is detected as a foreground (non-background).

[0117] Further, the first feature map group with higher resolution and the offset vector in the multiple feature maps are used as input values ​​of the regional feature aggregation module, and the regional feature aggregation module adopts the regional feature aggregation (ROIAlign) method to extract the feature information of the region where at least one target is located from the first feature map group according to the foreground probability of the above detection from high to low, and the extracted feature information of the region where at least one target is located is used as the input value of the T-head module to obtain the first detection result output by the T-head module. The first detection result includes the first classification feature information and the first regression feature information corresponding to the region where at least one target is located.

[0118] In addition, the second feature map group and the offset vector can be used as input values ​​of the S-head module to obtain a second detection result output by the S-head module, wherein the second detection result includes second classification feature information and second regression feature information corresponding to multiple regions marked on the second feature map group.

[0119] In the disclosed embodiment, the corresponding second classification feature information and second regression feature information can be extracted from the second detection result according to the position correspondence for each first regression feature information and classification feature information. Furthermore, the extracted second classification feature information and second regression feature information are supervised and trained based on the first classification feature information and the first regression feature information by means of knowledge distillation. The cosine similarity between the first classification feature information and the second classification feature information, and the cosine similarity between the first regression feature information and the second regression feature information can be determined respectively, and the first loss function can be obtained according to the above formula 1. Based on the first loss function, at least the S-head module of the neural network is trained.

[0120] In the embodiment of the present disclosure, the second loss function L can also be determined respectively R , L T and L S, combined with formula 2, the entire neural network is trained to obtain a trained neural network.

[0121] In the above embodiment, the second detection result of the single-stage target detection is supervised according to the first detection result of the two-stage target detection, and at least the second branch of the neural network is trained, so that the trained neural network has faster reasoning efficiency and higher accuracy when performing target detection using the single-stage target detection method.

[0122] In an optional embodiment, the structure of the T-head module is as follows: Fig.9A As shown, the size of the feature information of the area where each target is located can be 7×7×256, and the first classification feature information and the first regression feature information are obtained through the classification processing branch 1 and the regression processing branch 1 respectively.

[0123] The classification processing branch 1 may include but is not limited to two 1024-dimensional fully connected (FC) layers connected in sequence, an FC layer whose output corresponds to each category feature, and a normalization (softmax) layer. The regression processing branch may include but is not limited to four consecutive 3×3 convolutional layers, a 1024-dimensional FC layer, and a 4-dimensional FC layer.

[0124] The structure of the S-head module is as follows Fig. 9B As shown, the offset vector and the second feature map group are used as input values, and the convolution offsets corresponding to multiple regions on the second feature map group are first determined through the target convolution layer. Further, the second classification feature information and the second regression feature information are obtained through the classification processing branch 2 and the regression processing branch 2. Among them, the target convolution layer is a deformable convolution layer, which can be a 3×3 or 5×5 convolution layer but is not limited to it.

[0125] The classification processing branch 2 includes but is not limited to two sequentially connected 1×1 convolutional layers, and the regression processing branch 2 also includes but is not limited to two sequentially connected 1×1 convolutional layers.

[0126] In the above embodiment, the first detection result and the second detection result can be obtained through the T-head module and the S-head module respectively, so that the second detection result can be supervised based on the first detection result to complete the training of the neural network, so that the detection accuracy of the neural network finally trained is significantly improved when performing single-stage target detection.

[0127] In an optional embodiment, after the neural network training is completed, during the inference process, its network structure, for example Fig.10As shown in the figure, it includes a backbone network, an optimization module, and an S-head module corresponding to the second branch. That is, when performing inference, it is no longer necessary to retain the regional feature aggregation module in the first branch and the T-head module corresponding to the two-stage target detection. The target detection result can be obtained by only using the single-stage target detection S-head module of the second branch.

[0128] In the disclosed embodiment, after the image to be detected is input into the neural network, the detection result obtained by the single-stage target detection of the neural network through the second branch can be obtained, so that the network structure of the neural network is relatively simple, which ensures the inference efficiency while significantly improving the accuracy of the single-stage target detection of the neural network.

[0129] Corresponding to the aforementioned method embodiments, the present disclosure also provides device embodiments.

[0130] like Fig.11 As shown, Fig.11 It is a block diagram of a neural network training device shown in accordance with an exemplary embodiment of the present disclosure, the device comprising: a feature map determination module 310, for determining multiple feature maps of a sample image; wherein the multiple feature maps correspond to different resolutions; a first detection module 320, for performing two-stage target detection on a first feature map group corresponding to a first resolution in the multiple feature maps through a first branch of the neural network to obtain a first detection result; a second detection module 330, for performing single-stage target detection on a second feature map group corresponding to a second resolution in the multiple feature maps through a second branch of the neural network to obtain a second detection result; wherein the second resolution is less than or equal to the first resolution; and a first training module 340, for supervising the second detection result based on the first detection result and training at least the second branch.

[0131] In some optional embodiments, the device also includes: an offset vector determination module, which is used to determine an offset vector corresponding to an area where at least one target is located in the second feature map group; the first detection module includes: a first detection submodule, which is used to perform two-stage target detection on the first feature map group through the first branch according to the offset vector to obtain a first detection result; the second detection module includes: a second detection submodule, which is used to perform single-stage target detection on the second feature map group through the second branch according to the offset vector to obtain a second detection result.

[0132] In some optional embodiments, the first detection submodule is used to determine, through the first branch and according to the offset vector, the area where the at least one target is located in the first feature map group; obtain feature information of the area where the at least one target is located in the first feature map group; perform classification processing and regression processing on the feature information of the area where the at least one target is located, and use the obtained first classification feature information and first regression feature information as the first detection result.

[0133] In some optional embodiments, the second detection submodule is used to determine, through the target convolution layer in the second branch, the convolution offsets corresponding to multiple regions on the second feature map group according to the offset vector; on the second feature map group, extract feature information of the multiple regions according to the convolution offsets corresponding to the multiple regions; perform classification processing and regression processing on the feature information of the multiple regions on the second feature map group, respectively, and use the obtained second classification feature information and second regression feature information as the second detection result.

[0134] In some optional embodiments, the target convolution layer is a deformable convolution layer.

[0135] In some optional embodiments, the first training module includes: a loss function determination submodule, used to supervise the second detection result based on the first detection result by means of knowledge distillation to determine the first loss function; and a training submodule, used to train at least the second branch according to the first loss function.

[0136] In some optional embodiments, the loss function determination submodule is used to determine the similarity between the second detection result and the first detection result; and determine the first loss function according to the number of images of the sample images and the similarity.

[0137] In some optional embodiments, the device also includes: a loss function determination module, used to determine at least one second loss function based on the offset vector, at least one of the first detection result and the second detection result; a second training module, used to determine the target loss function based on the first loss function and the at least one second loss function, and train the neural network.

[0138] like Fig.12 As shown, Fig.12It is a block diagram of a target detection device according to an exemplary embodiment of the present disclosure, and the device includes: a target detection module 410, which is used to input an image to be detected into a neural network for target detection, and obtain a detection result output by a second branch of the neural network, wherein the second branch obtains the detection result by performing a single-stage target detection on the image to be detected; wherein the neural network is trained using any one of the training methods for the neural network.

[0139] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art may understand and implement it without creative work.

[0140] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is used to execute any of the above-mentioned neural network training methods or target detection methods.

[0141] In some optional embodiments, the embodiments of the present disclosure provide a computer program product, including computer-readable code. When the computer-readable code is run on a device, a processor in the device executes instructions for implementing the neural network training method or target detection method provided in any of the above embodiments.

[0142] In some optional embodiments, the embodiments of the present disclosure also provide another computer program product for storing computer-readable instructions, which, when executed, enable the computer to perform the operations of the neural network training method or target detection method provided in any of the above embodiments.

[0143] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0144] An embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the executable instructions stored in the memory to implement any of the above-mentioned neural network training methods or target detection methods.

[0145] Fig.13 The hardware structure diagram of an electronic device provided in an embodiment of the present disclosure is shown in FIG. The electronic device 510 includes a processor 511, and may also include an input device 512, an output device 513, and a memory 514. The input device 512, the output device 513, the memory 514, and the processor 511 are interconnected via a bus.

[0146] The memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable read-only memory (CD-ROM), which is used for related instructions and data.

[0147] The input device is used to input data and / or signals, and the output device is used to output data and / or signals. The output device and the input device can be independent devices or an integrated device.

[0148] The processor may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0149] The memory is used to store program codes and data of the network device.

[0150] The processor is used to call the program code and data in the memory to execute the steps in the above method embodiment. For details, please refer to the description in the method embodiment, which will not be repeated here.

[0151] Understandably, Fig.13 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, controllers, memories, etc., and all electronic devices that can implement the neural network training method or target detection method of the embodiment of the present disclosure are within the protection scope of the present disclosure.

[0152] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0153] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A neural network training method, characterized in that: include: Determine a plurality of feature maps of the sample image; wherein the plurality of feature maps correspond to different resolutions; Performing two-stage target detection on a first feature map group corresponding to a first resolution among the plurality of feature maps through a first branch of the neural network to obtain a first detection result; Performing single-stage target detection on a second feature map group corresponding to a second resolution among the plurality of feature maps through a second branch of the neural network to obtain a second detection result; wherein the second resolution is smaller than the first resolution; Supervising the second detection result based on the first detection result, and training at least the second branch; The method further comprises: In the second feature map group, determining an offset vector corresponding to a region where at least one target is located; The performing two-stage target detection on a first feature map group corresponding to a first resolution among the plurality of feature maps through the first branch of the neural network to obtain a first detection result includes: Through the first branch, according to the offset vector, performing two-stage target detection on the first feature map group to obtain a first detection result; The step of performing single-stage target detection on a second feature map group corresponding to a second resolution in the plurality of feature maps through the second branch of the neural network to obtain a second detection result includes: Through the second branch, single-stage target detection is performed on the second feature map group according to the offset vector to obtain the second detection result.

2. The method according to claim 1, characterized in that The step of performing two-stage target detection on the first feature map group according to the offset vector through the first branch to obtain a first detection result includes: Determining, by the first branch, a region where the at least one target is located in the first feature map group according to the offset vector; In the first feature map group, obtaining feature information of the area where the at least one target is located; Classification processing and regression processing are respectively performed on the feature information of the area where the at least one target is located, and the obtained first classification feature information and first regression feature information are used as the first detection result.

3. The method according to claim 1 or 2, characterized in that: The step of performing single-stage target detection on the second feature map group according to the offset vector through the second branch to obtain the second detection result includes: Determining, by the target convolution layer in the second branch, convolution offsets corresponding to a plurality of regions on the second feature map group according to the offset vector; On the second feature map group, extracting feature information of the multiple regions according to the convolution offsets corresponding to the multiple regions; Classification processing and regression processing are respectively performed on the feature information of the multiple regions on the second feature map group, and the obtained second classification feature information and second regression feature information are used as the second detection result.

4. The method according to claim 3, characterized in that: The target convolutional layer is a deformable convolutional layer.

5. The method according to claim 1, characterized in that: The supervising the second detection result based on the first detection result and training at least the second branch includes: Based on the first detection result, supervising the second detection result by using knowledge distillation to determine a first loss function; At least the second branch is trained according to the first loss function.

6. The method according to claim 5, characterized in that The step of supervising the second detection result by using knowledge distillation based on the first detection result to determine a first loss function includes: Determining a similarity between the second detection result and the first detection result; The first loss function is determined according to the number of images of the sample images and the similarity.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: Determine at least one second loss function according to at least one of the offset vector, the first detection result, and the second detection result; According to the first loss function and the at least one second loss function, a target loss function is determined to train the neural network.

8. A target detection method, characterized in that: include: Inputting the image to be detected into a neural network for target detection, and obtaining a detection result output by a second branch of the neural network, wherein the second branch obtains the detection result by performing single-stage target detection on the image to be detected; Wherein, the neural network is trained by the method described in any one of claims 1-7.

9. A neural network training device, characterized in that: include: A feature map determination module, used to determine multiple feature maps of a sample image; wherein the multiple feature maps correspond to different resolutions; A first detection module, configured to perform two-stage target detection on a first feature map group corresponding to a first resolution among the plurality of feature maps through a first branch of the neural network to obtain a first detection result; A second detection module, configured to perform single-stage target detection on a second feature map group corresponding to a second resolution among the plurality of feature maps through a second branch of the neural network to obtain a second detection result; wherein the second resolution is smaller than the first resolution; A first training module, configured to supervise the second detection result based on the first detection result, and to train at least the second branch; The device further includes: an offset vector determination module, configured to determine, in the second feature map group, an offset vector corresponding to a region where at least one target is located; The first detection module includes: a first detection submodule, configured to perform two-stage target detection on the first feature map group according to the offset vector through the first branch to obtain a first detection result; The second detection module includes: a second detection submodule, which is used to perform single-stage target detection on the second feature map group according to the offset vector through the second branch to obtain the second detection result.

10. A target detection device, characterized in that: The device comprises: A target detection module, used for inputting the image to be detected into a neural network for target detection, and obtaining a detection result output by a second branch of the neural network, wherein the second branch obtains the detection result by performing single-stage target detection on the image to be detected; Wherein, the neural network is trained by the method described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the neural network training method described in any one of claims 1 to 7 or to execute the target detection method described in claim 8.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to call the executable instructions stored in the memory to implement the neural network training method described in any one of claims 1 to 7 or the target detection method described in claim 8.

Images