Neural Networks and Their Training Methods

By designing an end-to-end detection model, using the combination of backbone network, feature network and prediction module, the problem of difficult to balance real-time and accuracy in detection tasks on mobile platforms is solved, efficient and real-time object detection is achieved, and detection accuracy is improved. and efficiency.

CN113554042BActive Publication Date: 2024-11-08FUJITSU LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010271823.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-04-08
Publication Date
2024-11-08
Estimated Expiration
2040-04-08

AI Technical Summary

Technical Problem

Existing detection models are difficult to balance between real-time and accuracy on mobile platforms, especially in detection tasks, where small models lose a lot of accuracy and rely on a lot of manual work and prior knowledge.

Method used

An end-to-end detection model is designed, including backbone network, feature network and prediction module. Multi-scale features are generated through deconvolution, merging and convolution operations, and prediction is combined with different convolutions to optimize the loss function for training. Neural network.

Benefits of technology

It realizes efficient and real-time object detection on resource-constrained platforms, improves detection accuracy and efficiency, is suitable for small platforms, and balances accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113554042B_ABST
    Figure CN113554042B_ABST
Patent Text Reader

Abstract

A neural network and a training method thereof are disclosed. The neural network is used for object detection in an image and includes a backbone network, a feature network and a prediction module, wherein the feature network includes a first module and a second module. The backbone network processes an image to output N first features of different sizes. The first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size to output N second features of different sizes. The second module of the feature network performs merging on the N first features to output N third features of different sizes. N fourth features of different sizes are generated by combining the N second features with the N third features, and convolution is performed on the N fourth features for different times. The prediction module predicts and calculates a first loss based on the N fourth features, and predicts and calculates a second loss based on the features obtained after the convolution. The neural network is trained based on a combination of the first loss and the second loss.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a neural network and a training method thereof, and more particularly to a neural network model for object detection and a corresponding training method thereof. Background Art

[0002] At present, with the popularity of smart phones, people are increasingly enjoying the convenience brought by technology, which has also stimulated the continuous development of artificial intelligence technology to a certain extent. Artificial intelligence often requires a lot of computing power to achieve certain results. However, the hardware processing power of mobile platforms is usually limited, so many mature algorithms cannot be deployed and applied on mobile phones.

[0003] To this end, researchers began to explore the implementation of small models. In recent years, many efficient architectures have been proposed, such as Pelee described in technical literature [1], ShuffleNetV2 described in technical literature [2], and MobileNetV3 described in technical literature [3]. These models can meet the real-time requirements of mobile platforms, but there is still much room for improvement in accuracy. In particular, in the case of detection tasks, the accuracy loss of small models is much greater than the accuracy loss in the case of classification tasks.

[0004] As we all know, detection tasks are the fundamental research of computer vision, which has been widely studied and applied. Most existing object detection models use networks designed for image classification as the backbone, and developers develop different feature representations for detectors.

[0005] In general, existing detection models usually have the following disadvantages:

[0006] 1) The detection model relies on a lot of manual work and prior knowledge. Although it can achieve good detection accuracy, it is not suitable for real-time tasks;

[0007] 2) Artificially designed small models or pruned models can handle real-time issues, but their backbone comes from networks designed for classification tasks, so their accuracy is often not high.

[0008] Technical Literature:

[0009] [1]Wang, RJ, Li,

[0010] [2] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenetv2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018.

[0011] [3]Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al. Searching for mobilenetv3.arXiv preprint arXiv:1905.02244, 2019 Summary of the invention

[0012] The inventors of the present disclosure have recognized that an end-to-end detection model is necessary and effective for detection tasks. Based on this, the present invention provides an end-to-end detection model suitable for resource-constrained platforms and a corresponding training method.

[0013] According to one aspect of the present invention, a method for training a neural network is provided, wherein the neural network is used for object detection in an image and comprises a backbone network, a feature network and a prediction module, the feature network comprises a first module and a second module, and the method comprises: the backbone network processes a sample image and outputs N first features of different sizes; the first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network and outputs N second features of different sizes; the second module of the feature network performs merging on the N first features output from the backbone network and outputs N third features of different sizes; N fourth features of different sizes are generated by combining each of the N second features with a corresponding one of the N third features having the same size, and convolution is performed on the N fourth features for different times respectively; the prediction module predicts based on the N fourth features and calculates a first loss; the prediction module predicts based on the features obtained after the convolution and calculates a second loss; based on the combination of the first loss and the second loss, the neural network is trained in a manner to optimize the configuration of the backbone network, the feature network and the prediction module.

[0014] According to another aspect of the present invention, a neural network for object detection in an image is provided, the neural network comprising a backbone network, a feature network and a prediction module, the feature network comprising a first module and a second module. The backbone network processes a sample image and outputs N first features of different sizes. The first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network, and outputs N second features of different sizes. The second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes. Each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are respectively convolved for different times. The prediction module predicts and calculates a first loss based on the N fourth features, and predicts and calculates a second loss based on the features obtained after the convolution. The neural network is trained based on a combination of the first loss and the second loss.

[0015] According to another aspect of the present invention, there is provided a device for training a neural network, wherein the neural network is used for object detection in an image and comprises a backbone network, a feature network and a prediction module, wherein the feature network comprises a first module and a second module. The backbone network processes a sample image and outputs N first features of different sizes. The first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network and outputs N second features of different sizes. The second module of the feature network performs merging on the N first features output from the backbone network and outputs N third features of different sizes. Each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are respectively convolved for different times. The prediction module predicts and calculates a first loss based on the N fourth features, and predicts and calculates a second loss based on the features obtained after the convolution. The device comprises one or more processors, and the processors are configured to train the neural network in a manner to optimize the configuration of the backbone network, the feature network and the prediction module based on the combination of the first loss and the second loss.

[0016] According to another aspect of the present invention, there is provided a storage medium storing a program, which, when executed, causes a computer to execute a method for training a neural network. The neural network is used for object detection in an image and includes a backbone network, a feature network and a prediction module, and the feature network includes a first module and a second module. The backbone network processes a sample image and outputs N first features of different sizes. The first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network, and outputs N second features of different sizes. The second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes. Each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are respectively subjected to different numbers of convolutions. The prediction module predicts and calculates a first loss based on the N fourth features, and predicts and calculates a second loss based on the features obtained after convolution. The method includes training the neural network in a manner that optimizes configurations of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The framework of the object detection model according to the present invention is shown.

[0018] Figure 2 The framework of the backbone network is schematically shown.

[0019] Figure 3 A convolution unit is schematically shown.

[0020] Figure 4 The framework of the feature network and target prediction module is schematically shown.

[0021] Figure 5 The merging performed by the second module of the feature network is schematically shown.

[0022] Figure 6 A flow chart of a method for training an object detection model according to the present invention is shown.

[0023] Figure 7 A block diagram showing an exemplary configuration of computer hardware for implementing the present invention is shown. DETAILED DESCRIPTION

[0024] Figure 1 The framework of the object detection model according to the present invention is schematically shown, and the object detection model is implemented by a neural network. Figure 1As shown, the object detection model includes a backbone network 110, a feature network 120, and a prediction module 130. The backbone network 110 constitutes the basic network of the detection model, the feature network 120 is used to extract feature representations, and the target prediction module 130 uses the extracted feature representations to perform object detection.

[0025] In addition, the image input to the detection model can be adjusted to have a uniform size, such as 320 × 320. The prediction module 130 can be composed of, for example, 3×3 convolutions, and it finally outputs an output image with a bounding box and a category label, the bounding box indicating the location of the detected object, and the category label indicating the category to which the object belongs.

[0026] Figure 2 The framework of the backbone network is schematically shown. Figure 2 As shown in FIG. 1 , the backbone network according to the present invention includes, for example, 17 layers, but the present invention is not limited thereto. The first layer is a backbone layer, which performs convolution with a predetermined step size, for example, a 3×3 convolution with a step size of 2. Each of the 2nd to 17th layers is composed of Figure 3 The 2-17 layers are composed of the convolutional units shown, but the configuration of the convolutional units can be different from each other.

[0027] like Figure 3 As shown, the current layer receives the output of the previous layer and performs random channel mixing on it, and then divides the channels into two equal parts to form two branches. The same processing is performed in the two branches, and the calculation can be performed in parallel, so the processing time can be saved compared to the convolution layer of a single branch. In each branch, the following operations are performed in sequence: (1) 1×1 convolution, and then the number of channels is enlarged by a certain multiple, that is, channel expansion; (2) K×K depth convolution; (3) 1×1 convolution, and then the number of channels is reduced by the same multiple, that is, channel reduction, so that the number of channels becomes the original number in operation (1). Finally, the outputs of the two branches are connected to output to the next layer.

[0028] Figure 3 The convolutional units shown may have different configurations in different layers. Specifically, at least one of the following is configurable: the channel dilation factor, the size of the kernel of the depthwise convolution, whether to perform addition of processed channels to unprocessed channels, and whether to add a squeeze-and-excitation block. The squeeze-and-excitation block is referred to as an SE block hereinafter. As an example, the optimal combination may be obtained based on prior knowledge.

[0029] For example, the channel dilation ratio can be 1, 3, or 6, and the kernel size of the depthwise convolution can be 3×3 or 5×5. Figure 3It is not shown in the figure, but in order to obtain higher accuracy, SE blocks can be added to the convolution units of a certain layer or some layers. In this case, SE blocks can be added between depthwise convolution and 1×1 convolution (channel reduction).

[0030] In addition, if Figure 3 As shown in FIG. 1 , in each branch, the convolution-produced channel and the unprocessed channel are added to generate the output of the branch, as shown by the arrows marked with "direct connection" and the symbol representing "addition" in the figure. However, this configuration can be changed. For example, when there is a change in channel or feature size between the current layer and the previous layer, such addition processing is not performed. That is, in this case, removing Figure 3 The “direct connection” arrows and “addition” symbols shown in are omitted, and only the channels that have undergone convolution are used as the output of the corresponding branch.

[0031] Table 1 below schematically shows the specific parameters of the backbone network, where “unit” represents Figure 3 The convolutional units shown, “Y” indicates adding SE blocks.

[0032] [Table 1]

[0033]

[0034]

[0035] like Figure 1 As shown, the output of the backbone network 110 is the input of the feature network 120. Since the feature pyramid network (FPN) can handle objects of different sizes in the image well, FPN is used as the basic feature structure in the present invention. To this end, the present invention generates a first feature based on the output of each layer (1st to 17th layer) of the backbone network 110, and the first feature will be input to the feature network 120 as the output of the backbone network 110.

[0036] The generation of the first feature will be described below with reference to an example. In this example, the features output by the 5th, 12th, and 17th layers of the backbone network 110 are selected, and these features are defined as {f1, f2, f3}. Features f1, f2, and f3 have different sizes, for example, 40×40, 20×20, and 10×10, respectively, where the size of feature f3 is the smallest. Then, the maximum pooling operation with a step size of 2 and 4 is applied to feature f3, respectively, to obtain two other features f4 and f5 of smaller size. The obtained five features of different sizes {f1, f2, f3, f4, f5} constitute the first feature and are input to the feature network 120. It should be noted that the present invention is not limited to this example, and it is also possible to generate the first feature in other ways. For example, features output by other layers other than the 5th, 12th, and 17th layers can be selected, or the number of features included in the first feature can be changed, etc.

[0037] Figure 4 The framework of the feature network and target prediction module is schematically shown. Figure 4 As shown, the feature network includes a first module 410 and a second module 420, and receives a first feature F1 from a backbone network. The first feature F1 includes N (schematically shown as 5 in the figure) features of different sizes.

[0038] The first module 410 performs N-1 (4) deconvolutions based on the feature f5 with the smallest size in the first feature F1 to generate a set of new features (denoted as the second feature F2). Specifically, the first deconvolution is performed on the feature f5, and then the second deconvolution is performed on the feature obtained after the first deconvolution operation. In this way, a total of N-1 deconvolutions are performed. For example, the generated second feature F2 includes the feature f5 and four features generated by four deconvolutions.

[0039] The second module 420 performs merging on a plurality of features included in the first feature F1 to generate a set of new features (denoted as a third feature F3 ). Figure 5 The method of performing the merging is schematically shown. Figure 5 As shown, a merge of the i+1th feature and the i-th feature is performed, where i=1, 2, ..., N-1, and the size of the i+1th feature is smaller than the size of the i-th feature. First, a process including bilinear interpolation and convolution is performed on the i+1th feature. Bilinear interpolation is used to upsample the i+1th feature to change its size to the same as the i-th feature. Convolution is used to achieve channel normalization of the two features. Then, the i-th feature is merged with the processed i+1th feature to obtain a new i-th feature. Then, the obtained new i-th feature is convolved with the original i-1th feature. Figure 5The merge shown is performed to obtain the new i-1th feature.

[0040] N-1 new features can be generated in the above manner. Then, the second module 420 performs maximum pooling on the N-1th new feature with the smallest size among these new features to obtain the Nth new feature. The size of the Nth new feature is smaller than the size of the N-1th new feature. So far, N new features have been generated, which constitute the output of the second module 420, that is, the third feature F3.

[0041] The first module 410 and the second module 420 can perform processing in parallel to improve efficiency. Subsequently, the second feature F2 output by the first module 410 and the third feature F3 output by the second module 420 are connected to obtain an enhanced multi-scale feature, i.e., a fourth feature F4. More specifically, each feature included in the second feature F2 is connected with a feature having the same size in the third feature F3, thereby generating a fourth feature F4 including N features.

[0042] Figure 4 The fourth feature F4 including five features {p1, p2, p3, p4, p5} is schematically shown in FIG. The prediction module 130 makes a prediction based on the fourth feature F4 and calculates the first loss. Since this is known to those skilled in the art, this process will not be described in detail herein.

[0043] On the other hand, the features p1, p2, p3, p4, and p5 included in the fourth feature F4 are respectively subjected to different numbers of convolution calculations (e.g., 3×3 convolutions), and the prediction module 130 performs prediction based on the features obtained after the convolution, and calculates the second loss. More specifically, as the size of the feature increases, the number of convolutions performed on it decreases. In this regard, Figure 4 It is schematically shown that the feature p1 with the largest size is not convolved, and the features p2-p5 with gradually decreasing sizes are convolved once to four times respectively. Figure 4 Example shown.

[0044] Since performing convolution on features of larger scales will generate a larger amount of calculations, resulting in reduced efficiency, in the present invention, fewer convolution calculations are performed on features of larger sizes, and more convolution calculations are performed on features of smaller sizes, thereby achieving a good balance between accuracy and efficiency.

[0045] The first and second losses obtained will be used for Figure 1Training of the object detection model shown. Specifically, the prediction module 130, for example, uses 3×3 convolution to calculate the regression representation and the classification representation respectively. In terms of model evaluation, for example, the regression loss method and the focal loss method can be used. The regression loss (RLOSS) represents the loss about the detection bounding box, and the focal loss (FLOSS) represents the loss about the category label. It should be noted that the focal loss is a type of classification loss. Although the focal loss is described in detail in this article, other classification losses can also be applied in the present invention.

[0046] The first loss includes regression loss RLOSS1 and focus loss FLOSS1, and the second loss includes regression loss RLOSS2 and focus loss FLOSS2. Therefore, in the training of the detection model m, the loss function represented by the following mathematical formula (1) can be used:

[0047] LOSS(m)=FLOSS1(m)+RLOSS1(m)+FLOSS2(m)+RLOSS2(m)--(1)

[0048] By minimizing the loss function LOSS(m), the configuration parameters of the detection model including the backbone network 110, the feature network 120 and the prediction module 130 can be optimized to obtain good detection accuracy.

[0049] Figure 6 FIG. 4 is a flow chart showing a method for training an object detection model according to the present invention. Figure 6 As shown, in step S610, the backbone network 110 processes the input image to generate and output a first feature F1. The first feature F1 includes N features of different sizes, such as {f1, f2, f3, f4, f5} described above.

[0050] In step S620, the first module 410 of the feature network 120 performs N-1 deconvolutions based on the smallest feature (such as feature f5) in the first feature F1 output by the backbone network 110 to generate a second feature F2. The second feature F2 includes the smallest feature and the features obtained after each deconvolution.

[0051] In step S630, the second module 420 of the feature network 120 performs a merge on the N features in the first feature F1 output by the backbone network 110 to generate a third feature F3. The merge method is as described above with reference to Figure 5 as described.

[0052] In step S640, the second feature F2 and the third feature F3 are combined according to the size level to generate a fourth feature F4. The fourth feature F4 includes N features of different sizes, such as {p1, p2, p3, p4, p5} described above.

[0053] In step S650, different convolutions are performed on each feature in the fourth feature F4. In particular, as the size of the feature increases, the number of convolutions performed on it decreases.

[0054] In step S660, the prediction module 130 performs prediction based on the fourth feature F4 that has not been convolved, and calculates the first loss. In addition, the prediction module 130 also performs prediction based on the feature obtained after convolution, and calculates the second loss.

[0055] In step S670 , the object detection model is trained based on the loss function shown in the above mathematical formula (1) to optimize the configuration of the backbone network 110 , the feature network 120 and the prediction module 130 .

[0056] The end-to-end object detection model and its training method proposed by the present invention have been described above in combination with specific embodiments. Compared with the existing models, the detection model according to the present invention has the following advantages:

[0057] -It is a model designed specifically for detection tasks on small platforms;

[0058] - The parallel processing capability of the device is fully considered in the design of the model architecture, so that data streams can be processed simultaneously;

[0059] -Flexibly configure the kernel size and number of channels in each layer of the backbone network (except the main layer). Compared with the model that uses a fixed kernel size and a fixed number of channels in each layer, higher accuracy can be achieved;

[0060] -Able to achieve a good balance between accuracy and efficiency.

[0061] The methods described in the above embodiments may be implemented by software, hardware, or a combination of software and hardware. The programs included in the software may be stored in advance in a storage medium provided inside or outside the device. As an example, during execution, these programs are written to a random access memory (RAM) and executed by a processor (e.g., CPU), thereby implementing the various methods and processes described herein.

[0062] Figure 7 An example configuration block diagram of computer hardware for executing the method of the present invention according to a program is shown, and the computer hardware is an example of an apparatus for training a detection model according to the present invention. In addition, the backbone network, feature network and prediction module in the detection model according to the present invention can also be implemented using the computer hardware.

[0063] like Figure 7As shown, in a computer 700 , a central processing unit (CPU) 701 , a read only memory (ROM) 702 , and a random access memory (RAM) 703 are connected to one another via a bus 704 .

[0064] The input / output interface 705 is further connected to the bus 704. The input / output interface 705 is connected with the following components: an input unit 706 formed with a keyboard, a mouse, a microphone, etc.; an output unit 707 formed with a display, a speaker, etc.; a storage unit 708 formed with a hard disk, a nonvolatile memory, etc.; a communication unit 709 formed with a network interface card (such as a local area network (LAN) card, a modem, etc.); and a drive 710 that drives a removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0065] In the computer having the above-described structure, the CPU 701 loads the program stored in the storage unit 708 into the RAM 703 via the input / output interface 705 and the bus 704 and executes the program so as to perform the method described above.

[0066] The program to be executed by the computer (CPU 701) may be recorded on a removable medium 711 as a package medium, which is formed by, for example, a magnetic disk (including a floppy disk), an optical disk (including a compact disk-read only memory (CD-ROM)), a digital versatile disk (DVD), etc.), a magneto-optical disk, or a semiconductor memory. In addition, the program to be executed by the computer (CPU 701) may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0067] When the removable medium 711 is installed in the drive 710, the program can be installed in the storage unit 708 via the input / output interface 705. In addition, the program can be received by the communication unit 709 via a wired or wireless transmission medium and installed in the storage unit 708. Alternatively, the program can be pre-installed in the ROM 702 or the storage unit 708.

[0068] The program executed by the computer may be a program that performs processing according to the sequence described in this specification, or may be a program that performs processing in parallel or performs processing when necessary such as when called.

[0069] The units or devices described herein are only in a logical sense and do not strictly correspond to physical devices or entities. For example, the functions of each unit described herein may be implemented by multiple physical entities, or the functions of multiple units described herein may be implemented by a single physical entity. In addition, the features, components, elements, steps, etc. described in one embodiment are not limited to the embodiment, but may also be applied to other embodiments, such as replacing specific features, components, elements, steps, etc. in other embodiments, or combining them.

[0070] The scope of the present invention is not limited to the specific embodiments described herein. It should be understood by those skilled in the art that, depending on design requirements and other factors, various modifications or changes may be made to the embodiments herein without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

[0071] Note:

[0072] 1. A method for training a neural network, wherein the neural network is used for object detection in an image and comprises a backbone network, a feature network and a prediction module, the feature network comprises a first module and a second module, the method comprising:

[0073] The backbone network processes the sample image and outputs N first features of different sizes;

[0074] The first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network, and outputs N second features of different sizes;

[0075] The second module of the feature network merges the N first features output from the backbone network and outputs N third features of different sizes;

[0076] Generating N fourth features of different sizes by combining each of the N second features with a corresponding one of the N third features having the same size, and performing convolutions of different numbers on the N fourth features respectively;

[0077] The prediction module performs prediction based on the N fourth features and calculates a first loss;

[0078] The prediction module makes predictions based on the features obtained after convolution and calculates a second loss;

[0079] Based on a combination of the first loss and the second loss, the neural network is trained in a manner that optimizes configurations of the backbone network, the feature network, and the prediction module.

[0080] 2. According to the method described in 1, the N second features output by the first module of the feature network include the first feature with the smallest size and the features obtained after each deconvolution.

[0081] 3. The method according to 1 also includes: the second module of the feature network performs processing including bilinear interpolation and convolution on the i+1th first feature, and merges the i-th first feature with the processed i+1th first feature, where i=1,2,…,N-1.

[0082] 4. The method according to item 1, further comprising:

[0083] The second module of the feature network performs a process including bilinear interpolation and convolution on the i+1th first feature, and merges the i-th first feature with the processed i+1th first feature to obtain the i-th third feature, where i=1, 2, ..., N-1; and

[0084] The second module performs maximum pooling on the N-1th third feature to obtain the Nth third feature.

[0085] 5. The method according to 1, wherein as the size of the fourth feature increases, the number of convolutions performed on it decreases.

[0086] 6. The method according to 1, wherein each of the first loss and the second loss includes a regression loss and a classification loss.

[0087] 7. The method according to 1, wherein the backbone network comprises a plurality of layers, and the channels of each layer are divided into two parts of equal number,

[0088] The method further includes performing a process including convolution, channel expansion, and channel reduction on a channel of each of the two parts, and combining the processed channels of the two parts to be input to a next layer.

[0089] 8. The method according to item 7, further comprising:

[0090] For each portion of channels at each layer, add the processed channels to the unprocessed channels to obtain the output of that portion, and

[0091] The outputs of the two parts are combined to be input to the next layer.

[0092] 9. The method of 8, wherein the processing performed on the channels of each of the two parts further comprises adding a compression-excitation (SE) block.

[0093] 10. The method according to claim 9, wherein, for each layer of the backbone network, at least one of the following is configurable: channel dilation ratio, convolution kernel size, whether to perform addition of processed channels and unprocessed channels, and whether to add SE blocks.

[0094] 11. The method according to item 7, further comprising:

[0095] Generating multiple features of different sizes by multiple layers of the backbone network;

[0096] selecting a subset of the plurality of features and determining a specific feature of the subset with the smallest size;

[0097] By processing the specific feature, one or more features having a size smaller than the specific feature are generated;

[0098] The N first features output by the backbone network are composed of the features in the subset and the one or more features whose size is smaller than the specific feature.

[0099] 12. A neural network for object detection in an image, the neural network comprising:

[0100] A backbone network, wherein the backbone network processes the sample image and outputs N first features of different sizes;

[0101] A feature network, the feature network comprising a first module and a second module, the first module performing N-1 deconvolutions based on the first feature with the smallest size output by the backbone network, and outputting N second features of different sizes, the second module performing merging on the N first features output from the backbone network, and outputting N third features of different sizes, wherein each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are respectively subjected to different numbers of convolutions; and

[0102] A prediction module, wherein the prediction module predicts and calculates a first loss based on the N fourth features, and predicts and calculates a second loss based on the features obtained after convolution,

[0103] Wherein, the neural network is trained based on a combination of the first loss and the second loss.

[0104] 13. A device for training a neural network, wherein the neural network is used for object detection in an image and includes a backbone network, a feature network and a prediction module, the feature network including a first module and a second module; the backbone network processes a sample image and outputs N first features of different sizes; the first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network, and outputs N second features of different sizes; the second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes; each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are respectively convolved different times; the prediction module predicts and calculates a first loss based on the N fourth features, and predicts and calculates a second loss based on the features obtained after the convolution,

[0105] The apparatus includes one or more processors configured to train the neural network in a manner that optimizes configurations of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss.

[0106] 14. A storage medium storing a program, which, when executed, causes a computer to perform a method for training a neural network,

[0107] The neural network is used for object detection in an image and includes a backbone network, a feature network and a prediction module, wherein the feature network includes a first module and a second module; the backbone network processes a sample image and outputs N first features of different sizes; the first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network and outputs N second features of different sizes; the second module of the feature network merges the N first features output from the backbone network and outputs N third features of different sizes; each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are respectively convolved different times; the prediction module predicts and calculates a first loss based on the N fourth features, and predicts and calculates a second loss based on the features obtained after the convolution,

[0108] The method includes: training the neural network in a manner that optimizes the configuration of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss.

Claims

1. A method for training a neural network, wherein: The neural network is used for object detection in an image and comprises a backbone network, a feature network and a prediction module, the feature network comprises a first module and a second module, the method comprises: The backbone network processes the sample image and outputs N first features of different sizes; The first module of the feature network performs N-1 deconvolutions based on the first feature with the smallest size output by the backbone network, and outputs N second features of different sizes, wherein the N second features include the first feature with the smallest size and features obtained after each deconvolution; The second module of the feature network merges the N first features output from the backbone network and outputs N third features of different sizes; Generating N fourth features of different sizes by combining each of the N second features with a corresponding one of the N third features having the same size, and performing convolutions of different numbers on the N fourth features respectively; The prediction module performs prediction based on the N fourth features and calculates a first loss; The prediction module makes predictions based on the features obtained after convolution and calculates a second loss; Based on a combination of the first loss and the second loss, the neural network is trained in a manner that optimizes configurations of the backbone network, the feature network, and the prediction module.

2. The method according to claim 1, wherein: The step of the second module of the feature network outputting the N third features also includes: a. performing a process including bilinear interpolation and convolution on the i+1th first feature, and merging the i-th first feature with the processed i+1th first feature to generate a new i-th feature, where i=N-1; b. For i=N-1, N-2, ..., 2, a process including bilinear interpolation and convolution is performed on the new i-th feature, and the i-1th first feature is merged with the processed new i-th feature to generate a new i-1th feature; c. Perform maximum pooling on the new feature with the smallest size among the N-1 new features generated by steps a and b; d. Using the N-1 new features and the features obtained by maximum pooling as the N third features.

3. The method according to claim 1, wherein: As the size of the fourth feature increases, the number of convolutions performed on it decreases.

4. The method according to claim 1, wherein: Each of the first loss and the second loss includes a regression loss and a classification loss.

5. The method according to claim 1, wherein: The backbone network includes multiple layers, and the channels of each layer are divided into two parts of equal number. The method further includes performing a process including convolution, channel expansion, and channel reduction on a channel of each of the two parts, and combining the processed channels of the two parts to be input to a next layer.

6. The method according to claim 5, further comprising: For each portion of channels at each layer, add the processed channels to the unprocessed channels to obtain the output of that portion, and The outputs of the two parts are combined to be input to the next layer.

7. The method according to claim 6, wherein: The processing performed on the channels of each of the two parts also includes adding a squeeze-excite block.

8. The method according to claim 7, wherein: For each layer of the backbone network, at least one of the following is configurable: a channel dilation factor, a convolution kernel size, whether to perform addition of processed channels and unprocessed channels, and whether to add a squeeze-excitation block.

9. The method according to claim 5, further comprising: Generating multiple features of different sizes by multiple layers of the backbone network; selecting a subset of the plurality of features and determining a specific feature of the subset with the smallest size; By processing the specific feature, one or more features having a size smaller than the specific feature are generated; The N first features output by the backbone network are composed of the features in the subset and the one or more features whose size is smaller than the specific feature.