Target detection method, device, system and storage medium

Abstract

The present invention provides a method, device, system and storage medium for target detection, the method comprising: inputting an image to be processed into a backbone neural network, obtaining multiple basic features output by multiple network layers of the backbone neural network, wherein the multiple basic features include features at multiple levels; combining the semantic information between the multiple basic features to obtain multiple fused features; and classifying and predicting bounding boxes of the multiple fused features based on a cascaded detection network to obtain target detection results. According to the method, device, system and storage medium of the present invention, the cascaded detection network is used to classify and predict bounding boxes of multiple features of fused semantic information, thereby improving the accuracy of target detection, and does not rely on the integration of post-processing, thereby improving the stability of the entire network.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to the processing of target detection. Background Art

[0002] Object detection is one of the most widely concerned issues in the field of computer vision. Although the object detection algorithm from CNN has made rapid progress in accuracy compared with traditional methods, the accuracy of object detection has been improved to a certain extent after object detection was constructed as a problem of object classification and bounding box regression. However, although the neural network model for object detection based on such object classification and bounding box regression has achieved growth in the mmAP indicator, improved the quality of candidate boxes, and improved performance indicators under high IoU (Intersection over Union) thresholds, it has declined in performance indicators under low IoU thresholds and depends on post-processing.

[0003] Therefore, the object detection in the prior art performs poorly at low IoU thresholds and relies on post-processing. Summary of the invention

[0004] The present invention is proposed in view of the above problems. The present invention provides a target detection method, device, system and computer storage medium to solve at least one of the above problems.

[0005] According to a first aspect of the present invention, there is provided a target detection method, comprising:

[0006] Inputting the image to be processed into the backbone neural network to obtain multiple basic features output by multiple network layers of the backbone neural network, wherein the multiple basic features include features at multiple levels;

[0007] Combining semantic information between the multiple basic features to obtain multiple fusion features;

[0008] The multiple fusion features are classified and bounding box predicted based on the cascade detection network to obtain the target detection result.

[0009] Exemplarily, the cascade-based detection network classifies and predicts bounding boxes on the multiple fusion features respectively to obtain a target detection result, including:

[0010] Inputting the multiple fused features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fused features;

[0011] The classification results and bounding box prediction results of the multiple fusion features are screened to obtain the target detection result.

[0012] Exemplarily, the cascaded detection network includes multiple regression modules, and the step of inputting the multiple fusion features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fusion features includes:

[0013] The current regression module receives the output result of the previous regression module and the multiple fusion features, and performs a convolution operation on the output result and the multiple fusion features;

[0014] The bounding box prediction result is obtained based on the convolution operation result of the last regression module of the multiple regression modules.

[0015] Exemplarily, the cascaded detection network includes multiple classification modules, and the step of inputting the multiple fusion features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fusion features includes:

[0016] The current classification module and the preceding classification module of the current classification module process the multiple fusion features respectively to obtain multiple current classification results corresponding to the current classification module;

[0017] Based on the multiple current classification results, a classification result corresponding to the current classification module is obtained.

[0018] Exemplarily, the filtering of the classification results and bounding box prediction results of the plurality of fusion features to obtain the target detection result includes:

[0019] The classification result and the bounding box prediction result are screened based on a non-maximum suppression algorithm to obtain the target detection result.

[0020] Exemplarily, the target detection result includes a bounding box of at least one target and a classification result corresponding to the at least one target.

[0021] According to a second aspect of the present invention, there is provided a target detection device, comprising:

[0022] A feature extraction module, used for inputting the image to be processed into the backbone neural network, and obtaining a plurality of basic features output by a plurality of network layers of the backbone neural network, wherein the plurality of basic features include features of a plurality of levels;

[0023] A feature fusion module, used to combine the semantic information between the multiple basic features to obtain multiple fusion features;

[0024] The target detection module is used to classify and predict bounding boxes of the multiple fusion features based on a cascaded detection network to obtain a target detection result.

[0025] According to a third aspect of the present invention, there is provided a target detection system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0026] According to a fourth aspect of the present invention, there is provided a computer storage medium on which a computer program is stored, wherein the computer program implements the steps of the method described in the first aspect when executed by a computer.

[0027] According to the target detection method, device, system and computer storage medium of the embodiments of the present invention, a cascade detection network is used to classify and predict bounding boxes of multiple features of fused semantic information, thereby improving the accuracy of target detection, being independent of post-processing integration, and improving the stability of the entire network. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0029] Figure 1 is a schematic block diagram of an example electronic device for implementing the target detection method and apparatus according to an embodiment of the present invention;

[0030] Figure 2 is a schematic flow chart of a target detection method according to an embodiment of the present invention;

[0031] Figure 3 is an example of a backbone neural network according to an embodiment of the present invention;

[0032] Figure 4 is a classification example of a target detection method according to an embodiment of the present invention;

[0033] Figure 5 is an example of bounding box prediction according to an object detection method according to an embodiment of the present invention;

[0034] Figure 6 is an example of a target detection method according to an embodiment of the present invention;

[0035] Figure 7 is a schematic block diagram of a target detection device according to an embodiment of the present invention;

[0036] Figure 8 is a schematic block diagram of a target detection system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0038] First, refer to Figure 1 An example electronic device 100 for implementing the target detection method and apparatus according to the embodiments of the present invention is described.

[0039] like Figure 1 As shown, the electronic device 100 includes one or more processors 101, one or more storage devices 102, an input device 103, an output device 104, and an image sensor 105, and these components are interconnected via a bus system 106 or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0040] The processor 101 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0041] The storage device 102 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used or generated by the application, may also be stored in the computer-readable storage medium.

[0042] The input device 103 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0043] The output device 104 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0044] The image sensor 105 can capture images desired by the user (eg, photos, videos, etc.), and store the captured images in the storage device 102 for use by other components.

[0045] For example, an example electronic device for implementing the target detection method and apparatus according to an embodiment of the present invention may be implemented as a smart phone, a tablet computer, an image acquisition terminal, and the like.

[0046] Next, we will refer to Figure 2 The target detection method 200 according to an embodiment of the present invention is described. Figure 2 As shown, a target detection method 200 includes:

[0047] First, in step S210, the image to be processed is input into the backbone neural network to obtain multiple basic features output by multiple network layers of the backbone neural network, wherein the multiple basic features include features at multiple levels;

[0048] In step S220, semantic information between the multiple basic features is combined to obtain multiple fusion features;

[0049] In step S230, the plurality of fused features are classified and bounding box predicted based on the cascaded detection network to obtain a target detection result.

[0050] Among them, the multiple network layers of the backbone neural network extract features of the image to be processed in turn to obtain multiple basic features of multiple levels. For example, the multiple network layers in the backbone neural network correspond to multiple levels respectively, such as the low-level network layer is used to extract shallow features, and the high-level network layer is used to improve the semantic features of the deep layer. After each network layer extracts features, it outputs the basic features corresponding to the level, and the basic features of each level can be the feature map output by the network layer of the level. By fusing the semantic information between the basic features of the corresponding levels, multiple fused features can be obtained. The detection network classifies the multiple fused features and predicts the bounding box of the target respectively, and improves the universal target detection performance (such as average precision AP, etc.) at all IoU thresholds. It does not rely on the integration of post-processing but relies on the network itself, making the network more stable and suitable for wide application in various occasions, further improving the quality and effect of target detection, and reducing costs.

[0051] Exemplarily, the target detection method according to the embodiment of the present invention may be implemented in a device, apparatus or system having a memory and a processor.

[0052] The target detection method according to the embodiment of the present invention can be deployed at an image acquisition terminal or a personal terminal, for example, it can be deployed at a camera, a personal terminal (such as a smart phone, a tablet computer, a personal computer, etc.) or a personal terminal with a camera (such as a smart phone, a tablet computer, a personal computer, etc. with a camera). For example, image data can be collected at an image acquisition terminal or a personal terminal with a camera, or image data obtained from other sources can be used as an image to be detected, and target detection can be performed at the image acquisition terminal or the personal terminal to obtain a corresponding target detection result.

[0053] Alternatively, the target detection method according to the embodiment of the present invention can also be distributedly deployed at the image acquisition end and the personal terminal, such as a smart phone, a tablet computer, a personal computer, etc. For example, image data can be collected at the image acquisition end as an image to be detected, the image to be detected is sent to the personal terminal, and target detection is performed at the personal terminal to obtain a corresponding target detection result. Furthermore, the personal terminal can return the target detection result to the image acquisition end.

[0054] Alternatively, the target detection method according to the embodiment of the present invention can also be distributedly deployed on the server side (or cloud) and the image acquisition side / personal terminal. For example, image data can be directly collected at the image acquisition side / personal terminal or obtained from other data sources as the image to be detected, and the image to be detected is passed to the server side (or cloud), and then the server side (or cloud) performs target detection based on the image to be detected to obtain the corresponding target detection result. Furthermore, the server side (or cloud) can return the target detection result to the image acquisition side / personal terminal.

[0055] According to the target detection method of the embodiment of the present invention, a cascaded detection network is used to classify and predict bounding boxes of multiple features that integrate semantic information, thereby improving the accuracy of target detection, being independent of post-processing integration, and improving the stability of the entire network.

[0056] In addition, according to the target detection method implemented in the present invention, the backbone neural network and the cascaded multiple detection networks can be combined into a target detection model.

[0057] According to step S210 of the embodiment of the present invention, the image to be processed is input into the backbone neural network to obtain multiple basic features output by multiple network layers of the backbone neural network, wherein the multiple basic features include features of multiple levels.

[0058] Exemplarily, the image to be processed may be a real-time image directly acquired by an image acquisition device, or may be an image acquired from a local data source or a remote data source.

[0059] Exemplarily, the image to be processed may also be each frame of image in real-time video data or non-real-time video data. When target detection is required for the video data, target detection may be performed on each frame of image in the video data based on the target detection method of the embodiment of the present invention to obtain the target detection result of the video data.

[0060] In some embodiments, the backbone neural network includes multiple network layers, and the multiple network layers can extract features at different levels, wherein the shallow network layer can extract shallow features of the image to be processed, and the deep network layer can extract deep semantic features of the image to be processed. The backbone neural network is used to extract features of the image to be processed to obtain the multiple basic features, which include shallow feature maps extracted by the shallow network layer of the backbone neural network and deep semantic feature maps extracted by the deep network layer of the neural network.

[0061] Exemplarily, the backbone neural network can be an existing trained model, such as a neural network model such as VGG16, Resnet, Xception14, or a self-designed neural network model. Exemplarily, the neural network model such as VGG16, Resnet, Xception, etc. can be trained using data on Imagenet to obtain the backbone neural network.

[0062] In some embodiments, the backbone neural network is, for example, a bottom-up network, including multiple network layers, through which convolution operations are performed on the processed image to obtain the multiple basic features, which include shallow features and deep semantic features, that is, multiple levels of features.

[0063] The feature semantic information of the shallower layers in the bottom-up network is relatively less, but the target position is accurate, and the feature semantic information of the deeper layers is relatively rich, but the target position is relatively rough. The high resolution of the shallower layers and the high semantic information of the deeper layers can be used at the same time. By utilizing features at multiple levels, the results of image classification and bounding box detection can be more accurate.

[0064] According to step S220 of the embodiment of the present invention, multiple fusion features are obtained by combining semantic information between the multiple basic features.

[0065] In some embodiments, semantic information between multiple basic features can be directly fused to obtain multiple fused features.

[0066] In some other embodiments, combining the semantic information between the basic features to obtain multiple fusion features may include:

[0067] Based on the multiple basic features and the corresponding output features output by multiple network layers in the top-down network, multiple fusion features are obtained.

[0068] Exemplarily, the multiple basic features can also be fused through a top-down network to obtain the multiple fused features. The top-down network can upsample the deep semantic features and the shallow features respectively to obtain the upsampled results corresponding to the deep semantic features and the upsampled results corresponding to the shallow features respectively, and fuse the upsampled results corresponding to the deep semantic features with the corresponding deep basic features generated by the backbone neural network to obtain the corresponding fused features; and fuse the upsampled results corresponding to the shallow features with the corresponding shallow basic features generated by the backbone neural network to obtain the corresponding fused features. Based on this backbone neural network and the top-down network, the semantic information between the multiple basic features can be fused to obtain the multiple fused features.

[0069] In some embodiments, the multiple fusion features may also include multiple regions of interest obtained based on feature predictions of different multiple levels of the backbone neural network and the top-down network, and the regions of interest characterize the location of the target in the image to be processed, so as to achieve a more accurate prediction effect. For example, the top-down network may be, for example, a FPN (Feature Pyramid Networks) network, but the embodiments of the present invention are not limited thereto. For targets that are smaller in size than the image to be processed, the top-down network can increase the resolution of the feature map, and even if operations are performed on feature images of larger size, more useful information about the smaller target can be obtained. Therefore, regardless of whether it is a smaller target or a larger target, the target can be detected quickly and accurately by combining the backbone neural network and the top-down network, that is, the region of interest corresponding to the image to be processed can be quickly and accurately determined, which is conducive to improving the accuracy of subsequent classification and bounding box prediction.

[0070] In some embodiments, the backbone neural network may include at least n network layers C1...Cn, and the top-down network includes at least n-1 network layers P2...Pn, where n is a natural number.

[0071] The nth network layer Cn is subjected to 1×1 convolution to reduce the dimension to a predetermined number of channels to obtain the nth network layer Pn of the top-down network;

[0072] Performing 2-fold upsampling on the nth network layer Pn to obtain the nth sampling result, and performing 1×1 convolution on the n-1th network layer Cn-1 to reduce the dimension to a predetermined number of channels, and then fusing it with the sampling result and performing 3×3 convolution to obtain the n-1th network layer Pn-1 of the top-down network;

[0073] By analogy, the third network layer P2 is upsampled by a factor of 2 to obtain the third sampling result, and the second network layer C1 is convolved by 1×1 to reduce the dimension to a predetermined number of channels and then fused with the third sampling result and then convolved by 3×3 to obtain the second network layer P2 of the top-down network.

[0074] Among them, the top-down network upsamples multiple basic features respectively, and fuses the upsampled results (that is, the output features corresponding to multiple network layers in the top-down network) and the basic features output by multiple network layers in the bottom-up network. For example, convolution operations, such as 3×3 convolutions, can also be performed on the fused basic features to eliminate the aliasing effect of upsampling. It should be understood that the network layers of the backbone neural network correspond one-to-one to the layers of the network layers of the bottom-up network, and the fused basic features have the same size as the corresponding upsampling results.

[0075] It should be noted that the nth network layer in the bottom-up network may refer to the nth convolution stage (the convolution stage may be several convolution layers that do not change the size of the feature image input at that stage), which does not mean that the bottom-up network only contains n convolution layers. The number of layers of the bottom-up network can be set according to actual needs.

[0076] In some embodiments, the predetermined number of channels includes 256.

[0077] Exemplarily, the backbone neural network also includes a pooling layer for pooling the fused features.

[0078] In one embodiment, see Figure 3 , Figure 3 An example of a backbone neural network and a top-down network according to an embodiment of the present invention is shown, and steps S210-S220 are described by taking n=6 in the backbone neural network as an example. Figure 3 As shown, the image to be processed is input into the backbone neural network to obtain multiple basic features output by multiple network layers of the backbone neural network, wherein the multiple basic features include multiple levels; and multiple fusion features are obtained by combining the semantic information between the basic features, which may specifically include:

[0079] Input the image to be processed into the backbone neural network (i.e., bottom-up network) to obtain the basic features output by the network layers C1-C6 of the backbone neural network;

[0080] The basic features output by the network layer C6 are subjected to 1×1 convolution to reduce the dimension to 256 channels and then input into the network layer P6 of the top-down network. The network layer P6 outputs the fusion feature P6';

[0081] The fused feature P6' is upsampled by 2 times to obtain the sixth sampling result, and the basic feature output by the network layer C5 is convolved by 1×1 to reduce the dimension to 256 channels, and then the semantic information is fused with the sixth sampling result, and the result is input into the network layer P5 of the top-down network, and the network layer P5 outputs the fused feature P5';

[0082] The fused feature P5' is upsampled by 2 times to obtain the fifth sampling result, and the basic feature output by the network layer C4 is convolved by 1×1 to reduce the dimension to 256 channels, and then the semantic information is fused with the fifth sampling result, and the result is input into the network layer P4 of the top-down network, and the network layer P4 outputs the fused feature P4';

[0083] The fused feature P4' is upsampled by 2 times to obtain the fourth sampling result, and the basic feature output by the network layer C3 is convolved by 1×1 to reduce the dimension to 256 channels, and then the semantic information is fused with the third sampling result, and input into the network layer P2 of the top-down network, and the network layer P3 outputs the fused feature P3';

[0084] The fused image P3' is upsampled by a factor of 2 to obtain the third sampling result, and the feature image output by the network layer C3 is convolved by 1×1 to reduce the dimension to 256 channels, and then semantic information is fused with the third sampling result, and input into the feature layer P2 of the top-down network. The network layer P2 outputs the fused feature P2'; at this time, multiple fused features P2', P3', P4', P5', and P6' are obtained.

[0085] like Figure 3 As shown, the method may further include: pooling the multiple fusion features.

[0086] According to step S230 of the embodiment of the present invention, the cascade-based detection network classifies and predicts bounding boxes of the multiple fusion features respectively to obtain a target detection result, including:

[0087] Inputting the multiple fused features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fused features;

[0088] The classification results and bounding box prediction results of the multiple fusion features are screened to obtain the target detection result.

[0089] Exemplarily, each of the detection networks comprises: a classification module and a bounding box prediction module, wherein the classification modules in the cascaded multiple detection networks are all in parallel, and the bounding box regression modules in the cascaded multiple detection networks are all in parallel.

[0090] The classification module can be used to classify the input data and output the corresponding classification score. The category of the input data can be determined based on the classification score; while the bounding box regression module can be used to fine-tune the bounding box of the input data to improve the accuracy of the bounding box.

[0091] According to the multiple detection networks cascaded in the embodiment of the present invention, it includes connecting multiple detection networks obtained by training based on different IoU thresholds in series, and the output of the previous detection network is used as the input of the next detection network, wherein the classification modules are all parallel and the bounding box regression modules are all serial. Through multiple classifications of input features and multiple bounding box regressions of the multi-level detection network, the accuracy of classification and bounding box is continuously improved, and the classification score and bounding box result of each fused feature are obtained based on the shared fusion of the classification score and bounding box regression result of each detection network, and then the classification result and bounding box of the target detected in the image to be detected are screened on this basis. Since the classification scores and bounding box regression features of different stages in the cascaded multiple detection networks, i.e., different detection networks, are shared and fused to obtain the result and bounding box position of each fused feature, better target detection results are achieved.

[0092] In some embodiments, each classification module may include two fully connected convolutional layers and a classification layer. The fused features are respectively input into the current classification module and the previous classification module, and then input into the classification layer after passing through the two fully connected convolutional layers in the current classification module and the previous classification module, to obtain the classification results of each classification module for the fused features.

[0093] In some embodiments, the classification result includes a probability or a confidence score that the fused feature belongs to a certain type of target.

[0094] Exemplarily, the bounding box regression module may include several convolutional layers. In some embodiments, the bounding box regression module may include 2 3×3 convolutional layers, or 1 3×3 convolutional layer plus 1 1×1 convolutional layer.

[0095] Exemplarily, the method 200 further includes:

[0096] The training data is input into a plurality of cascaded neural networks for training to obtain the plurality of cascaded detection networks; wherein the IoU threshold of the current-level detection network is higher than that of the previous-level detection network, the output data of the previous-level detection network is input into the current-level detection network, the output data of the previous-level detection network is divided into positive and negative samples of the current level according to the IoU threshold of the current-level detection network, and the current-level detection network is trained based on the positive and negative samples of the current level.

[0097] Exemplarily, the cascaded detection network includes multiple classification modules, and the step of inputting the multiple fusion features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fusion features includes:

[0098] The current classification module and the preceding classification module of the current classification module process the multiple fusion features respectively to obtain multiple current classification results corresponding to the current classification module;

[0099] Based on the multiple current classification results, a classification result corresponding to the current classification module is obtained.

[0100] Specifically, when the current classification module classifies each fused feature, the output result may include the classification results of the fused feature by all the previous classification modules before the current classification module, as well as the classification result of the fused feature by the current classification module; further, the sum of all the classification results of the fused feature by the current classification module and its previous classification modules may be used as the classification result corresponding to the current classification module. Among them, since each of the multiple detection networks in the cascade is trained based on different IoU thresholds, the classification modules in the multiple detection networks are used and their outputs are fused as the classification score of the fused feature, and the features of different detection networks, i.e., different stages of the cascade, are shared and fused, thereby improving the classification accuracy.

[0101] In one example, see Figure 4 , Figure 4 FIG. 2 shows a classification example of the target detection method according to an embodiment of the present invention. Figure 4 As shown, taking three detection networks as an example, the classification results of the fusion features may include:

[0102] Assuming that each detection network can detect k types of targets and background classes, the classification result S = [S1, S2, ... Sk+1], where Si corresponds to the confidence score that the fused feature belongs to the i-th type of target, i = 1, 2, 3 ... k+1, k+1 represents the background class;

[0103] The fusion features are input into the three detection networks of the cascade, and after passing through the two 1024-dimensional fully connected layers and the classification layer of each detection network, three classification results are obtained, including:

[0104] Inputting the fused features into the three detection networks of the cascade, passing through the two 1024-dimensional fully connected layers and the classification layer of the first-level detection network, to obtain the classification result of the first-level detection network;

[0105] The fusion features are passed through two 1024-dimensional fully connected layers and a classification layer of the first-level detection network to obtain a first classification result, and two 1024-dimensional fully connected layers and a classification layer of the second-level detection network to obtain a second classification result, and the sum of the first classification result and the second classification result is used as the classification result of the second-level detection network;

[0106] The fused features are passed through two 1024-dimensional fully connected layers and a classification layer of the first-level detection network to obtain a first classification result, the two 1024-dimensional fully connected layers and a classification layer of the second-level detection network to obtain a second classification result, and the two 1024-dimensional fully connected layers and a classification layer of the third-level detection network to obtain a third classification result, and the sum of the first classification result, the second classification result and the third classification result is used as the classification result of the third-level detection network, that is, the classification result S=[S1, S2, ...Sk+1] of the fused features;

[0107] The maximum value Smax among S1, S2, ...Sk+1 is selected, and the category corresponding to Smax is used as the classification result of the fusion feature (for example, if S4 is the maximum value, the region of interest feature belongs to the fourth category of targets).

[0108] Exemplarily, the cascaded detection network includes multiple regression modules, and the step of inputting the multiple fusion features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fusion features includes:

[0109] The current regression module receives the output result of the previous regression module and the multiple fusion features, and performs a convolution operation on the output result and the multiple fusion features;

[0110] The bounding box prediction result is obtained based on the convolution operation result of the last regression module of the multiple regression modules.

[0111] Among them, each level of the cascaded multiple detection networks is trained based on different IoU thresholds, and multiple fusion features are input into multiple regression modules respectively. For each fusion feature, the output of the regression module of the current level bounding box is used together with the fusion feature as the input of the next level bounding box regression module. The output features of each level of the detection network are shared and fused with the fusion features, and the prediction results of the bounding box are continuously fine-tuned to achieve more accurate bounding box positioning.

[0112] In one example, see Figure 5 , Figure 5 FIG. 2 shows an example of bounding box prediction of a target detection method according to an embodiment of the present invention. Figure 5 As shown, taking three detection networks as an example, the bounding box of the fused feature can include:

[0113] Inputting the fused features into the three detection networks of the cascade, passing through two 3×3 convolutional layers of the first-level detection network, and obtaining the bounding box result of the first-level detection network;

[0114] The fusion feature is fused with the bounding box result of the first-level detection network to obtain a region of interest corresponding to the bounding box result of the first-level detection network, and the bounding box result of the second-level detection network is obtained by passing through two 3×3 convolutional layers of the first-level detection network and one 3×3 convolutional layer and one 1×1 convolutional layer of the second-level detection network;

[0115] After fusing the fused features with the bounding box results of the second-level detection network, the region of interest corresponding to the bounding box results of the second-level detection network is obtained, and after passing through two 3×3 convolutional layers of the first-level detection network, one 3×3 convolutional layer and one 1×1 convolutional layer of the second-level detection network, and one 3×3 convolutional layer and one 1×1 convolutional layer of the third-level detection network, the bounding box result of the third-level detection network is obtained as the final bounding box of the fused features.

[0116] Exemplarily, the filtering of the classification results and bounding box prediction results of the plurality of fusion features to obtain the target detection result includes:

[0117] The classification result and the bounding box prediction result are screened based on a non-maximum suppression algorithm to obtain the target detection result.

[0118] Among them, all bounding boxes are sorted according to the classification scores of the corresponding fusion features, and the box with the highest classification score is taken as the candidate box, and all the IoUs with the candidate box that are higher than a threshold (which can be set according to actual needs) are removed; then, the bounding box with the second largest classification score is found in the remaining bounding boxes, and all other bounding boxes with the second classification score whose IoUs are higher than a threshold (which can be set according to actual needs) are removed, and so on. Finally, the IoUs between all bounding boxes are less than the threshold, and the remaining bounding boxes are the final target detection results.

[0119] In some embodiments, the target detection result includes a bounding box of at least one target and a classification result corresponding to the at least one target.

[0120] It should be understood that the number of the detection networks can be set as needed and is not limited here.

[0121] See also Figure 6 , Figure 6 An example of a target detection method according to an embodiment of the present invention is shown. As shown in the figure, taking three cascaded detection networks as an example, the target detection method 600 includes:

[0122] First, obtain the image to be processed;

[0123] Then, the image to be processed is input into the backbone neural network to obtain multiple basic features output by each network layer C1-C6, and multiple fusion features of the network layer P2-P6 are obtained by combining the semantic information of the multiple basic features; the multiple fusion features are input into the ROI pooling layer in the backbone neural network for pooling;

[0124] Next, the multiple fused features are input into the three cascaded detection networks; on the one hand, the fused features are input into the classification module Classification1 in the Stage1 detection network, the classification module Classification2 in the Stage2 detection network, and the classification module Classification3 in the Stage3 detection network, and the classification scores of the three detection networks are obtained respectively. The three classification scores are added together to obtain the classification score of the fused features, and the classification result is determined;

[0125] On the other hand, the fused feature is input into the bounding box regression module BoundingBOX1 in the Stage1 detection network, and after the output result of the Bounding BOX1 is fused with the feature of interest, the output result of the Bounding BOX2 is fused with the feature of interest, and then the output result of the Bounding BOX2 is fused with the feature of interest, and then the output result of the Bounding BOX3 is used as the bounding box of the fused feature after passing through the bounding box regression module Bounding BOX1, Bounding BOX2 and Stage3 networks;

[0126] Finally, based on the bounding boxes and classification results of all fused features, the NMS method (non-maximum suppression method) is used to screen and obtain the final target detection results.

[0127] The detection results obtained by the target detection method according to the embodiment of the present invention have been greatly improved in terms of performance such as average precision. As shown in Table 1 below, Cascade R-CNN in Table 1 is the traditional cascade R-CNN with ResNet-101 as the backbone network, and FSCascade R-CNN is the target detection network in the target detection method according to the embodiment of the present application. AP represents the overall average precision, AP 50 It represents the average precision when the IoU threshold is greater than 0.50, AP 75 It represents the average precision when the IoU threshold is greater than 0.75, AP S 、AP M 、AP L They represent the average accuracy of detecting small targets, medium targets, and large targets, respectively. By comparing the detection accuracy given when the traditional Cascade R-CNN was proposed, the detection accuracy obtained by conducting experiments based on the traditional Cascade R-CNN structure, and the detection accuracy of the target detection network FSCascade R-CNN in the target detection method according to an embodiment of the present invention, it can be proved that in any IoU threshold or for the detection of targets of different sizes, the detection accuracy of FSCascade R-CNN is greatly improved in many aspects.

[0128]

[0129] Table 1

[0130] It can be seen that the target detection method according to the embodiment of the present invention improves the accuracy of target detection by classifying and predicting bounding boxes of multiple features that integrate semantic information through a cascade detection network, does not rely on post-processing integration, and improves the stability of the entire network.

[0131] Figure 7 FIG. 7 is a schematic block diagram of a target detection device 700 according to an embodiment of the present invention. Figure 7 As shown, the target detection device 700 according to an embodiment of the present invention includes:

[0132] The feature extraction module 710 is used to input the image to be processed into the backbone neural network to obtain multiple basic features output by multiple network layers of the backbone neural network, wherein the multiple basic features include features of multiple levels;

[0133] A feature fusion module 720, used to combine the semantic information between the multiple basic features to obtain multiple fusion features;

[0134] The target detection module 730 is used to classify and predict bounding boxes of the multiple fused features based on a cascaded detection network to obtain a target detection result.

[0135] The modules can respectively execute the above combined Figure 2-Figure 6 Each step / function in the target detection method described above. The above only describes the main functions of each component of the target detection device 700, and omits the details already described above.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0137] Figure 8 FIG. 8 is a schematic block diagram of an object detection system 800 according to an embodiment of the present invention. The object detection system 800 includes an image sensor 810 , a storage device 820 , and a processor 830 .

[0138] The image sensor 810 is used to collect image data.

[0139] The storage device 820 stores program codes for implementing corresponding steps in the target detection method according to the embodiment of the present invention.

[0140] The processor 830 is used to run the program code stored in the storage device 820 to execute the corresponding steps of the target detection method according to the embodiment of the present invention, and is used to implement the feature extraction module 710, feature fusion module 720, and target detection module 730 in the target detection device according to the embodiment of the present invention.

[0141] In addition, according to an embodiment of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are run by a computer or a processor, the corresponding steps of the target detection method of the embodiment of the present invention are used to execute, and the corresponding modules in the target detection device of the embodiment of the present invention are implemented. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media, such as a computer-readable storage medium containing a computer-readable program code for randomly generating a sequence of action instructions, and another computer-readable storage medium containing a computer-readable program code for performing target detection.

[0142] In one embodiment, the computer program instructions, when executed by a computer, may implement various functional modules of the target detection apparatus according to the embodiment of the present invention, and / or may execute the target detection method according to the embodiment of the present invention.

[0143] Each module in the target detection system according to an embodiment of the present invention can be implemented by running computer program instructions stored in a memory by a processor of an electronic device for target detection according to an embodiment of the present invention, or can be implemented when computer instructions stored in a computer-readable storage medium of a computer program product according to an embodiment of the present invention are executed by a computer.

[0144] According to the target detection method, device, system and storage medium of the embodiments of the present invention, the target classification and bounding box regression of each stage of the cascade detection network are shared and integrated, so as to improve the performance of target detection under all IoU threshold conditions, and obtain better target detection results without relying on post-processing integration.

[0145] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0146] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0148] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0149] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0150] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0151] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0152] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the article analysis device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0153] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0154] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment, and the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A target detection method, It is characterized in that The method comprises: Inputting the image to be processed into the backbone neural network to obtain multiple basic features output by multiple network layers of the backbone neural network, wherein the multiple basic features include features at multiple levels; Combining semantic information between the multiple basic features to obtain multiple fusion features; Based on the cascaded detection network, the multiple fusion features are classified and bounding box predicted respectively to obtain the target detection result, which specifically includes: Inputting the multiple fused features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fused features; Screening the classification results and bounding box prediction results of the multiple fusion features to obtain the target detection result; The cascaded detection network includes multiple regression modules, and the multiple fusion features are input into the cascaded detection network to obtain the classification results and bounding box prediction results of the multiple fusion features, including: The current regression module receives the output result of the previous regression module and the multiple fusion features, and performs a convolution operation on the output result and the multiple fusion features; The bounding box prediction result is obtained based on the convolution operation result of the last regression module of the multiple regression modules.

2. The method according to claim 1, It is characterized in that The cascaded detection network includes a plurality of classification modules, and the inputting the plurality of fusion features into the cascaded detection network to obtain the classification results and bounding box prediction results of the plurality of fusion features includes: The current classification module and the preceding classification module of the current classification module process the multiple fusion features respectively to obtain multiple current classification results corresponding to the current classification module; Based on the multiple current classification results, a classification result corresponding to the current classification module is obtained.

3. The method according to claim 1, It is characterized in that Screening the classification results and bounding box prediction results of the plurality of fusion features to obtain the target detection result includes: The classification result and the bounding box prediction result are screened based on a non-maximum suppression algorithm to obtain the target detection result.

4. The method according to claim 1, It is characterized in that The target detection result includes a bounding box of at least one target and a classification result corresponding to the at least one target.

5. A target detection device, It is characterized in that The device comprises: A feature extraction module, used for inputting the image to be processed into the backbone neural network, and obtaining a plurality of basic features output by a plurality of network layers of the backbone neural network, wherein the plurality of basic features include features of a plurality of levels; A feature fusion module, used to combine the semantic information between the multiple basic features to obtain multiple fusion features; The target detection module is used to classify and predict the bounding boxes of the multiple fusion features based on the cascade detection network to obtain the target detection result; the target detection module is used to: Inputting the multiple fused features into the cascaded detection network to obtain classification results and bounding box prediction results of the multiple fused features; Screening the classification results and bounding box prediction results of the multiple fusion features to obtain the target detection result; The cascaded detection network includes multiple regression modules, and the multiple fusion features are input into the cascaded detection network to obtain the classification results and bounding box prediction results of the multiple fusion features, including: The current regression module receives the output result of the previous regression module and the multiple fusion features, and performs a convolution operation on the output result and the multiple fusion features; The bounding box prediction result is obtained based on the convolution operation result of the last regression module of the multiple regression modules.

6. A target detection system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a computer, the steps of the method according to any one of claims 1 to 4 are implemented.

Images