Method, device, equipment, storage medium and product for predicting small objects in video

Abstract

This article relates to the field of financial technology. This article provides a method, device, equipment, storage medium and product for predicting small targets in videos, wherein the method includes: extracting features of multiple different scales from the image of the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale feature map; obtaining enhanced information corresponding to the small target based on the image of the previous frame corresponding to the current frame, wherein the previous frame is the frame corresponding to the previous moment of the current frame; performing small-scale information enhancement processing on the small-scale feature map through the enhanced information to obtain an enhanced small-scale feature map; inputting the enhanced small-scale feature map into the head network in the neural network model to predict the position information, confidence and classification probability corresponding to the small target, and then obtaining the probability of the existence of the small target at the position indicated by the position information, and the category to which the small target belongs. This article can reduce the occurrence of missed detection and false detection of small targets.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular, to a method, device, equipment, storage medium and product for predicting small objects in a video. Background Art

[0002] The target prediction method in the prior art only predicts a single frame of image data. For a video, the prior art mostly collects the picture of the current frame in the video and obtains the target in the video by predicting the picture of the current frame.

[0003] However, for moving targets such as human body, in the current frame, the human body is far away from the camera and the camera resolution is low, resulting in the human body in the current frame being a small target. When predicting the current frame, it is very easy to cause missing small target information. Therefore, the target prediction method of the prior art is prone to missed detection and false detection of small targets.

[0004] Therefore, there is an urgent need for a prediction method for small targets in videos that can solve the problem of insufficient information about small targets in the current frame, thereby reducing the occurrence of missed detection and false detection of small targets. Summary of the invention

[0005] The purpose of the embodiments of this article is to provide a method, device, equipment, storage medium and product for predicting small objects in a video, so as to reduce the occurrence of missed detection and false detection of small objects.

[0006] To achieve the above objectives, on the one hand, an embodiment of this invention provides a method for predicting small objects in a video, including:

[0007] Extract features of multiple scales from the image of the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale feature map, wherein the multi-scale feature map includes a small-scale feature map corresponding to a small target;

[0008] Obtaining enhancement information corresponding to the small target according to a picture of a previous frame corresponding to the current frame, wherein the previous frame is a frame corresponding to a moment before the current frame;

[0009] Performing small-scale information enhancement processing on the small-scale feature map according to the enhancement information to obtain an enhanced small-scale feature map;

[0010] The enhanced small-scale feature map is input into the head network in the neural network model to predict the position information, confidence and classification probability corresponding to the small target, thereby obtaining the probability of the existence of the small target at the position indicated by the position information and the category to which the small target belongs.

[0011] Preferably, obtaining the enhancement information corresponding to the small target according to the picture of the previous frame corresponding to the current frame further includes:

[0012] Obtaining a multi-scale front-enhancement mask feature map according to a picture of a previous frame corresponding to the current frame;

[0013] The multi-scale front-enhancement mask feature map is subjected to spatiotemporal fusion processing to obtain enhancement information corresponding to the small target.

[0014] Preferably, obtaining a multi-scale front enhancement mask feature map according to a picture of a previous frame corresponding to the current frame further comprises:

[0015] According to the picture of the previous frame corresponding to the current frame, a multi-scale front feature map is obtained, wherein the multi-scale front feature map includes a small-scale front feature map corresponding to the small target in the previous frame;

[0016] Performing small-scale information enhancement processing on the small-scale front feature map to obtain a processed multi-scale front feature map;

[0017] A mask enhancement operation is performed on the processed multi-scale front feature map to obtain a multi-scale front enhanced mask feature map.

[0018] Preferably, obtaining a multi-scale front feature map according to a picture of a previous frame corresponding to the current frame further comprises:

[0019] Multi-scale feature extraction is performed on the image of the previous frame corresponding to the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale previous feature map.

[0020] Preferably, performing small-scale information enhancement processing on the small-scale front feature map to obtain a processed multi-scale front feature map further comprises:

[0021] Obtaining previous enhancement information corresponding to the small target in the previous frame according to a picture of a historical frame corresponding to the previous frame, wherein the historical frame is a frame corresponding to a moment before the previous frame;

[0022] The small-scale front feature map is subjected to small-scale information enhancement processing by using the front enhancement information to obtain a processed multi-scale front feature map.

[0023] Preferably, performing an enhancement mask enhancement operation on the processed multi-scale front feature map to obtain a multi-scale front enhanced mask feature map further comprises:

[0024] Performing mask operations on the processed multi-scale front feature maps respectively to obtain multi-scale front mask feature maps;

[0025] Information enhancement is performed on the multi-scale front mask feature maps respectively to obtain multi-scale front enhanced mask feature maps.

[0026] Preferably, performing mask operations on the processed multi-scale front feature maps to obtain multi-scale front mask feature maps further comprises:

[0027] Inputting the processed multi-scale front feature maps into the head network in the neural network model respectively, and predicting the front position information, front confidence and front classification probability corresponding to the multi-scale targets respectively;

[0028] Determining front target frames corresponding to the multi-scale targets respectively according to the front position information corresponding to the multi-scale targets respectively;

[0029] The background pixels in the front target frames corresponding to the multi-scale targets are all set to 0 to obtain a multi-scale front mask feature map.

[0030] Preferably, the performing information enhancement on the multi-scale front mask feature maps respectively to obtain the multi-scale front enhanced mask feature maps further comprises:

[0031] The multi-scale front enhancement mask feature map is calculated by the following formula:

[0032]

[0033]

[0034] Among them, i is any scale in the multi-scale, is the previous feature map of any scale after processing, is the front mask feature map of any scale, is the prior confidence corresponding to the target of any scale, is the maximum value of the previous classification probability corresponding to the target of any scale, is the front-enhanced mask feature map of any scale.

[0035] Preferably, performing spatiotemporal fusion processing on the multi-scale front-enhancement mask feature map to obtain the enhancement information corresponding to the small target further includes:

[0036] Performing spatial domain fusion on the multi-scale front-enhancement mask feature map to obtain spatial domain features;

[0037] The spatial domain features are fused with RNN temporal domain features to obtain enhanced information corresponding to the small target.

[0038] Preferably, the performing spatial domain fusion on the multi-scale front-enhancement mask feature map to obtain the spatial domain feature further comprises:

[0039] Using an average pooling operator to unify the multi-scale dimensions of the multi-scale front-enhanced mask feature map to obtain a multi-scale feature;

[0040] The multi-scale features are spatially fused using a self-attention mechanism to obtain the spatial domain features.

[0041] Preferably, performing small-scale information enhancement processing on the small-scale feature map by using the enhancement information to obtain the enhanced small-scale feature map further comprises:

[0042] The enhanced small-scale feature map is calculated by the following formula:

[0043]

[0044] in, is the enhanced small-scale feature map, is a small-scale feature map, h (t) To enhance information.

[0045] On the other hand, an embodiment of the present invention provides a device for predicting small objects in a video, the device comprising:

[0046] A feature extraction module is used to extract features of multiple scales on the image of the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale feature map, wherein the multi-scale feature map includes a small-scale feature map corresponding to a small target;

[0047] An enhancement information determination module, used to obtain enhancement information corresponding to the small target according to a picture of a previous frame corresponding to the current frame, wherein the previous frame is a frame corresponding to a moment before the current frame;

[0048] An enhancement module, used to perform small-scale information enhancement processing on the small-scale feature map according to the enhancement information to obtain an enhanced small-scale feature map;

[0049] A prediction module is used to input the enhanced small-scale feature map into the head network in the neural network model, predict the position information, confidence and classification probability corresponding to the small target, and then obtain the probability of the existence of the small target at the position indicated by the position information, and the category to which the small target belongs.

[0050] On the other hand, the embodiments of the present invention further provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of any one of the above methods are executed.

[0051] On the other hand, the embodiments of the present invention further provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the computer program executes instructions of any one of the above-described methods.

[0052] On the other hand, the embodiments of this document further provide a computer program product, which, when executed by a processor of a computer device, executes instructions of any one of the above methods.

[0053] It can be seen from the technical solution provided in the above embodiments of this article that, after the small-scale feature map corresponding to the small target is subjected to small-scale information enhancement processing, the amount of information in the small-scale feature map corresponding to the small target becomes larger and the features are clear and complete, so that the small target can be determined more accurately and clearly, reducing the possibility of missed detection or false detection of small targets.

[0054] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of a process flow of a method for predicting small objects in a video provided by an embodiment of this invention is shown;

[0057] Figure 2 A schematic diagram of a process for obtaining enhanced information corresponding to a small target provided by an embodiment of this invention is shown;

[0058] Figure 3 A schematic diagram of a process for obtaining a multi-scale front feature map provided in an embodiment of this invention is shown;

[0059] Figure 4 A schematic diagram of a process for obtaining a processed multi-scale front feature map provided in an embodiment of this invention is shown;

[0060] Figure 5 A schematic diagram of a process for obtaining a multi-scale front-enhancement mask feature map provided by an embodiment of this invention is shown;

[0061] Figure 6 A schematic diagram of a process for obtaining a multi-scale front mask feature map provided in an embodiment of this invention is shown;

[0062] Figure 7 A schematic diagram of a process for obtaining enhanced information corresponding to a small target provided by an embodiment of this invention is shown;

[0063] Figure 8 A schematic diagram of a process for obtaining spatial domain features provided in an embodiment of this invention is shown;

[0064] Fig. 9 A schematic diagram of the module structure of a device for predicting small objects in a video provided by an embodiment of this invention is shown;

[0065] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of this invention is shown.

[0066] Description of the accompanying symbols:

[0067] 100. Feature extraction module;

[0068] 200. Enhanced information determination module;

[0069] 300, Enhancement module;

[0070] 400. prediction module;

[0071] 1002. Computer equipment;

[0072] 1004. Processor;

[0073] 1006. Memory;

[0074] 1008. Driving mechanism;

[0075] 1010, input / output module;

[0076] 1012. Input device;

[0077] 1014. Output device;

[0078] 1016. Presentation equipment;

[0079] 1018. Graphical user interface;

[0080] 1020. Network interface;

[0081] 1022. Communication link;

[0082] 1024. Communication bus. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of this article to clearly and completely describe the technical solutions in the embodiments of this article. Obviously, the described embodiments are only part of the embodiments of this article, not all of the embodiments. Based on the embodiments of this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article.

[0084] The target prediction method in the prior art only predicts single-frame image data. For a video, the prior art mostly collects the picture of the current frame in the video and predicts the target in the video by predicting the picture of the current frame. However, for moving targets such as the human body, the human body in the picture of the current frame is a small target because the human body is far away from the camera and the camera resolution is low. When predicting the picture of the current frame, it is very easy to cause the small target information to be missing. Therefore, the target prediction method in the prior art is prone to small target omission and false detection.

[0085] In order to solve the above problems, the embodiments of this article provide a method for predicting small objects in a video. Figure 1 It is a flowchart of a method for predicting small targets in a video provided in an embodiment of this invention. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only order of execution. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiment or the accompanying drawings or in parallel.

[0086] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0087] Banks are generally equipped with multiple cameras to collect videos of the bank's daily work process. When necessary, these videos can be extracted to obtain portraits or objects in the video. However, there are often many small targets in the video that need to be predicted and identified. To this end, this paper proposes a prediction method for small targets in videos, which can be used on the server to predict small targets in videos.

[0088] Reference Figure 1 , a method for predicting small objects in a video, comprising:

[0089] S101: extracting features of multiple scales from the image of the current frame through a backbone network and a neck network in a neural network model to obtain a multi-scale feature map, wherein the multi-scale feature map includes a small-scale feature map corresponding to a small target;

[0090] S102: Obtaining enhancement information corresponding to the small target according to a picture of a previous frame corresponding to the current frame, wherein the previous frame is a frame corresponding to a moment before the current frame;

[0091] S103: performing small-scale information enhancement processing on the small-scale feature map according to the enhancement information to obtain an enhanced small-scale feature map;

[0092] S104: Input the enhanced small-scale feature map into the head network in the neural network model, predict the position information, confidence and classification probability corresponding to the small target, and then obtain the probability of the existence of the small target at the position indicated by the position information, and the category to which the small target belongs.

[0093] The CNN neural network model is generally composed of three parts: backbone, neck, and head. The backbone network is generally used to realize feature extraction, the neck network is generally used to realize feature enhancement, and the head part is generally used to realize result output. The picture of the current frame in the video is first input into the backbone network for feature extraction, and then input into the neck network for feature enhancement, so that multiple feature maps of different scales can be obtained, including small-scale feature maps corresponding to small targets in the multiple feature maps of different scales. It should be clear that the small-scale feature map described in this article has a clear meaning, which refers to the smallest scale among multiple feature maps of different scales, and the target in the small-scale feature map is the small target.

[0094] For small-scale feature maps, it is necessary to perform small-scale information enhancement processing on them. The enhanced small-scale feature maps have a large amount of information and clear and complete features compared to the previous ones, which can reduce the occurrence of missed detections and false detections due to insufficient information on small targets. Small-scale information enhancement processing requires enhanced information, which is obtained based on the image of the previous frame corresponding to the previous moment of the current frame.

[0095] Finally, the enhanced small-scale feature map is input into the head network to predict the position information, confidence and classification probability of the small target. The position of the small target can be obtained according to the position information, the probability of the small target existing at the position can be obtained according to the confidence, and the category to which the small target at the position may belong can be obtained according to the classification probability. According to the position information, confidence and classification probability, the small target can be determined to be a certain type of object at a certain position.

[0096] In this paper, the enhanced small-scale feature map is input into the head network to predict the position information, confidence and classification probability corresponding to the small target. It can also be processed by NMS (non-maximum suppression) and other methods to filter out the position information with low confidence and improve the accuracy of small target prediction.

[0097] In the embodiments of this article, after the small-scale feature map corresponding to the small target is subjected to small-scale information enhancement processing, the amount of information in the small-scale feature map corresponding to the small target becomes larger and the features are clear and complete, so that the small target can be determined more accurately and clearly, and the possibility of missed detection or false detection of the small target is reduced.

[0098] Reference Figure 2 In the embodiment of this article, it is necessary to further determine the enhancement information corresponding to the small target, and obtaining the enhancement information corresponding to the small target according to the picture of the previous frame corresponding to the current frame further includes:

[0099] S201: Obtain a multi-scale front enhancement mask feature map according to a picture of a previous frame corresponding to the current frame;

[0100] S202: Performing spatiotemporal fusion processing on the multi-scale front-enhancement mask feature map to obtain enhancement information corresponding to the small target.

[0101] Reference Figure 3 Specifically, S201 further includes:

[0102] S301: obtaining a multi-scale front feature map according to a picture of a previous frame corresponding to a current frame, wherein the multi-scale front feature map includes a small-scale front feature map corresponding to a small target in the previous frame;

[0103] S302: performing small-scale information enhancement processing on the small-scale front feature map to obtain a processed multi-scale front feature map;

[0104] S303: performing a mask enhancement operation on the processed multi-scale front feature map to obtain a multi-scale front enhanced mask feature map.

[0105] The picture of the previous frame refers to the picture of the frame corresponding to the previous moment of the current frame. According to the picture of the previous frame, a multi-scale feature map corresponding to the previous frame can be obtained, and the multi-scale feature map corresponding to the previous frame is the multi-scale previous feature map. In the embodiment of this article, S301 further includes:

[0106] Multi-scale feature extraction is performed on the image of the previous frame corresponding to the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale previous feature map.

[0107] Similar to the method of obtaining the multi-scale feature map of the current frame, the multi-scale front feature map of the previous frame is obtained by using the backbone network and the neck network.

[0108] After obtaining the multi-scale front feature map, it is also necessary to perform small-scale information enhancement processing on the small-scale front feature map. It should be clear that, except for the small-scale front feature map in the processed multi-scale front feature map, other multi-scale front feature maps have not been subjected to information enhancement processing. Then, it is necessary to perform mask enhancement operation on the processed multi-scale front feature map to obtain a multi-scale front enhanced mask feature map.

[0109] As shown in S202, it is necessary to perform spatiotemporal fusion on the multi-scale front-enhancement mask feature maps. After fusing the front-enhancement mask feature maps of multiple different scales, the enhancement information corresponding to the small target can be obtained.

[0110] It can be understood that in order to obtain the enhancement information corresponding to the small target and then perform small-scale information enhancement processing on the small-scale feature map through the enhancement information, it is necessary to first perform small-scale information enhancement processing on the small-scale front feature map through step S302.

[0111] Reference Figure 4 Specifically, S302 further includes:

[0112] S401: obtaining previous enhancement information corresponding to a small target in the previous frame according to a picture of a historical frame corresponding to the previous frame, wherein the historical frame is a frame corresponding to a moment before the previous frame;

[0113] S402: Performing small-scale information enhancement processing on the small-scale front feature map using the front enhancement information to obtain a processed multi-scale front feature map.

[0114] It can be concluded from the above S401 to S402 that when performing small-scale information enhancement processing on the small-scale front feature map, it is necessary to first obtain the front enhancement information corresponding to the small target in the previous frame, and then perform small-scale information enhancement processing on the small-scale front feature map according to the front enhancement information.

[0115] It should be clarified that the historical frame is the frame corresponding to the previous moment of the previous frame. The small target in the previous frame in S401 is not necessarily the same as the small target described in S101. The small target described in S101 is the small target in the current frame, which is also the small target that needs to be predicted in this article. The small target in the previous frame refers to the small target corresponding to the small-scale front feature map of the smallest scale in the multi-scale front feature map of the previous frame. The small target in the previous frame may be the same as the small target in the current frame, or it may be different.

[0116] However, according to the above S401 to S402, it can be obtained that: when it is necessary to determine the enhancement information corresponding to the small target in the current frame based on the picture of the previous frame, it is necessary to determine the previous enhancement information corresponding to the small target in the previous frame based on the picture of the historical frame. Therefore, the enhancement information corresponding to the small target in the current frame is obtained by iterating the pictures of the frames at each moment before the current frame, and after obtaining the enhancement information based on the picture of the frame at the initial moment, it is iterated to the small-scale feature map in the frame at the next moment, and then the enhancement information is obtained based on the picture of the frame at this moment, and then it is iterated to the small-scale feature map in the frame at the next moment corresponding to this moment, and so on.

[0117] Reference Figure 5 In the embodiment of this article, S303 further includes:

[0118] S501: performing mask operations on the processed multi-scale front feature maps respectively to obtain multi-scale front mask feature maps;

[0119] S502: Perform information enhancement on the multi-scale front mask feature maps respectively to obtain multi-scale front enhanced mask feature maps.

[0120] Reference Figure 6 , where S501 is specifically:

[0121] S601: Inputting the processed multi-scale front feature maps into the head network in the neural network model respectively, and predicting the front position information, front confidence and front classification probability corresponding to the multi-scale targets respectively;

[0122] S602: Determine front object frames corresponding to the multi-scale objects respectively according to the front position information corresponding to the multi-scale objects respectively;

[0123] S603: setting the background pixels in the front target frames corresponding to the multi-scale targets to 0 to obtain a multi-scale front mask feature map.

[0124] The purpose of setting the background pixels in the front target frame corresponding to the multi-scale targets to 0 is to filter out the background information in the front target frame corresponding to the multi-scale targets and only retain the multi-scale target area information, which is the multi-scale front mask feature map.

[0125] Wherein S502 further includes:

[0126] The multi-scale front-enhancement mask feature map is calculated by the following formula:

[0127]

[0128]

[0129] Among them, i is any scale in the multi-scale, is the previous feature map of any scale after processing, is the front mask feature map of any scale, is the prior confidence corresponding to the target of any scale, is the maximum value of the previous classification probability corresponding to the target of any scale, is the front enhanced mask feature map of any scale, where ⊙ is a logical operator, indicating the XOR operation.

[0130] The multi-scale front mask feature map is enhanced respectively by the above formula to obtain a multi-scale front enhanced mask feature map. The enhanced mask feature map reversely optimizes the feature information through the target detection output result, enhances the feature value of the target area in the feature map, suppresses the feature value of the non-target area in the feature map, strengthens the target information of each scale, reduces background interference, and further improves the subsequent information fusion and small target detection effect. After obtaining the multi-scale front enhanced mask feature map, it is necessary to perform spatiotemporal fusion processing on it according to S202 to obtain the enhanced information corresponding to the small target.

[0131] Reference Figure 7 Further, the performing spatiotemporal fusion processing on the multi-scale front-enhanced mask feature map to obtain the enhanced information corresponding to the small target further includes:

[0132] S701: Perform spatial domain fusion on the multi-scale front-enhancement mask feature map to obtain spatial domain features;

[0133] S702: Fusing the spatial domain features with the RNN temporal domain features to obtain enhanced information corresponding to the small target.

[0134] Reference Figure 8 , wherein S701 further comprises:

[0135] S801: using an average pooling operator to unify the multi-scale dimensions of the multi-scale front-enhancement mask feature map to obtain a multi-scale feature;

[0136] S802: Use a self-attention mechanism to spatially fuse the multi-scale features to obtain the spatial domain features.

[0137] Specifically, the avgPooling average pooling operator is used to unify the multi-scale dimensions, and then the self-attention mechanism is used to adaptively fuse the multi-scale features in space. It should be noted that the goal of average pooling is to sample the scale features other than the small-scale features into the dimension of the small-scale features, so as to perform self-attention fusion in the same dimension and form a fused feature.

[0138] For S702, the spatial domain features of the previous moment corresponding to the current frame are input into the RNN network ConvLSTM, and the memory, fusion and prediction capabilities of ConvLSTM are used to predict the enhanced information corresponding to the small target in the current frame to achieve spatial domain feature fusion.

[0139] In the embodiment of this article, performing small-scale information enhancement processing on the small-scale feature map by using the enhancement information to obtain the enhanced small-scale feature map in S103 further includes:

[0140] The enhanced small-scale feature map is calculated by the following formula:

[0141]

[0142] in, is the enhanced small-scale feature map, is a small-scale feature map, h (t) To enhance information.

[0143] For various targets, especially moving targets such as human bodies, the current frame lacks small target information due to its long distance from the camera and low resolution. Traditional target detection models are prone to missed detection and false detection of small targets. However, there may be targets with close distance and high resolution in the frames before the current frame. This paper solves the problem of insufficient small target information in the current frame by fusing the images of the frames before the current frame, thereby reducing the probability of missed detection and false detection of small targets.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0145] Based on the above-described method for predicting small targets in a video, the embodiments of this article also provide a prediction device for small targets in a video. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiments of this article and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of this article is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiments of this article can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0146] Specifically, Fig. 9 This is a schematic diagram of the module structure of an embodiment of a device for predicting small objects in a video provided by the embodiments of this article, referring to Fig. 9 As shown, a prediction device for small objects in a video provided by an embodiment of the present invention includes: a feature extraction module 100, an enhancement information determination module 200, an enhancement module 300, and a prediction module 400.

[0147] The feature extraction module 100 is used to extract features of multiple scales on the image of the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale feature map, wherein the multi-scale feature map includes a small-scale feature map corresponding to a small target;

[0148] The enhancement information determination module 200 is used to obtain the enhancement information corresponding to the small target according to the picture of the previous frame corresponding to the current frame, wherein the previous frame is the frame corresponding to the previous moment of the current frame;

[0149] An enhancement module 300 is used to perform small-scale information enhancement processing on the small-scale feature map according to the enhancement information to obtain an enhanced small-scale feature map;

[0150] The prediction module 400 is used to input the enhanced small-scale feature map into the head network in the neural network model, predict the position information, confidence and classification probability corresponding to the small target, and then obtain the probability of the existence of the small target at the position indicated by the position information, and the category to which the small target belongs.

[0151] Reference Fig.10 As shown, based on the above-mentioned method for predicting small targets in a video, a computer device 1002 is also provided in one embodiment of this invention, wherein the above-mentioned method is run on the computer device 1002. The computer device 1002 may include one or more processors 1004, such as one or more central processing units (CPUs) or graphics processing units (GPUs), and each processing unit may implement one or more hardware threads. The computer device 1002 may also include any memory 1006, which is used to store any kind of information such as code, settings, data, etc. In a specific implementation, the computer program on the memory 1006 and which can be run on the processor 1004, when the computer program is run by the processor 1004, can execute instructions according to the above-mentioned method. Non-limitingly, for example, the memory 1006 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1002. In one embodiment, when the processor 1004 executes the associated instructions stored in any memory or combination of memories, the computer device 1002 can perform any operation of the associated instructions. The computer device 1002 also includes one or more drive mechanisms 1008 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0152] The computer device 1002 may also include an input / output module 1010 (I / O) for receiving various inputs (via input device 1012) and for providing various outputs (via output device 1014). A specific output mechanism may include a presentation device 1016 and an associated graphical user interface 1018 (GUI). In other embodiments, the input / output module 1010 (I / O), input device 1012, and output device 1014 may not be included, and the computer device 1002 may be used as a computer device in a network. The computer device 1002 may also include one or more network interfaces 1020 for exchanging data with other devices via one or more communication links 1022. One or more communication buses 1024 couple the components described above together.

[0153] The communication link 1022 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1022 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0154] Corresponds to Figure 1-Figure 8 The method in this article also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0155] The embodiment of the present invention also provides a computer readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 8 The method shown.

[0156] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0157] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0158] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0162] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0164] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. A method for predicting small objects in a video, characterized in that: include: Extract features of multiple scales from the image of the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale feature map, wherein the multi-scale feature map includes a small-scale feature map corresponding to a small target; Obtaining enhancement information corresponding to the small target according to a picture of a previous frame corresponding to the current frame, wherein the previous frame is a frame corresponding to a moment before the current frame; Performing small-scale information enhancement processing on the small-scale feature map according to the enhancement information to obtain an enhanced small-scale feature map; Inputting the enhanced small-scale feature map into the head network in the neural network model, predicting the position information, confidence and classification probability corresponding to the small target, and then obtaining the probability of the existence of the small target at the position indicated by the position information and the category to which the small target belongs; The obtaining, according to the picture of the previous frame corresponding to the current frame, the enhancement information corresponding to the small target further comprises: Obtaining a multi-scale front-enhancement mask feature map according to a picture of a previous frame corresponding to the current frame; Performing spatiotemporal fusion processing on the multi-scale front-enhanced mask feature map to obtain enhanced information corresponding to the small target; The step of obtaining a multi-scale front enhancement mask feature map according to a picture of a previous frame corresponding to the current frame further includes: According to the picture of the previous frame corresponding to the current frame, a multi-scale front feature map is obtained, wherein the multi-scale front feature map includes a small-scale front feature map corresponding to the small target in the previous frame; Performing small-scale information enhancement processing on the small-scale front feature map to obtain a processed multi-scale front feature map; A mask enhancement operation is performed on the processed multi-scale front feature map to obtain a multi-scale front enhanced mask feature map.

2. The method for predicting small objects in a video according to claim 1, characterized in that: The obtaining of a multi-scale front feature map according to a picture of a previous frame corresponding to the current frame further includes: Multi-scale feature extraction is performed on the image of the previous frame corresponding to the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale previous feature map.

3. The method for predicting small objects in a video according to claim 1, characterized in that: The performing small-scale information enhancement processing on the small-scale front feature map to obtain a processed multi-scale front feature map further comprises: Obtaining previous enhancement information corresponding to the small target in the previous frame according to a picture of a historical frame corresponding to the previous frame, wherein the historical frame is a frame corresponding to a previous moment of the previous frame; The small-scale front feature map is subjected to small-scale information enhancement processing by using the front enhancement information to obtain a processed multi-scale front feature map.

4. The method for predicting small objects in a video according to claim 1, characterized in that: The step of performing an enhancement mask enhancement operation on the processed multi-scale front feature map to obtain a multi-scale front enhanced mask feature map further comprises: Performing mask operations on the processed multi-scale front feature maps respectively to obtain multi-scale front mask feature maps; Information enhancement is performed on the multi-scale front mask feature maps respectively to obtain multi-scale front enhanced mask feature maps.

5. The method for predicting small objects in a video according to claim 4, characterized in that: The performing mask operations on the processed multi-scale front feature maps respectively to obtain the multi-scale front mask feature maps further comprises: Inputting the processed multi-scale front feature maps into the head network in the neural network model respectively, and predicting the front position information, front confidence and front classification probability corresponding to the multi-scale targets respectively; Determining front target frames corresponding to the multi-scale targets respectively according to the front position information corresponding to the multi-scale targets respectively; The background pixels in the front target frames corresponding to the multi-scale targets are all set to 0 to obtain a multi-scale front mask feature map.

6. The method for predicting small objects in a video according to claim 5, characterized in that: The performing information enhancement on the multi-scale front mask feature maps respectively to obtain the multi-scale front enhanced mask feature maps further comprises: The multi-scale front enhancement mask feature map is calculated by the following formula: Among them, i is any scale in the multi-scale, is the previous feature map of any scale after processing, is the front mask feature map of any scale, is the prior confidence corresponding to the target of any scale, is the maximum value of the previous classification probability corresponding to the target of any scale, is the front-enhanced mask feature map of any scale.

7. The method for predicting small objects in a video according to claim 1, characterized in that: The performing spatiotemporal fusion processing on the multi-scale front-enhanced mask feature map to obtain the enhanced information corresponding to the small target further includes: Performing spatial domain fusion on the multi-scale front-enhancement mask feature map to obtain spatial domain features; The spatial domain features are fused with RNN temporal domain features to obtain enhanced information corresponding to the small target.

8. The method for predicting small objects in a video according to claim 7, characterized in that: The performing spatial domain fusion on the multi-scale front-enhancement mask feature map to obtain the spatial domain feature further comprises: Using an average pooling operator to unify the multi-scale dimensions of the multi-scale front-enhanced mask feature map to obtain a multi-scale feature; The multi-scale features are spatially fused using a self-attention mechanism to obtain the spatial domain features.

9. The method for predicting small objects in a video according to claim 1, characterized in that: The performing small-scale information enhancement processing on the small-scale feature map by using the enhancement information to obtain the enhanced small-scale feature map further comprises: The enhanced small-scale feature map is calculated by the following formula: in, is the enhanced small-scale feature map, is a small-scale feature map, h (t) To enhance information.

10. A device for predicting small objects in a video, characterized in that: The device comprises: A feature extraction module is used to extract features of multiple scales on the image of the current frame through the backbone network and the neck network in the neural network model to obtain a multi-scale feature map, wherein the multi-scale feature map includes a small-scale feature map corresponding to a small target; An enhancement information determination module, used to obtain enhancement information corresponding to the small target according to a picture of a previous frame corresponding to the current frame, wherein the previous frame is a frame corresponding to a moment before the current frame; An enhancement module, used to perform small-scale information enhancement processing on the small-scale feature map according to the enhancement information to obtain an enhanced small-scale feature map; A prediction module is used to input the enhanced small-scale feature map into the head network in the neural network model, predict the position information, confidence and classification probability corresponding to the small target, and then obtain the probability of the existence of the small target at the position indicated by the position information, and the category to which the small target belongs; The obtaining, according to the picture of the previous frame corresponding to the current frame, the enhancement information corresponding to the small target further comprises: Obtaining a multi-scale front-enhancement mask feature map according to a picture of a previous frame corresponding to the current frame; Performing spatiotemporal fusion processing on the multi-scale front-enhanced mask feature map to obtain enhanced information corresponding to the small target; The step of obtaining a multi-scale front enhancement mask feature map according to a picture of a previous frame corresponding to the current frame further includes: According to the picture of the previous frame corresponding to the current frame, a multi-scale front feature map is obtained, wherein the multi-scale front feature map includes a small-scale front feature map corresponding to the small target in the previous frame; Performing small-scale information enhancement processing on the small-scale front feature map to obtain a processed multi-scale front feature map; A mask enhancement operation is performed on the processed multi-scale front feature map to obtain a multi-scale front enhanced mask feature map.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes the instructions of the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that When the computer program product is executed by a processor of a computer device, the computer program product executes the instructions of the method according to any one of claims 1 to 9.

Images