A video detection method, device, storage medium and equipment
By using a backbone network and a modified Transformer model for feature interaction and fusion processing, the problems of singleness and redundant parameters in existing video object detection methods are solved, realizing unified detection of multiple types of target objects and improving the effect and generalization ability of video detection.
Patent Information
- Application Number
- CN202210712928.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-22
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2042-06-22
AI Technical Summary
Existing video object detection methods typically only achieve detection of a single type, and suffer from problems such as overfitting to specific type characteristics, lack of generalization ability, and redundant parameters, resulting in poor detection performance.
A backbone network is used to extract classification features from the reference frame image and the target frame image respectively. The modified Transformer model is used for feature interaction processing. By fusing the reference prior tensor and the interaction feature matrix, unified detection of single target tracking, multi-target tracking, video target segmentation, and multi-target tracking and segmentation is achieved.
It enables the simultaneous detection of four different types of target objects, improving the effect and generalization ability of video detection, reducing redundant parameters, and improving the accuracy and efficiency of detection.
Smart Images
Figure CN115187895B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and particularly relates to a video detection method and device, a storage medium and equipment. BACKGROUND
[0002] With the explosive growth of video data such as intelligent video monitoring, face recognition access control systems, automatic driving, robot vision navigation, which are close to people's daily life, the research of video target detection has greater practical research significance and application value.
[0003] At present, when video target detection is performed, it is usually divided into four types for tracking detection, which are: single target tracking, multi-target tracking, video target segmentation, multi-target tracking and segmentation. This leads to the fact that most existing tracking detection methods can only realize single type detection or partial type detection. This type of tracking detection method has the following two shortcomings: first, the tracking detector may overfit the characteristics of a specific type, and lacks generalization ability; second, independent model setting will bring redundant parameters, for example: a detection tracker based on deep learning usually adopts a similar backbone network structure, but the independent design philosophy of different detection types hinders the reuse of model parameters. It can be seen that the current video target detection method has poor detection effect. SUMMARY
[0004] The main purpose of the embodiments of the present application is to provide a video detection method, device, storage medium and equipment, which can simultaneously realize single target tracking, multi-target tracking, video target segmentation, multi-target tracking and segmentation in a video, thereby improving the video detection effect.
[0005] The embodiments of the present application provide a video detection method, comprising:
[0006] Obtaining a target video to be detected, and cutting a reference frame image and a target frame image from the target video;
[0007] Using a backbone network to respectively perform classification feature extraction on the reference frame image and the target frame image, to obtain respective hierarchical features of the reference frame image and the target frame image;
[0008] Performing interactive processing on the respective hierarchical features of the reference frame image and the target frame image, to obtain an interactive feature matrix;
[0009] Processing the reference frame image to obtain a reference prior tensor; and performing fusion processing on the reference prior tensor and the interactive feature matrix to obtain a target prior tensor;
[0010] The target prior tensor is fused with the hierarchical features corresponding to the reference frame image and the target frame image respectively, and a target object in the target video is detected according to a processing result obtained, to obtain a detection result.
[0011] In a possible implementation, the hierarchical features corresponding to the reference frame image and the target frame image are obtained by performing classification feature extraction on the reference frame image and the target frame image respectively by using a backbone network.
[0012] The hierarchical features corresponding to the reference frame image and the target frame image are obtained by performing classification feature extraction on the reference frame image and the target frame image respectively by using a weight-shared backbone network.
[0013] In a possible implementation, the hierarchical features are feature pyramid network (FPN) features.
[0014] In a possible implementation, the interaction between the hierarchical features corresponding to the reference frame image and the target frame image is performed to obtain an interaction feature matrix, and the interaction feature matrix includes:
[0015] The interaction between the hierarchical features corresponding to the reference frame image and the target frame image is performed by using a modified neural network Transformer model to obtain an interaction feature matrix.
[0016] The modified Transformer model is obtained by replacing a full attention layer in a Transformer model structure with a deformable attention layer.
[0017] In a possible implementation, the reference prior tensor is obtained by processing the reference frame image, and the processing includes:
[0018] The reference prior tensor is obtained by performing binarization processing on the reference frame image.
[0019] In a possible implementation, the target prior tensor is obtained by fusing the reference prior tensor and the interaction feature matrix, and the fusing includes:
[0020] The target prior tensor is obtained by multiplying the reference prior tensor and the interaction feature matrix.
[0021] In a possible implementation, the target object in the target video is detected according to a processing result obtained by fusing the target prior tensor with the hierarchical features corresponding to the reference frame image and the target frame image, and the detection result includes:
[0022] The target prior tensor and the hierarchical features corresponding to the reference frame image and the target frame image are fused by addition in a broadcast form, and a target object in the target video is detected according to a processing result, to obtain a detection result.
[0023] The embodiment of the present application also provides a video detection device, which comprises:
[0024] An acquisition unit is configured to acquire a target video to be detected, and cut a reference frame image and a target frame image from the target video;
[0025] An extraction unit is configured to perform classification feature extraction on the reference frame image and the target frame image respectively by using a backbone network, to obtain hierarchical features corresponding to the reference frame image and the target frame image respectively;
[0026] An interaction unit is configured to perform interaction processing on the hierarchical features corresponding to the reference frame image and the target frame image respectively, to obtain an interaction feature matrix;
[0027] A fusion unit is configured to perform processing on the reference frame image to obtain a reference prior tensor, and perform fusion processing on the reference prior tensor and the interaction feature matrix to obtain a target prior tensor;
[0028] A detection unit is configured to perform fusion processing on the target prior tensor and the hierarchical features corresponding to the reference frame image and the target frame image respectively, and detect a target object in the target video according to a processing result, to obtain a detection result.
[0029] In a possible implementation manner, the extraction unit is specifically configured to:
[0030] The classification feature extraction is performed on the reference frame image and the target frame image respectively by using a weight-shared backbone network, to obtain the hierarchical features corresponding to the reference frame image and the target frame image respectively.
[0031] In a possible implementation manner, the hierarchical features are feature pyramid network (FPN) features.
[0032] In a possible implementation manner, the interaction unit is specifically configured to:
[0033] The interaction processing is performed on the hierarchical features corresponding to the reference frame image and the target frame image respectively by using a modified neural network Transformer model, to obtain the interaction feature matrix.
[0034] The modified Transformer model is obtained by replacing a full attention layer in a Transformer model structure with a deformable attention layer.
[0035] In a possible implementation, the fusion unit is specifically configured to:
[0036] The reference frame image is binarized to obtain a reference prior tensor.
[0037] In a possible implementation, the fusion unit is specifically further configured to:
[0038] The reference prior tensor and the interaction feature matrix are multiplied to obtain a target prior tensor.
[0039] In a possible implementation, the detection unit is specifically configured to:
[0040] The target prior tensor and the respective hierarchical features of the reference frame image and the target frame image are fused through addition in a broadcast form, and a target object in the target video is detected according to a processing result to obtain a detection result.
[0041] Embodiments of the present application further provide a video detection device, which comprises a processor, a memory and a system bus.
[0042] The processor and the memory are connected through the system bus.
[0043] The memory is configured to store one or more programs, and the one or more programs comprise instructions, which, when executed by the processor, cause the processor to perform any one of the implementation manners of the video detection method.
[0044] Embodiments of the present application further provide a computer readable storage medium, which stores instructions, and when the instructions run on a terminal device, cause the terminal device to perform any one of the implementation manners of the video detection method.
[0045] The video detection method, device, storage medium and equipment provided by the embodiment of the application first acquire a target video to be detected, and intercept a reference frame image and a target frame image from the target video, then respectively extract classification features of the reference frame image and the target frame image by using a backbone network to obtain respective hierarchical features of the reference frame image and the target frame image; next, the respective hierarchical features of the reference frame image and the target frame image are interactively processed to obtain an interactive feature matrix, the reference frame image is processed to obtain a reference prior tensor, the reference prior tensor and the interactive feature matrix are fused to obtain a target prior tensor, and the target prior tensor, the respective hierarchical features of the reference frame image and the target frame image are fused, and the target object in the target video is detected according to the obtained processing result to obtain a detection result. It can be seen that, since the embodiment of the application does not perform detection on the target object by type, but simultaneously realizes detection of the target object corresponding to any one or more types of single target tracking, multi-target tracking, video target segmentation, multi-target tracking and segmentation in four different types by using a unified detection manner, the video detection effect is improved. BRIEF DESCRIPTION OF DRAWINGS
[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0047] Figure 1 A flowchart of a video detection method provided by the embodiment of the present application is shown in the figure.
[0048] Figure 2 A whole schematic diagram of a video detection process provided by the embodiment of the present application is shown in the figure.
[0049] Figure 3 A composition schematic diagram of a video detection device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0050] As is known to all, video target detection is one of the core problems in the field of computer vision, which aims to establish a pixel-level or instance-level correspondence between frames and output the trajectory of the target in the form of a bounding box or a mask.
[0051] Currently, when detecting target objects in a video, four types of tracking detection are usually divided, which are: single object tracking (SOT), multiple object tracking (MOT), video object segmentation (VOS), and multiple object tracking and segmentation (MOTS). This results in most existing tracking detection methods being able to only implement single type detection or partial type detection, which has the following two shortcomings: one is that the tracking detector may overfit the characteristics of a specific type, and lacks generalization ability; the other is that independent models will bring redundant parameters, resulting in poor video target detection detection effect.
[0052] To solve the above defects, the present application provides a video detection method, first, a target video to be detected is obtained, and a reference frame image and a target frame image are intercepted from the target video, then the reference frame image and the target frame image are classified and feature extracted by using a backbone network, to obtain respective hierarchical features corresponding to the reference frame image and the target frame image; next, the respective hierarchical features corresponding to the reference frame image and the target frame image are interactively processed to obtain an interactive feature matrix, the reference frame image is processed to obtain a reference prior tensor, and the reference prior tensor and the interactive feature matrix are fused to obtain a target prior tensor; further, the target prior tensor, the respective hierarchical features corresponding to the reference frame image and the target frame image are fused, and the target objects in the target video are detected according to the obtained processing result to obtain a detection result. It can be seen that, since the embodiments of the present application do not divide the detection of target objects into types, but simultaneously implement the detection of target objects corresponding to at least one type of single target tracking, multiple target tracking, video target segmentation, and multiple target tracking and segmentation in the target video by using a unified detection method, the video detection effect is improved.
[0053] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.
[0054] First embodiment
[0055] Reference is made to Figure 1A flowchart of a video detection method provided in the embodiment is shown in the figure, and the method comprises the following steps:
[0056] S101: Obtain a target video to be detected, and cut a reference frame image and a target frame image from the target video.
[0057] In the embodiment, any video on which the target object detection is performed in the embodiment is defined as a target video. Moreover, the target video can be obtained by shooting, downloading, etc. according to actual needs, and the target video can be saved as a target video, for example, a driving video on a road within a period of time shot by a monitoring camera installed on a roadside, or a scenic spot introduction video downloaded from a network.
[0058] In order to simultaneously realize detection of target objects corresponding to any one or more of single target tracking, multi-target tracking, video target segmentation, and multi-target tracking and segmentation in a target video, and improve the detection effect of the target objects in the target video, after obtaining the target video to be detected, the reference frame image and the target frame image are first cut from the target video for performing the subsequent step S102.
[0059] It should be noted that the type of the reference frame image and the target frame image is not limited in the embodiment, for example, the reference frame image and the target frame image can be a color image composed of red (R), green (G), and blue (B) three primary colors, or a grayscale image, etc.
[0060] Moreover, in order to efficiently locate a plurality of potential target objects possibly existing in the target video, the reference frame image and the target frame image cut by the application are both the entire image of the target video, instead of a partial search region in the video. This also makes the application have high resistance to tracking failure when performing target image tracking detection, and the target object can be detected again in the subsequent frame image after the target object disappears.
[0061] It should be noted that in order to further improve the detection effect, when it is determined that only the detection of the target object corresponding to the single target tracking is performed, the reference frame image and the target frame image cut by the application are the entire image of the first frame and the current frame to be detected of the target video, respectively; when it is determined that only the detection of the target object corresponding to the multi-target tracking is performed, the target frame image and the reference frame image cut by the application are the entire image of the current frame to be detected and the previous frame of the target video, respectively.
[0062] S102: Perform classification feature extraction on the reference frame image and the target frame image by using a backbone network, to obtain hierarchical features corresponding to the reference frame image and the target frame image, respectively.
[0063] In the embodiment, after the reference frame image and the target frame image in the target video are intercepted by step S101, further, the reference frame image and the target frame image can be classified and feature extracted by using an existing or future backbone network, such as a VGG (Visual Geometry Group Network) network model or a deep residual network (Deep residual network, referred to as ResNet), to obtain respective hierarchical features of the reference frame image and the target frame image, and then the subsequent steps S103-S105 are executed to realize accurate detection of the target object in the target video.
[0064] It should be noted that, in order to improve the accuracy of the detection result, one optional implementation is that the application adopts a weight-shared backbone network to respectively classify and extract features of the reference frame image and the target frame image, as shown in FIG. 1. Figure 2 The extracted feature pyramid network (Feature Pyramid Networks, referred to as FPN) features are used as respective hierarchical features of the reference frame image and the target frame image. Here, the feature vector corresponding to the FPN features can be defined as F, and F ∈ R h×w×c where h represents the height (Height) of the FPN feature vector F, w represents the width (Width) of the FPN feature vector F, and c represents the channel number (Channel) corresponding to the FPN feature vector F.
[0065] S103: Interactively processing the respective hierarchical features of the reference frame image and the target frame image to obtain an interactive feature matrix.
[0066] It should be noted that, since the information interaction between different frame images is not considered when detecting the target object in the video at present, fuzzy or incorrect matching may occur in the tracking and detection process, especially when there are many similar patterns or objects on the target frame image, which may result in poor detection effect.
[0067] Therefore, in the embodiment, in order to more accurately detect the target object in the target video, after obtaining the respective hierarchical features of the reference frame image and the target frame image by step S102, further, the respective hierarchical features of the reference frame image and the target frame image need to be interactively processed to obtain an interactive feature matrix, for example, the respective FPN features of the reference frame image and the target frame image need to be interactively processed to obtain an interactive feature matrix, as shown in the “interaction” processing process of the “feature interaction” position in FIG. 1, to execute the subsequent step S104. Figure 2
[0068] Specifically, one possible implementation is to achieve feature interaction to enhance the information of two frames and improve the discriminative ability of image features. This application uses a modified neural network Transformer model to interactively process the hierarchical features (such as FPN features) corresponding to the reference frame and the target frame, obtaining an interactive feature matrix. This is because the excellent long-distance association modeling capability of the Transformer model is considered, which helps to improve the interaction effect. Therefore, it is thought that the Transformer model can be used for feature interaction processing. However, when the reference frame and the target frame are of high resolution, using a conventional Transformer model will result in significant memory consumption. Therefore, to solve this problem, this application replaces the "full attention layer" in the conventional Transformer model structure with a "deformable attention layer", obtaining a modified Transformer model. Further using the modified Transformer model for feature interaction processing can improve the accuracy of the interaction results.
[0069] S104: By processing the reference frame image, a reference prior tensor is obtained; and the reference prior tensor and the interaction feature matrix are fused to obtain the target prior tensor.
[0070] It should be noted that, since the core task of target tracking and detection is to establish accurate correspondences between video frame images, for single-target tracking and video target segmentation, existing or future pixel-by-pixel correspondences C can be used. pix The method of determination is based on the obtained C. pix The target object to be detected (which can be pre-specified by the user) in the reference frame (usually the first frame) of the target video is propagated to the current frame image, providing a strong prior for the final bounding box or mask prediction. For multi-object tracking and multi-object tracking and segmentation, a correspondence C at the level of existing or future instances (i.e., detected objects) can be used. inst The method of determination is based on the obtained C. inst The image to be detected (which can be pre-specified by the user) in the reference frame (usually the first frame) of the target video is used to help correlate (e.g.) Figure 2 (As shown in the "association" section) The existing trajectory in the reference frame image and the target object detected in the current frame image can also provide a strong prior for the final bounding box or mask prediction.
[0071] Specifically, since the four different detection types of single target tracking, multi-target tracking, video target segmentation, multi-target tracking and segmentation in the target video need to detect different target objects, such as multi-target tracking needs to detect target objects of a given category, single target tracking needs to detect target objects of any category in the reference frame image to be detected (which can be specified by the user in advance).
[0072] In order to realize the detection of different target objects by using a unified network structure, that is, to realize the detection of the target objects corresponding to at least one of the four different types of single target tracking, multi-target tracking, video target segmentation, multi-target tracking and segmentation in the target video, so as to improve the detection effect of the target objects in the target video. In this embodiment, after obtaining the reference frame image and the target frame image cut from the target video through step S101, further, the reference frame image can be detected and processed by using existing or future image detection methods to obtain the reference prior tensor. For example, the reference frame image can be processed by binaryzation to obtain the reference target feature map, such as the "reference target" SOT, VOS, MOT, MOTS and the corresponding reference prior tensor shown in Figure 2
[0073] Among them, Figure 2 The MOT and MOTS in the "reference target" shown in are only schematic representations, and in actual applications, the priors corresponding to multi-target tracking and multi-target tracking and segmentation are both equal to 0.
[0074] Further, after obtaining the reference prior tensor (such as the "reference target" SOT, VOS shown in Figure 2 Further, after obtaining the reference prior tensor (such as the "reference target" SOT, VOS shown in Figure 2 The processing process of "propagation" in Figure 2 The feature map shown in the "target prior" SOT, VOS, MOT, MOTS position, is used to perform the subsequent step S105.
[0075] The feature map corresponding to the target prior tensor can be a feature map with a shape of h x w x 1, where h represents the height of the target prior tensor, w represents the width of the target prior tensor, and 1 represents that the number of channels corresponding to the target prior tensor is 1. For single target tracking and video target segmentation, the target prior tensor is a result of propagation from a reference frame image, and provides strong prior information for subsequent detection. For multi-target tracking and multi-target tracking and segmentation, the target prior tensor is equal to 0, and at this time, the detection head is also degraded to the original specific class detection head. In this way, after the target prior tensor is determined, the original detection head does not need to be modified, and can be used for detecting target objects corresponding to the four types of SOT, VOS, MOT, and MOTS.
[0076] S105: Fuse the target prior tensor with the hierarchical features corresponding to the reference frame image and the target frame image respectively, and perform detection on the target objects in the target video according to the obtained processing result to obtain a detection result.
[0077] In the embodiment, after obtaining the hierarchical features corresponding to the reference frame image and the target frame image respectively through step S102, and obtaining the target prior tensor through step S104, the target prior tensor can be fused with the hierarchical features corresponding to the reference frame image and the target frame image respectively by using an existing or future feature fusion method, such as the processing process shown in "fusion detection" in the following Figure 2 , and according to the obtained processing result, the target objects in the target video are detected to obtain the detection result in the form of "classification", "bounding box", "mask", etc. as shown in Figure 2 , such as the detection results of the four types of SOT, VOS, MOT, and MOTS corresponding to the target objects respectively in Figure 2 .
[0078] Specifically, an optional implementation manner is that the target prior tensor can be fused with the hierarchical features corresponding to the reference frame image and the target frame image respectively by using broadcast addition, and according to the obtained processing result, the target objects in the target video are detected to obtain a detection result.
[0079] In the embodiment, it is assumed that the hierarchical features corresponding to the reference frame image and the target frame image are FPN features F, and F e R h×w×c , and the target prior tensor is defined as P, and P e R h×w×1But since the shapes of the feature maps corresponding to F and P are different, to improve the fusion effect, P can be copied to obtain a vector with c channels to ensure the same format as F, and then the copied P is added to F to obtain the corresponding fusion feature vector, and then the fusion feature vector can be used to detect the target object in the target video to obtain a detection result.
[0080] It should be noted that the application does not limit the fusion processing mode of the target prior tensor and the hierarchical features corresponding to the reference frame image and the target frame image. The above fusion mode through broadcast addition is only a preferred implementation, and other fusion processing modes such as "splicing" can also be used.
[0081] In this way, when tracking and detecting the target object in the target video by performing the above steps S101-S105, it is no longer necessary to detect by type, but is applicable to the detection of the target object corresponding to any one or more of the four different types of single target tracking, multi-target tracking, video target segmentation, and multi-target tracking and segmentation, thereby improving the detection effect of the target object in the target video, and by inputting different target prior tensors, the unified prediction head is also used to detect the target object corresponding to the four different types.
[0082] In summary, the video detection method provided by the embodiment first acquires a target video to be detected, and extracts a reference frame image and a target frame image from the target video. Then, the backbone network is used to extract classification features from the reference frame image and the target frame image respectively to obtain hierarchical features corresponding to the reference frame image and the target frame image respectively. Next, the hierarchical features corresponding to the reference frame image and the target frame image are interactively processed to obtain an interactive feature matrix, and the reference frame image is processed to obtain a reference prior tensor. The reference prior tensor and the interactive feature matrix are fused to obtain a target prior tensor. Then, the target prior tensor and the hierarchical features corresponding to the reference frame image and the target frame image are fused, and the target object in the target video is detected according to the obtained processing result to obtain a detection result. It can be seen that since the embodiment does not detect the target object by type, but uses a unified detection method to simultaneously detect the target object corresponding to any one or more of the four different types of single target tracking, multi-target tracking, video target segmentation, and multi-target tracking and segmentation, the video detection effect is improved.
[0083] Second embodiment
[0084] The embodiment will introduce a video detection device, and the related content can be referred to the above method embodiment.
[0085] Referring toFigure 3 A schematic diagram of a video detection device according to an embodiment is provided in FIG. 3. The device 300 includes:
[0086] An acquisition unit 301 configured to acquire a target video to be detected, and extract a reference frame image and a target frame image from the target video;
[0087] An extraction unit 302 configured to perform classification feature extraction on the reference frame image and the target frame image respectively using a backbone network, to obtain respective hierarchical features corresponding to the reference frame image and the target frame image;
[0088] An interaction unit 303 configured to perform interaction processing on the respective hierarchical features corresponding to the reference frame image and the target frame image, to obtain an interaction feature matrix;
[0089] A fusion unit 304 configured to perform processing on the reference frame image to obtain a reference prior tensor, and perform fusion processing on the reference prior tensor and the interaction feature matrix to obtain a target prior tensor;
[0090] A detection unit 305 configured to perform fusion processing on the target prior tensor and the respective hierarchical features corresponding to the reference frame image and the target frame image, and perform detection on a target object in the target video according to a processing result obtained by the fusion processing, to obtain a detection result.
[0091] In an implementation form of the embodiment, the extraction unit 302 is specifically configured to:
[0092] perform classification feature extraction on the reference frame image and the target frame image respectively using a weight-shared backbone network, to obtain respective hierarchical features corresponding to the reference frame image and the target frame image.
[0093] In an implementation form of the embodiment, the hierarchical features are feature pyramid network (FPN) features.
[0094] In an implementation form of the embodiment, the interaction unit 303 is specifically configured to:
[0095] perform interaction processing on the respective hierarchical features corresponding to the reference frame image and the target frame image using a modified neural network Transformer model, to obtain an interaction feature matrix.
[0096] The modified Transformer model is obtained by replacing a full attention layer in a structure of the Transformer model with a deformable attention layer.
[0097] In an implementation form of the embodiment, the fusion unit 304 is specifically configured to:
[0098] The reference frame image is binarized to obtain a reference prior tensor.
[0099] In an implementation form of the embodiment, the fusion unit 304 is specifically further configured to:
[0100] The reference prior tensor and the interaction feature matrix are multiplied to obtain a target prior tensor.
[0101] In an implementation form of the embodiment, the detection unit 305 is specifically configured to:
[0102] The target prior tensor and the respective hierarchical features of the reference frame image and the target frame image are fused by addition in a broadcast form, and a target object in the target video is detected according to the obtained processing result to obtain a detection result.
[0103] Further, the embodiment of the present application further provides a video detection device, comprising: a processor, a memory, a system bus;
[0104] The processor and the memory are connected through the system bus;
[0105] The memory is configured to store one or more programs, the one or more programs comprising instructions, which, when executed by the processor, cause the processor to perform any of the implementation methods of the above video detection method.
[0106] Further, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores instructions, when the instructions run on a terminal device, the terminal device executes any of the implementation methods of the above video detection method.
[0107] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software and necessary universal hardware platforms. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) execute the methods described in the various embodiments or some parts of the embodiments.
[0108] It should be noted that the various embodiments described herein are described in progressive order of complexity where appropriate. Therefore, features and concepts described in one or more embodiments should be understood to be applicable to other embodiments as well, unless explicitly described otherwise.
[0109] Still yet, any references to claims, items, elements or components of the exemplary embodiments are intended to be illustrative of the broader aspects of the application and can be applied to any novel variations of the present application and can not be limited to a specific embodiment described herein. In addition, the phrase "means for" can be interpreted to mean "means plus function" or "structure plus function," where the structure performs the recited function, whether explicitly identified or not. In addition, the phrase "means for" should not be construed as a disclaimer of additional claim elements that are recited in the claims.
[0110] The above description of disclosed embodiments provides information sufficient to understand how to make and use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without the use of the innovative techniques disclosed in this patent. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A video detection method, characterized in that, The method includes: The target video to be detected is acquired, and reference frame images and target frame images are extracted from the target video; The reference frame image and the target frame image are respectively classified and feature extracted using a backbone network to obtain the hierarchical features corresponding to the reference frame image and the target frame image. The hierarchical features corresponding to the reference frame image and the target frame image are interactively processed to obtain an interactive feature matrix. By performing image detection processing on the reference frame image, a reference prior tensor is obtained; and the reference prior tensor and the interaction feature matrix are fused to obtain the target prior tensor. The target prior tensor is fused with the hierarchical features corresponding to the reference frame image and the target frame image, respectively. Based on the processing results, the target object in the target video is detected to obtain the detection results.
2. The method according to claim 1, characterized in that, The process of extracting classification features from the reference frame image and the target frame image using a backbone network to obtain hierarchical features corresponding to each of the reference frame image and the target frame image includes: The reference frame image and the target frame image are respectively classified and feature extracted using a weight-sharing backbone network to obtain the hierarchical features corresponding to the reference frame image and the target frame image.
3. The method according to claim 1 or 2, characterized in that, The hierarchical features are Feature Pyramid Network (FPN) features.
4. The method according to claim 1, characterized in that, The step of interactively processing the hierarchical features corresponding to the reference frame image and the target frame image to obtain an interactive feature matrix includes: Using the modified neural network Transformer model, the hierarchical features corresponding to the reference frame image and the target frame image are interactively processed to obtain the interactive feature matrix. The modified Transformer model is obtained by replacing the full attention layer in the Transformer model structure with a deformable attention layer.
5. The method according to claim 1, characterized in that, The process of processing the reference frame image to obtain the reference prior tensor includes: By binarizing the reference frame image, a reference prior tensor is obtained.
6. The method according to claim 1, characterized in that, The step of fusing the reference prior tensor and the interaction feature matrix to obtain the target prior tensor includes: The target prior tensor is obtained by multiplying the reference prior tensor and the interaction feature matrix.
7. The method according to claim 1, characterized in that, The process of fusing the target prior tensor with the hierarchical features corresponding to the reference frame image and the target frame image, and detecting the target object in the target video based on the processing result, includes: By adding the target prior tensor and the hierarchical features corresponding to the reference frame image and the target frame image in a broadcast manner, the target object in the target video is detected based on the processing result, and the detection result is obtained.
8. A video detection device, characterized in that, The device includes: The acquisition unit is used to acquire the target video to be detected and to extract a reference frame image and a target frame image from the target video; The extraction unit is used to extract classification features from the reference frame image and the target frame image using the backbone network, so as to obtain the hierarchical features corresponding to the reference frame image and the target frame image respectively. An interaction unit is used to perform interactive processing on the hierarchical features corresponding to the reference frame image and the target frame image respectively to obtain an interaction feature matrix. The fusion unit is configured to obtain a reference prior tensor by performing image detection processing on the reference frame image; and to fuse the reference prior tensor and the interaction feature matrix to obtain a target prior tensor. The detection unit is used to fuse the target prior tensor with the hierarchical features corresponding to the reference frame image and the target frame image, and to detect the target object in the target video based on the processing result, thereby obtaining the detection result.
9. A video inspection device, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1-7.