Video target detection method and system based on time domain ROI feature fusion

Abstract

The application discloses a video target detection method and system based on time domain ROI feature fusion, and solves the problem of insufficient mining of time sequence information in video target detection. The technical scheme is as follows: step 1: for a target frame, two frames of images in the same video are randomly selected as reference frames; step 2: ResNet-101 is used as a backbone network to extract features of each frame of image in the video; step 3: a region generation network is input to generate a candidate frame; step 4: the candidate frame is projected onto a feature map to generate an ROI feature; step 5: the most similar ROI feature of the reference frame is obtained according to the ROI feature of the target frame; step 6: the time domain ROI feature of the target frame is generated according to the most similar ROI feature of the reference frame and the ROI feature of the target frame; and step 7: the time domain ROI feature of the target frame and the ROI feature of the reference frame are input into a detection head to obtain a video target detection result. The application can improve the accuracy of video target detection in a complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video target detection, and particularly relates to a video target detection method and system based on time domain ROI feature fusion. BACKGROUND

[0002] Image target detection has been very mature, while video target detection is still in the development stage. As a basic task in the field of computer vision, video target detection is increasingly widely used in life and has been applied in many scenes such as unmanned driving technology, intelligent video monitoring, robot navigation and the like. Different from image target detection, video target detection often has many problems such as motion blur, defocus blur, object occlusion, light change, rare posture and the like, which bring challenges to video target detection.

[0003] Traditional target detection methods mainly use histogram of oriented gradients (HOG), scale invariant feature transform (SIFT), frame difference method (FD) and optical flow method to detect targets in videos, but the traditional algorithms cannot meet the accuracy requirements of video data analysis. With the development of deep learning, video target detection based on deep learning has become the mainstream method at present and has made significant progress in the past few years. Video target detection based on deep learning mainly includes two types of methods, the first type of method is based on single frame image, which regards each frame in the video as an independent image, and performs image target detection frame by frame, and then performs post-processing on the image detection result. The video target detection method based on single frame image fails to fully exploit the time sequence information added by video relative to image, resulting in unstable detection results, prone to target loss, detection box size jump and the like, and is difficult to be applied to complex scenes such as motion blur and object occlusion. The second type of method is based on time domain information, which aggregates the motion and time context information of other frames in the same video to the current frame to fuse useful features in the video sequence, and from the aspects of selection of other frames, selection of feature fusion level and selection of feature fusion mode, the time domain information is exploited to relieve the problem of target missing and wrong detection in complex situations and improve the accuracy of video object detection algorithm.

[0004] How to fully utilize the time sequence information contained in the video sequence has become one of the important research directions in the field of video target detection, and the design of similar semantic feature extraction and fusion mode in the sequence is crucial to the accuracy of video target detection, and inefficient design often leads to unstable detection results in complex scenes. SUMMARY

[0005] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0006] The purpose of this invention is to solve the above-mentioned problems and provide a video target detection method and system based on temporal ROI feature fusion, which solves the problem of insufficient mining of temporal information in video target detection.

[0007] The technical solution of this invention is as follows: This invention discloses a video target detection method based on temporal ROI feature fusion, the method comprising:

[0008] Step 1: For the target frame, randomly select two frames from the same video as reference frames;

[0009] Step 2: Use a residual network as the backbone network to extract features from each frame of the video to obtain feature maps for each frame, including target frame feature maps and reference frame feature maps;

[0010] Step 3: Use the feature map obtained in Step 2 as input to the region generation network to obtain candidate boxes for each frame of the image;

[0011] Step 4: Project the candidate bounding boxes of each frame in Step 3 onto the feature map extracted in Step 2 to obtain the corresponding ROI features, including the target frame ROI features and the reference frame ROI features.

[0012] Step 5: Calculate the similarity between each ROI feature map of the target frame and the feature map of the reference frame using cosine similarity, and use similarity as a weight metric to weight the part of the reference frame that is most similar to the current ROI feature to obtain the most similar ROI feature of the reference frame.

[0013] Step 6: Use the most similar ROI features of all reference frames obtained in Step 5 and the target frame ROI features in Step 4 as input for temporal ROI feature fusion processing to obtain the temporal ROI features of the target frame.

[0014] Step 7: Use the temporal ROI features of the target frame in Step 6 and the ROI features of the reference frame in Step 4 as input to the detection head to correct the position of the target frame detection box and classify the targets contained in the detection box to obtain the video target detection result.

[0015] According to an embodiment of the video target detection method based on the time domain ROI feature fusion provided in the application, the residual network used in step 1 is a ResNet-101 network, wherein the ResNet-101 network reduces the stride of the first convolutional block of the convolutional layer conv5 stage to improve the resolution of the feature map, and meanwhile, increases the expansion rate of the convolutional layer to make up for the problem of reduced receptive field caused by the stride reduction.

[0016] According to an embodiment of the video target detection method based on the time domain ROI feature fusion provided in the application, step 5 further includes:

[0017] Step 5-1: calculating the cosine similarity of the feature point with all feature points on the feature map of the current reference frame;

[0018] Step 5-2: selecting two points on the feature map of the reference frame with the highest similarity to the feature point, and obtaining the features of the two points;

[0019] Step 5-3: performing the softmax operation on the cosine similarity corresponding to the two points to convert the cosine similarity into weights;

[0020] Step 5-4: weighting the features of the two points with the highest similarity with the weights generated by the cosine similarity to obtain the features extracted from the feature map of other frames, i.e., the most similar ROI features of the current feature point;

[0021] Each feature point on the ROI feature map of the target frame and each reference frame repeats the above steps.

[0022] According to an embodiment of the video target detection method based on the time domain ROI feature fusion provided in the application, step 6 further includes:

[0023] Step 6-1: performing the splicing operation on the ROI features of the target frame and the most similar ROI features of the reference frame respectively, and inputting the three-layer convolutional network;

[0024] Step 6-2: splicing the output results of step 6-1, and performing the softmax operation along the number dimension of the ROI features to convert the cosine similarity into weights;

[0025] Step 6-3: weighting the output results of step 6-1 with the weights generated in step 6-2 to obtain the time domain ROI features of the target frame containing the beneficial information of the reference frame.

[0026] According to an embodiment of the video target detection method based on the time domain ROI feature fusion provided in the application, the detection head of the Faster RCNN is used in step 7, and a similar semantic aggregation module is added after each of the two fully connected layers in the detection head, and the processing of the similar semantic aggregation module further includes:

[0027] Step 7-1: Calculate the weight between the target frame feature and the reference frame feature using the multi-head attention mechanism.

[0028] Step 7-2: Perform a softmax operation on the obtained weight along the dimension of the reference frame candidate box number.

[0029] Step 7-3: Weighted sum the reference frame features using the normalized weight.

[0030] The application also discloses a video target detection system based on time domain ROI feature fusion, which comprises:

[0031] A reference frame selection module configured to randomly select two frames of images in the same video as reference frames for the target frame;

[0032] A feature map acquisition module configured to extract the features of each frame of image in the video using a residual network as the backbone network to obtain the feature map of each frame of image, including the target frame feature map and the reference frame feature map;

[0033] A candidate box acquisition module configured to take the feature map obtained in the feature map acquisition module as the input of the region generation network to obtain the candidate box of each frame of image;

[0034] An ROI feature acquisition module configured to project the candidate box of each frame of image in the candidate box acquisition module onto the feature map extracted in step 1 to obtain the corresponding ROI feature, including the target frame ROI feature and the reference frame ROI feature;

[0035] A most similar ROI feature acquisition module configured to calculate the similarity between each ROI feature map of the target frame and the reference frame feature map using the cosine similarity, and to weight the most similar part in the reference frame to the current ROI feature using the similarity as the weight measure to obtain the most similar ROI feature of the reference frame;

[0036] A time domain ROI feature fusion module configured to take all the most similar ROI features of the reference frame obtained in the most similar ROI feature acquisition module of the reference frame and the target frame ROI feature in the ROI feature acquisition module as the input of the time domain ROI feature fusion processing to obtain the time domain ROI feature of the target frame;

[0037] A video target detection module configured to take the time domain ROI feature of the target frame in the time domain ROI feature fusion module and the reference frame ROI feature in the ROI feature acquisition module as the input of the detection head to realize the correction of the target frame detection box position and the classification of the target contained in the detection box to obtain the video target detection result.

[0038] According to an embodiment of the video target detection system based on the time domain ROI feature fusion provided in the application, the residual network used in the feature map acquisition module is a ResNet-101 network, wherein the ResNet-101 network reduces the stride of the first convolutional block in the convolutional layer conv5 stage to improve the resolution of the feature map, and meanwhile, increases the expansion rate of the convolutional layer to make up for the problem of reduced receptive field caused by the stride reduction.

[0039] According to an embodiment of the video target detection system based on the time domain ROI feature fusion provided in the application, the most similar ROI feature acquisition module is further configured to perform the following processing:

[0040] Calculate the cosine similarity between the feature point and all feature points on the current reference frame feature map;

[0041] Select the two points on the reference frame feature map that have the highest similarity with the feature point, and obtain the features of the two points;

[0042] Perform the softmax operation on the cosine similarities corresponding to the two points to convert them into weights;

[0043] Weight the features of the two points with the highest similarity with the weights generated by the cosine similarity to obtain the feature of the feature point extracted from the feature map of other frames, i.e., the most similar ROI feature of the current feature point;

[0044] Repeat the above steps for each feature point on the target frame ROI feature map and each reference frame.

[0045] According to an embodiment of the video target detection system based on the time domain ROI feature fusion provided in the application, the time domain ROI feature fusion module is further configured to perform the following processing:

[0046] Perform the splicing operation on the target frame ROI feature and its own and the most similar ROI features of the reference frames respectively, and input the three-layer convolutional network;

[0047] Splice the output results of the convolutional network, and perform the softmax operation along the number dimension of the ROI feature to convert them into weights;

[0048] Weight the output results of the convolutional network with the generated weights to obtain the target frame time domain ROI feature containing the beneficial information of the reference frames.

[0049] According to an embodiment of the video target detection system based on the time domain ROI feature fusion provided in the application, the detection head of Faster RCNN is used in the video target detection module, and a similar semantic aggregation module is added after each of the two fully connected layers in the detection head, and the processing of the similar semantic aggregation module further includes:

[0050] The multi-head attention mechanism is used to calculate the weight between the target frame feature and the reference frame feature.

[0051] The obtained weight is subjected to a softmax operation along the dimension of the number of reference frame candidate boxes.

[0052] The reference frame features are weighted and summed using the normalized weight.

[0053] The present application has the following beneficial effects compared with the prior art: first, in the process of extracting ROI features, the video target detection method based on time domain ROI feature fusion of the present application combines the semantic information contained in other frames by weighting and summing the most similar part of the reference frame feature map to the ROI feature of the target frame according to the semantic similarity between the ROI feature of the target frame and the feature map of the reference frame extracted by the backbone network. Second, in the process of time domain ROI feature fusion, the video target detection method based on time domain ROI feature fusion of the present application takes the most similar ROI feature of all reference frames and the ROI feature of the target frame as input, extracts the similarity information therefrom by using three convolutional layers, and fuses the ROI features of the target frame and the reference frame by taking the similarity information as weight, so as to fully utilize the features of the target frame and the reference frame. When the target frame has problems such as blur or occlusion, the model can obtain useful semantic features from other frames to assist the detection of the target frame. Third, the video target detection method based on time domain ROI feature fusion of the present application further mines the part of the reference frame feature that is beneficial to the detection of the target frame by using the multi-head attention mechanism in the detection head part, so that the useful features are more aggregated, the detection accuracy of the video target detection model in a complex scene is improved, and more stable detection results are obtained. BRIEF DESCRIPTION OF DRAWINGS

[0054] The above features and advantages of the present application can be better understood by reading the following detailed description of embodiments of the present application in conjunction with the attached drawings. In the drawings, components are not necessarily drawn to scale, and components having similar or related properties or features can have the same or similar reference numbers.

[0055] Figure 1 An overall framework schematic diagram of an embodiment of the video target detection method based on time domain ROI feature fusion of the present application is shown.

[0056] Figure 2 An overall framework schematic diagram of an embodiment of the video target detection method based on time domain ROI feature fusion of the present application is shown. Figure 1 An overall framework schematic diagram of an embodiment of the video target detection method based on time domain ROI feature fusion of the present application is shown.

[0057] Figure 3 An overall framework schematic diagram of an embodiment of the video target detection method based on time domain ROI feature fusion of the present application is shown. Figure 1 An overall framework schematic diagram of an embodiment of the video target detection method based on time domain ROI feature fusion of the present application is shown.

[0058] Figure 4 A schematic diagram of an embodiment of a video object detection system based on time domain ROI feature fusion of the present application is shown. DETAILED DESCRIPTION

[0059] The present application is described in detail below with reference to the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be understood as limiting the scope of protection of the present application in any way.

[0060] Figure 1 A schematic diagram of an embodiment of a video object detection system based on time domain ROI feature fusion of the present application is shown. Figure 1 The implementation steps of the method of the present embodiment are described in detail as follows.

[0061] Step 1: For the target frame, randomly select two frames of images within the same video as reference frames, rather than using adjacent frames as reference frames.

[0062] Since the information redundancy degree of adjacent frames is generally high, and appearance degradation phenomena such as motion blur, object occlusion, and illumination change usually last for a period of time, using adjacent frames as reference frames will weaken the advantage of feature aggregation and reduce the effectiveness of the time domain information-based method, therefore, random frames are used instead of adjacent frames to provide available semantic information.

[0063] Step 2: Use a residual network (e.g. ResNet-101) as a backbone network to extract features of each frame of images in the video, to obtain feature maps of each frame of images, including target frame feature maps and reference frame feature maps.

[0064] Among them, the ResNet-101 network modifies the stride of the first convolutional block in the conv5 stage from 2 to 1 to improve the resolution of the feature maps, and at the same time, to make up for the problem of reduced receptive field caused by stride reduction, the dilation rate of all 3x3 convolutional layers in this stage is increased.

[0065] Step 3: Use the feature maps obtained in step 1 above (including target frame feature maps and reference frame feature maps) as the input of the region generation network to obtain candidate boxes for each frame of images.

[0066] Among them, the input of the region generation network is the output of the conv4 of the backbone network, and 12 sizes of anchor boxes are set, including 4 scales {64^2, 128^2, 256^2, 512^2} and 3 aspect ratios {0.5, 1.0, 2.0}, and 300 candidate boxes are generated on each image.

[0067] Step 4: Project the candidate boxes of each image in step 3 onto the feature maps extracted in step 2 (including the target frame feature map and the reference frame feature map) to obtain the corresponding ROI (Region of Interest) features (including the corresponding target frame ROI features and the reference frame ROI features).

[0068] In the above, the length and width of the ROI features are both set to 7, i.e., the size of the feature map is 49.

[0069] Step 5: Calculate the similarity between each ROI feature map of the target frame and the reference frame feature map using the cosine similarity, and use the similarity as a weight measure to weight the most similar part of the reference frame to the current ROI feature, to obtain the most similar ROI feature of the reference frame.

[0070] The detailed processing of step 5 is shown in Figure 2 , which is illustrated as follows by taking a feature point on a ROI feature map of the target frame and a reference frame feature map as an example.

[0071] Step 5-1: Calculate the cosine similarity between the feature point and all feature points on the current reference frame feature map.

[0072] Step 5-2: Select the two points on the reference frame feature map that have the highest similarity to the feature point, and obtain the features of the two points.

[0073] Step 5-3: Perform a softmax operation on the cosine similarity corresponding to the two points to convert it into a weight.

[0074] Step 5-4: Weight the features of the two points with the highest similarity using the weight generated by the cosine similarity to obtain the feature extracted from the feature map of the other frame, i.e., the most similar ROI feature of the current feature point.

[0075] Similarly, each feature point on the ROI feature map of the target frame and each reference frame repeats the above steps.

[0076] Step 6: Take the most similar ROI features of all reference frames obtained in step 5 and the ROI features of the target frame in step 4 as inputs for the time domain ROI feature fusion processing to obtain the time domain ROI features of the target frame.

[0077] The details of the time domain ROI feature fusion processing are shown in Figure 3 , and the implementation is as follows.

[0078] Step 6-1: Perform a concatenation operation on the target frame ROI feature and its own and the most similar ROI features of the reference frame, and input them into a three-layer convolutional network.

[0079] Specifically, the structure of the three-layer convolutional network comprises: a convolutional layer with a convolution kernel size of 1*1 and a number of convolution kernels equal to the number of input channels, a convolutional layer with a convolution kernel size of 3*3 and a number of convolution kernels equal to the number of input channels, and a convolutional layer with a convolution kernel size of 1*1 and a number of convolution kernels equal to half of the number of input channels, which is used to fuse the two types of features and keep the size of the output features consistent with the size of the input target frame ROI feature.

[0080] Step 6-2: The output results of step 6-1 are spliced, and a softmax operation is performed along the number dimension of the ROI feature, to obtain the similarity between the target frame ROI feature and itself and other reference frame ROI features as the weight of the input feature.

[0081] Step 6-3: The output results of step 6-1 are weighted by the weight generated in step 6-2 to obtain the target frame temporal ROI feature containing the beneficial information of the reference frame.

[0082] Step 7: The temporal ROI feature of the target frame in step 6 and the ROI feature of the reference frame in step 4 are used as the input of the detection head to correct the position of the detection frame of the target frame and classify the target contained in the detection frame to obtain the video target detection result.

[0083] The detection head of Faster RCNN is used, and a similar semantic aggregation module is added after each of the two fully connected layers in the detection head, and the similar semantic aggregation module is implemented in the following manner:

[0084] Step 7-1: The multi-head attention mechanism is used to calculate the weight between the target frame feature and the reference frame feature.

[0085] Step 7-2: The obtained weight is subjected to a softmax operation along the dimension of the number of reference frame candidate frames.

[0086] Step 7-3: The reference frame features are weighted and summed using the normalized weight.

[0087] Figure 4 The principle of an embodiment of the video target detection system based on temporal ROI feature fusion of the application is shown.

[0088] Please refer to Figure 4 The system of the embodiment comprises a reference frame selection module, a feature map acquisition module, a candidate frame acquisition module, an ROI feature acquisition module, a most similar ROI feature acquisition module, a temporal ROI feature fusion module, and a video target detection module.

[0089] The reference frame selection module is configured to randomly select two frames of images in the same video as reference frames for the target frame.

[0090] Since the information redundancy of adjacent frames is generally high, and appearance degradation phenomena such as motion blur, object occlusion, and illumination change usually last for a period of time, using adjacent frames as reference frames will weaken the advantage of feature aggregation and reduce the effectiveness of the time domain information-based method, so random frames are used instead of adjacent frames to provide available semantic information.

[0091] The feature map acquisition module is configured to extract features of each image in the video using a residual network as a backbone network to obtain a feature map of each image, including a target frame feature map and a reference frame feature map.

[0092] The residual network used in the feature map acquisition module is a ResNet-101 network. The stride of the first convolutional block in the conv5 stage of the ResNet-101 network is modified from 2 to 1 to improve the resolution of the feature map, and to compensate for the reduction of the receptive field caused by the reduction of the stride, the dilation rate of all 3x3 convolutional layers in this stage is increased.

[0093] The candidate box acquisition module is configured to use the feature map obtained in the feature map acquisition module as the input of the region generation network to obtain the candidate box of each image.

[0094] The input of the region generation network is the output of the conv4 of the backbone network, and 12 sizes of anchor boxes are set, including 4 scales {642, 1282, 2562, 5122} and 3 aspect ratios {0.5, 1.0, 2.0}, and 300 candidate boxes are generated on each image.

[0095] The ROI feature acquisition module is configured to project the candidate box of each image in the candidate box acquisition module onto the feature map extracted in step 1 to obtain the corresponding ROI feature, including the target frame ROI feature and the reference frame ROI feature. The length and width of the ROI feature are both set to 7, i.e., the size of the feature map is 49.

[0096] The most similar ROI feature acquisition module is configured to calculate the similarity between each ROI feature map of the target frame and the reference frame feature map using the cosine similarity, and to weight the most similar part in the reference frame to the current ROI feature using the similarity as the weight measure to obtain the most similar ROI feature of the reference frame.

[0097] The most similar ROI feature acquisition module is further configured to perform the following processing:

[0098] Calculate the cosine similarity between the feature point and all feature points on the current reference frame feature map;

[0099] Select the two points on the reference frame feature map that have the highest similarity to the feature point to obtain the features of the two points;

[0100] The cosine similarity corresponding to the two points is subjected to a softmax operation to convert into weights;

[0101] The weights generated by the cosine similarity are used to weight the features of the two points with the highest similarity, to obtain the features of the current feature point extracted from the feature map of other frames, i.e., the most similar ROI features of the current feature point;

[0102] Each feature point on the ROI feature map of the target frame and each reference frame repeats the above steps.

[0103] As shown in the specific Figure 2 The foregoing method embodiments have been described in detail, and will not be repeated here.

[0104] The time domain ROI feature fusion module is configured to take the most similar ROI features of all reference frames obtained by the most similar ROI feature acquisition module and the ROI features of the target frame in the ROI feature acquisition module as inputs of time domain ROI feature fusion processing, to obtain the time domain ROI features of the target frame;

[0105] The time domain ROI feature fusion module is further configured to perform the following processing:

[0106] The target frame ROI features are respectively concatenated with their own and the most similar ROI features of the reference frames, and input into a three-layer convolutional network.

[0107] The output results of the convolutional network are concatenated, and a softmax operation is performed along the number dimension of the ROI features to convert into weights.

[0108] The output results of the convolutional network are weighted by the generated weights to obtain the target frame time domain ROI features containing useful information of the reference frames.

[0109] As shown in the specific Figure 3 The foregoing method embodiments have been described in detail, and will not be repeated here.

[0110] The video target detection module is configured to take the time domain ROI features of the target frame in the time domain ROI feature fusion module and the ROI features of the reference frames in the ROI feature acquisition module as inputs of the detection head, to realize correction of the detection frame position of the target frame, and to classify the target contained in the detection frame to obtain a video target detection result.

[0111] The Faster RCNN detection head is used in the video target detection module, and a similar semantic aggregation module is added after each of the two fully connected layers in the detection head. The processing of the similar semantic aggregation module further includes:

[0112] The multi-head attention mechanism is used to calculate the weights between the features of the target frame and the features of the reference frames.

[0113] The resulting weights are subjected to a softmax operation along the dimension of the number of reference frame candidate boxes.

[0114] The reference frame features are weighted and summed using the normalized weights.

[0115] Although the above-described methods are illustrated and described as a series of acts, it will be appreciated that the methods are not limited by the order of acts, as some acts can, in accordance with one or more embodiments, occur simultaneously or in different orders with respect to other acts discussed herein or with respect to acts not discussed but will be apparent to those having ordinary skill in the art.

[0116] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0117] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0118] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0119] In one or more exemplary embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0120] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video object detection method based on time domain ROI feature fusion, characterized in that, The method comprises: Step 1: randomly selecting two frames of images in the same video as reference frames for a target frame; Step 2: extracting features of each frame of images in the video by using a residual network as a backbone network to obtain feature maps of each frame of images, including a target frame feature map and a reference frame feature map; Step 3: taking the feature maps obtained in step 2 as inputs of a region generation network to obtain candidate boxes of each frame of images; Step 4: projecting the candidate boxes of each frame of images in step 3 onto the feature maps extracted in step 2 to obtain corresponding ROI features, including target frame ROI features and reference frame ROI features; Step 5: calculating the similarity between each ROI feature map of the target frame and the reference frame feature map by using cosine similarity, and weighting the most similar part in the reference frame to the current ROI feature as a weight measure to obtain the most similar ROI feature of the reference frame; Step 6: taking the most similar ROI features of all reference frames obtained in step 5 and the target frame ROI features in step 4 as inputs of a time domain ROI feature fusion process to obtain time domain ROI features of the target frame; Step 7: taking the time domain ROI features of the target frame in step 6 and the reference frame ROI features in step 4 as inputs of a detection head to correct the position of the detection box of the target frame and classify the target contained in the detection box to obtain a video target detection result. 2.The video object detection method based on time domain ROI feature fusion according to claim 1, characterized in that, The residual network used in step 1 is a ResNet-101 network, wherein the ResNet-101 network reduces the stride of the first convolutional block in the convolutional layer conv5 stage to improve the resolution of the feature map, and increases the expansion rate of the convolutional layer to make up for the problem of reduced receptive field caused by the stride reduction. 3.The video object detection method based on time domain ROI feature fusion according to claim 1, characterized in that, Step 5 further comprises: Step 5-1: calculating the cosine similarity between the feature points on the ROI feature map and all feature points on the current reference frame feature map; Step 5-2: selecting the two points with the highest similarity to the feature point on the reference frame feature map to obtain the features of the two points; Step 5-3: performing a softmax operation on the cosine similarity corresponding to the two points to convert it into a weight; Step 5-4: weighting the features of the two points with the highest similarity by the weight generated by the cosine similarity to obtain the feature extracted from the feature map of the other frame, i.e. the most similar ROI feature of the current feature point; Each feature point on the target frame ROI feature map and each reference frame repeats the above steps. 4.The video object detection method based on time domain ROI feature fusion according to claim 1, characterized in that, Step 6 further comprises: Step 6-1: performing a splicing operation on the target frame ROI feature and its own and the most similar ROI feature of the reference frame, and inputting a three-layer convolutional network; Step 6-2: splicing the output results of step 6-1, and performing a softmax operation along the number dimension of the ROI feature to convert it into a weight; Step 6-3: weighting the output results of step 6-1 by the weight generated in step 6-2 to obtain the target frame time domain ROI feature containing useful information of the reference frame.

5. The video object detection method based on time domain ROI feature fusion according to claim 1, characterized in that, In step 7, the detection head of Faster RCNN is used, and a similar semantic aggregation module is added after each of the two fully connected layers in the detection head. The processing of the similar semantic aggregation module further includes: Step 7-1: using a multi-head attention mechanism to calculate the weight between the target frame feature and the reference frame feature; Step 7-2: performing a softmax operation on the obtained weight along the dimension of the number of reference frame candidate boxes; Step 7-3: using the normalized weight to perform weighted summation on the reference frame feature. 6.A video object detection system based on time domain ROI feature fusion, characterized in that, The system comprises: a reference frame selection module configured to randomly select two frames of images in the same video as reference frames for a target frame; a feature map acquisition module configured to extract features of each frame of image in the video using a residual network as a backbone network to obtain a feature map of each frame of image, including a target frame feature map and a reference frame feature map; a candidate box acquisition module configured to take the feature map obtained in the feature map acquisition module as an input of a region generation network to obtain a candidate box of each frame of image; an ROI feature acquisition module configured to project the candidate box of each frame of image in the candidate box acquisition module onto the feature map extracted in step 1 to obtain corresponding ROI features, including a target frame ROI feature and a reference frame ROI feature; a most similar ROI feature acquisition module configured to calculate the similarity between each ROI feature map of the target frame and the reference frame feature map using a cosine similarity, and to weight the most similar part in the reference frame to the current ROI feature using the similarity as a weight measure to obtain the most similar ROI feature of the reference frame; a temporal ROI feature fusion module configured to take all the most similar ROI features of the reference frame obtained in the reference frame most similar ROI feature acquisition module and the target frame ROI feature in the ROI feature acquisition module as inputs of temporal ROI feature fusion processing to obtain the temporal ROI feature of the target frame; a video target detection module configured to take the temporal ROI feature of the target frame in the temporal ROI feature fusion module and the reference frame ROI feature in the ROI feature acquisition module as inputs of a detection head to correct the position of the detection frame of the target frame, classify the target contained in the detection frame, and obtain a video target detection result.

7. The video object detection system based on time domain ROI feature fusion of claim 6, wherein, The residual network used in the feature map acquisition module is a ResNet-101 network, wherein the ResNet-101 network reduces the stride of the first convolutional block in the convolutional layer conv5 stage to improve the resolution of the feature map, and increases the dilation rate of the convolutional layer to make up for the problem of reduced receptive field caused by the stride reduction. 8.The video object detection system based on time domain ROI feature fusion of claim 6, wherein, The most similar ROI feature acquisition module is further configured to perform the following processing: calculating the cosine similarity between the feature points on the ROI feature map and all feature points on the current reference frame feature map; selecting the two points with the highest similarity to the feature points on the reference frame feature map to obtain the features of the two points; performing a softmax operation on the cosine similarity corresponding to the two points to convert it into a weight; The weight generated by the cosine similarity is used to weight the features of the two points with the highest similarity, to obtain the features of the feature point extracted from the feature map of other frames, i.e., the most similar ROI feature of the current feature point; Each feature point on the target frame ROI feature map and each reference frame repeats the above steps. 9.The video object detection system based on time domain ROI feature fusion of claim 6, wherein, The time domain ROI feature fusion module is further configured to perform the following processing: The target frame ROI feature and its own and the most similar ROI feature of the reference frame are respectively subjected to a splicing operation, and input into a three-layer convolutional network; The output result of the convolutional network is spliced, and a softmax operation is performed along the number dimension of the ROI feature to convert it into a weight; The output result of the convolutional network is weighted by the generated weight to obtain the target frame time domain ROI feature containing the beneficial information of the reference frame.

10. The video object detection system based on time domain ROI feature fusion of claim 6, wherein, The detection head of Faster RCNN is used in the video target detection module, and a similar semantic aggregation module is added after each of the two fully connected layers in the detection head. The processing of the similar semantic aggregation module further includes: The multi-head attention mechanism is used to calculate the weight between the target frame feature and the reference frame feature; The obtained weight is subjected to a softmax operation along the dimension of the number of reference frame candidate boxes; The reference frame features are weighted and summed using the normalized weight.

Images