A mission target detection method and system based on fragmented video information
Through the fuzzy prior method and feature extraction technology, the weight of the video frame is calculated, which solves the problem of neglecting the impact of fuzzy in the prior art, and improves the accuracy and reliability of video object detection.
Patent Information
- Application Number
- CN202210375278.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-11
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-04-11
AI Technical Summary
The existing video object detection method based on feature aggregation lacks special considerations on the impact of blur when processing fragmented video information, resulting in a decrease in detection accuracy.
The fuzzy prior method was introduced, and the weight of each frame of image was calculated through the fuzzy mapping network and the significance detection network. The deep convolutional neural network and optical flow information module were used to extract features, and combined with the bilinear distortion function and the detection neural network were used to perform the target character's Test.
It improves the accuracy and reliability of character detection, especially when video discontinuity, and improves the accuracy of video object detection.
Smart Images

Figure CN114882523B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a task target detection method based on fragmented video information, and also relates to a system for realizing the task target detection method based on fragmented video information. Background Art
[0002] Deep learning networks have made significant progress in object detection. In recent years, excellent image-based object detection algorithms have been directly transferred to video object detection. Compared with static image object detection, video object detection is more challenging. Since video detection scenes are usually more complex, such as the lack of cooperation of the detected person and the discontinuous video information obtained, the extracted image information is incomplete, such as motion blur, defocus, and rare postures, which greatly reduces the detection accuracy.
[0003] Existing feature aggregation-based methods compensate for the misalignment between frames by aggregating features from multiple adjacent frames. A key question is whether these frames should be treated equally. There are currently two approaches to this problem. One solution is to treat each frame equally and give them the same weight, and the other is to use a light network to learn weights during training. Both solutions lack special consideration for the impact of blur.
[0004] In this invention, we propose a method for detecting human targets based on fragmented video information. The weight distribution of aggregated frames is improved by using blur priors. In particular, a blur mapping network is introduced to mark each pixel as blurry or non-blurry. Since the invention only cares about the blur degree of the target without considering the background, a saliency detection network is used to focus on the target, and the saliency map is used for calibration to obtain a calibrated blur map with the target blur as the focus, and the weight of each frame is calculated; this method has better performance than the most advanced video target detection algorithm at present, while increasing the amount of calculation. Summary of the invention
[0005] The purpose of the present invention is to provide a task target detection method and system based on fragmented video information, which improves the weight distribution of aggregated frames through the fuzzy prior method, calculates the weight of each frame image instead of assigning the same weight to each frame, and effectively improves the accuracy and reliability of human detection.
[0006] To achieve the above functions, a task target detection method based on fragmented video information executes steps S1 to S7 according to a preset period to obtain a target person detection model, and then applies the target person detection model to complete the detection of the target person;
[0007] S1. Real-time acquisition of a video containing a target person walking, converting the video containing the target person walking into a time-sequentially arranged video frame sequence, extracting a preset number of consecutive video frames at a preset position in the video frame sequence as a valid frame sequence, taking the video frame sequence as input and the valid frame sequence as output, and constructing a valid frame sequence extraction module;
[0008] S2. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the deep convolutional neural network, taking each deep convolutional feature map corresponding to each video frame in the valid frame sequence as output, construct a deep convolutional feature map extraction module;
[0009] S3. Taking the effective frame sequence output by the effective frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, calculate the optical flow parameters of each group of video frame pairs as the motion information of the target person, and take the optical flow parameters of each group of video frame pairs in the effective frame sequence as output to construct an optical flow information module;
[0010] S4. Taking each deep convolution feature map output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear warping function, taking the deformation features of each group of video frame pairs as output, a deformation feature module is constructed;
[0011] S5. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence, and according to the fuzzy features and saliency features, and based on the softmax classification network, obtain the weight coefficient of each video frame in the valid frame sequence; taking the weight coefficient of each video frame in the valid frame sequence as output, construct a weight coefficient calculation module;
[0012] S6. Taking the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtain the aggregate features corresponding to the video frame group composed of the video frame pairs, and construct a detection network module by detecting the neural network with the preset information of the target person as output;
[0013] S7. Taking the video frame sequence corresponding to the video containing the target person walking in real time as input and the preset information of the target person as output, a target person detection model to be trained is constructed based on the effective frame sequence extraction module, the deep convolution feature map extraction module, the optical flow information module, the deformation feature module and the weight coefficient calculation module. Based on the training of the video samples containing the target person walking, the target person detection model is obtained to complete the detection of the target person.
[0014] As a preferred technical solution of the present invention: Step S3 takes the effective frame sequence output by the effective frame sequence extraction module as input, and based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, calculates the optical flow parameters of each group of video frame pairs as the motion information of the target person, and takes the optical flow parameters of each group of video frame pairs in the effective frame sequence as output. The specific steps of constructing the optical flow information module are as follows:
[0015] S31: define the tth video frame I in the valid frame sequence t is the reference frame, the t-τth frame video frame I in the valid frame sequence t-τ , the t+τth frame video frame I t+τ As the supporting frame, the reference frame I t , support frame I t-τ , support frame I t+τ Input optical flow neural network;
[0016] S32: Optical flow neural network includes convolutional layer, expansion layer, reference frame I t , support frame I t-τ , support frame I t+τ The reference frame I is obtained by the contraction part composed of the convolutional layer of the optical flow neural network t , support frame I t-τ , support frame I t+τ The corresponding feature maps respectively;
[0017] S33: Reference frame I t , support frame I t-τ , support frame I t+τ The corresponding feature maps are passed through the expansion layer of the optical flow neural network to obtain a reference frame I with a size expanded to the original image size. t , support frame I t-τ , support frame I t+τ The corresponding feature maps respectively;
[0018] S34: Based on the reference frame I obtained in step S33 t , support frame I t-τ , support frame I t+τ The corresponding feature maps are used for optical flow prediction, respectively, with reference frame I t , support frame I t-τ As a video frame pair, and the reference frame I t , support frame I t+τ As a video frame pair, obtain the reference frame I t , support frame I t-τ The optical flow parameters M between the corresponding feature maps t-τ→t , and reference frame It , support frame I t+τ The optical flow parameters M between the corresponding feature maps t+τ→t As follows:
[0019] M t-τ→t =FlowNet(I t-τ ,I t )
[0020] M t+τ→t =FlowNet(I t+τ ,I t )
[0021] Where M t-τ→t The reference frame I t , support frame I t-τ The optical flow parameters between the corresponding feature maps, t-τ→t represents the reference frame I t , support frame I t-τ The corresponding relationship, M t+τ→t The reference frame I t , support frame I t+τ The optical flow parameters between the corresponding feature maps, t+τ→t represents the reference frame I t , support frame I t+τ , FlowNet represents the optical flow neural network calculation.
[0022] As a preferred technical solution of the present invention: Step S4 takes the deep convolution feature maps output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, and takes the deformation features of each group of video frame pairs as output based on the bilinear warping function. The specific method of constructing the deformation feature module is as follows:
[0023] f t-τ→t =W(f t-τ ,M t-τ→t )
[0024] f t+τ→t =W(f t+τ ,M t+τ→t )
[0025] In the formula, f t-τ→t The reference frame I t , support frame I t-τ The deformation characteristics between t+τ→t The reference frame I t , support frame I t+τ The deformation characteristics between them, W represents the calculation based on the bilinear distortion function, f t-τ The support frame I is output by the deep convolution feature map extraction module t-τ The corresponding feature map, ft+τ The support frame I is output by the deep convolution feature map extraction module t-τ The corresponding feature map.
[0026] As a preferred technical solution of the present invention: Step S5 takes the valid frame sequence output by the valid frame sequence extraction module as input, obtains the fuzzy features and the saliency features corresponding to each video frame in the valid frame sequence based on the fuzzy mapping network and the saliency detection network, respectively, and obtains the weight coefficient of each video frame in the valid frame sequence based on the fuzzy features and the saliency features and the softmax classification network; takes the weight coefficient of each video frame in the valid frame sequence as output, and constructs the specific steps of the weight coefficient calculation module as follows:
[0027] S51: Inputting each video frame in the valid frame sequence into the fuzzy mapping network and the saliency detection network respectively, and obtaining the fuzzy features and saliency features corresponding to each video frame respectively;
[0028] S52: Obtain the corrected blur map M corresponding to each video frame by multiplying the blur features and saliency features corresponding to each video frame obtained in step S51. blur-sali ;
[0029] S53: Based on the step function with a threshold of 0.5, the corrected blur map M corresponding to each video frame is blur-sali Binarization is performed, where the step function is as follows:
[0030]
[0031] Where m is the corrected blur map M corresponding to each video frame blur-sali The value of u(m) is the corrected blur map M corresponding to each video frame after binarization. blur-sali The value of
[0032] S54: For each video frame, all u(m) obtained in step S53 are added together to obtain the blur parameter Vcb of each video frame, and the blur parameter Vcb of each video frame is standardized. The standardization method is as follows:
[0033]
[0034] Where, Vcb i Indicates the blur parameter of video frame i, VcbNorm i represents the blur parameter of the standardized video frame i, and the value of i is {t-τ, t, t+τ};
[0035] S55: The blur parameter VcbNorm of each video frame obtained in step S54 after the normalization process isi Input to the softmax classification network to obtain the support frame I t-τ , reference frame I t , support frame I t+τ The corresponding weight coefficients ω t-τ ,ω t ,ω t+τ .
[0036] As a preferred technical solution of the present invention: Step S6 takes the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficients of each video frame output by the weight coefficient calculation module as input, obtains the aggregate features corresponding to the video frame group composed of the video frame pairs, and constructs the detection network module by detecting the neural network with the preset information of the target person as output. The specific steps are as follows:
[0037] S61: Deformation feature f based on the output of the shape feature module t-τ→t 、f t+τ→t , the reference frame I output by the deep convolutional feature map extraction module t The corresponding feature map f t , and the weight coefficient ω corresponding to each deformation feature output by the weight coefficient calculation module t-τ ,ω t ,ω t+τ , according to the following formula, the support frame I is obtained t-τ , reference frame I t , support frame I t+τ Aggregate features J of the formed video frame group;
[0038] J = f t-τ→t ω t-τ +f t ω t +f t+τ→t ω t+τ
[0039] S62: Input the aggregated features into the detection neural network to obtain preset information of the target person.
[0040] The present invention also designs a task target detection system based on fragmented video information, comprising:
[0041] one or more processors;
[0042] A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations to obtain a target person detection model through the following steps, and then apply the target person detection model to complete the detection of a preset target person:
[0043] S1. Real-time acquisition of a video containing a target person walking, converting the video containing the target person walking into a time-sequentially arranged video frame sequence, extracting a preset number of consecutive video frames at a preset position in the video frame sequence as a valid frame sequence, taking the video frame sequence as input and the valid frame sequence as output, and constructing a valid frame sequence extraction module;
[0044] S2. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the deep convolutional neural network, taking each deep convolutional feature map corresponding to each video frame in the valid frame sequence as output, construct a deep convolutional feature map extraction module;
[0045] S3. Taking the effective frame sequence output by the effective frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, calculate the optical flow parameters of each group of video frame pairs as the motion information of the target person, and take the optical flow parameters of each group of video frame pairs in the effective frame sequence as output to construct an optical flow information module;
[0046] S4. Taking each deep convolution feature map output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear warping function, taking the deformation features of each group of video frame pairs as output, a deformation feature module is constructed;
[0047] S5. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence, and according to the fuzzy features and saliency features, and based on the softmax classification network, obtain the weight coefficient of each video frame in the valid frame sequence; taking the weight coefficient of each video frame in the valid frame sequence as output, construct a weight coefficient calculation module;
[0048] S6. Taking the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtain the aggregate features corresponding to the video frame group composed of the video frame pairs, and construct a detection network module by detecting the neural network with the preset information of the target person as output;
[0049] S7. Taking the video frame sequence corresponding to the video containing the target person walking in real time as input and the preset information of the target person as output, a target person detection model to be trained is constructed based on the effective frame sequence extraction module, the deep convolution feature map extraction module, the optical flow information module, the deformation feature module and the weight coefficient calculation module. Based on the training of the video samples containing the target person walking, the target person detection model is obtained to complete the detection of the target person.
[0050] The present invention also provides a computer-readable medium for storing software, characterized in that the computer-readable medium includes instructions that can be executed by one or more computers, and when the instructions are executed by the one or more computers, the operations of the task target detection method based on fragmented video information are performed.
[0051] Beneficial effects: Compared with the prior art, the advantages of the present invention include:
[0052] 1. The present invention introduces an optical flow neural network to calculate the smoothness between any two frames, not focusing only on the features of a certain frame, but on the relationship between the upper and lower frames.
[0053] 2. The present invention proposes a new video target detection algorithm, which mainly studies the influence of blur on video target detection. Frames with clear object appearance contribute more to the results than frames with blurred object appearance.
[0054] 3. The present invention uses a human target detection method based on fragmented video information, which helps to detect humans when the video is discontinuous and improves the accuracy of video target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 is a flowchart of a task target detection method based on fragmented video information provided by an embodiment of the present invention;
[0056] Figure 2 It is a schematic diagram of a network framework for task target detection based on fragmented video information provided according to an embodiment of the present invention. DETAILED DESCRIPTION
[0057] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0058] Reference Figure 1 , Figure 2 , a task target detection method based on fragmented video information provided by an embodiment of the present invention performs steps S1 to S7 according to a preset period to obtain a target person detection model, and then applies the target person detection model to complete the detection of a preset target person;
[0059] S1. Real-time acquisition of a video containing a target person walking, converting the video containing the target person walking into a time-sequentially arranged video frame sequence, extracting a preset number of consecutive video frames at a preset position in the video frame sequence as a valid frame sequence, taking the video frame sequence as input and the valid frame sequence as output, and constructing a valid frame sequence extraction module;
[0060] S2. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the deep convolutional neural network, taking each deep convolutional feature map corresponding to each video frame in the valid frame sequence as output, construct a deep convolutional feature map extraction module;
[0061] In one embodiment, the deep convolutional neural network is Restnet-101.
[0062] S3. Taking the effective frame sequence output by the effective frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, calculate the optical flow parameters of each group of video frame pairs as the motion information of the target person, and take the optical flow parameters of each group of video frame pairs in the effective frame sequence as output to construct an optical flow information module;
[0063] Since the resolution of the optical flow parameters output by the optical flow neural network does not match the resolution of the deep convolutional feature map, the size of the optical flow parameters needs to be adjusted to match the feature map.
[0064] In step S3, the effective frame sequence output by the effective frame sequence extraction module is used as input, and based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, the optical flow parameters of each group of video frame pairs are calculated as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the effective frame sequence are used as output. The specific steps of constructing the optical flow information module are as follows:
[0065] S31: define the tth video frame I in the valid frame sequence t is the reference frame, the t-τth frame video frame I in the valid frame sequence t-τ , the t+τth frame video frame I t+τ As the supporting frame, the reference frame I t , support frame I t-τ , support frame I t+τ Input optical flow neural network;
[0066] S32: Optical flow neural network includes convolutional layer, expansion layer, reference frame I t , support frame I t-τ , support frame I t+τ The reference frame I is obtained by the contraction part composed of the convolutional layer of the optical flow neural network t , support frame I t-τ , support frame I t+τ The corresponding feature maps respectively;
[0067] S33: Since the size of each feature map is reduced in step S32, it is necessary to go through an expansion layer to expand the size of each feature map to the original image size. t , support frame I t-τ, support frame I t+τ The corresponding feature maps are passed through the expansion layer of the optical flow neural network to obtain a reference frame I with a size expanded to the original image size. t , support frame I t-τ , support frame I t+τ The corresponding feature maps respectively;
[0068] S34: The so-called optical flow parameters utilize the changes in the pixels in each video frame in the effective frame sequence in the time domain and the correlation between two video frames to find the corresponding relationship between two video frames, thereby calculating the motion information of the target person.
[0069] Based on the reference frame I obtained in step S33 t , support frame I t-τ , support frame I t+τ The corresponding feature maps are used for optical flow prediction, respectively, with reference frame I t , support frame I t-τ As a video frame pair, and the reference frame I t , support frame I t+τ As a video frame pair, obtain the reference frame I t , support frame I t-τ The optical flow parameters M between the corresponding feature maps t-τ→t , and reference frame I t , support frame I t+τ The optical flow parameters M between the corresponding feature maps t+τ→t As follows:
[0070] M t-τ→t =FlowNet(I t-τ ,I t )
[0071] M t+τ→t =FlowNet(I t+τ ,I t )
[0072] Where M t-τ→t The reference frame I t , support frame I t-τ The optical flow parameters between the corresponding feature maps, t-τ→t represents the reference frame I t , support frame I t-τ The corresponding relationship, M t+τ→t The reference frame I t , support frame I t+τ The optical flow parameters between the corresponding feature maps, t+τ→t represents the reference frame I t , support frame I t+τ , FlowNet represents the optical flow neural network calculation.
[0073] S4. References Figure 2 In the figure, WARP represents a bilinear warping function, aggregation represents deformation features, and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the deep convolution feature map extraction module and the optical flow information module are used as input. Based on the bilinear warping function, the deformation features of each group of video frame pairs are used as output to construct a deformation feature module;
[0074] In step S4, each deep convolution feature map output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module are used as input, and based on the bilinear warping function, the deformation features of each group of video frame pairs are used as output. The specific method of constructing the deformation feature module is as follows:
[0075] f t-τ→t =W(f t-τ ,M t-τ→t )
[0076] f t+τ→t =W(f t+τ ,M t+τ→t )
[0077] In the formula, f t-τ→t The reference frame I t , support frame I t-τ The deformation characteristics between t+τ→t The reference frame I t , support frame I t+τ The deformation characteristics between them, W represents the calculation based on the bilinear distortion function, f t-τ The support frame I is output by the deep convolution feature map extraction module t-τ The corresponding feature map, f t+τ The support frame I is output by the deep convolution feature map extraction module t-τ The corresponding feature map.
[0078] S5. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence, and according to the fuzzy features and saliency features, and based on the softmax classification network, obtain the weight coefficient of each video frame in the valid frame sequence; taking the weight coefficient of each video frame in the valid frame sequence as output, construct a weight coefficient calculation module;
[0079] The fuzzy mapping network is DBM, and the saliency detection network is CSNet. The fuzzy mapping network is used to obtain the blur degree of the video frame, and the saliency detection network is used to eliminate background interference in the image.
[0080] Step S5 takes the valid frame sequence output by the valid frame sequence extraction module as input, obtains the fuzzy features and saliency features corresponding to each video frame based on the fuzzy mapping network and the saliency detection network, and obtains the weight coefficient of each video frame based on the fuzzy features and the saliency features and the softmax classification network; takes the weight coefficient of each video frame in the valid frame sequence as output, and constructs the specific steps of the weight coefficient calculation module as follows:
[0081] S51: Inputting each video frame in the valid frame sequence into the fuzzy mapping network and the saliency detection network respectively, and obtaining the fuzzy features and saliency features corresponding to each video frame respectively;
[0082] S52: Obtain the corrected blur map M corresponding to each video frame by multiplying the blur features and saliency features corresponding to each video frame obtained in step S51. blur-sali ;
[0083] S53: Based on the step function with a threshold of 0.5, the corrected blur map M corresponding to each video frame is blur-sali Binarization is performed, where the step function is as follows:
[0084]
[0085] Where m is the corrected blur map M corresponding to each video frame blur-sali The value of u(m) is the corrected blur map M corresponding to each video frame after binarization. blur-sali The value of
[0086] S54: For each video frame, all u(m) obtained in step S53 are added together to obtain the blur parameter Vcb of each video frame, and the blur parameter Vcb of each video frame is standardized. The standardization method is as follows:
[0087]
[0088] Where, Vcb i Indicates the blur parameter of video frame i, VcbNorm i represents the blur parameter of the standardized video frame i, and the value of i is {t-τ, t, t+τ};
[0089] S55: The blur parameter VcbNorm of each video frame obtained in step S54 after the normalization process is i Input to the softmax classification network to obtain the support frame I t-τ , reference frame I t , support frame I t+τ The corresponding weight coefficients ω t-τ ,ωt ,ω t+τ .
[0090] S6. Taking the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtain the aggregate features corresponding to the video frame group composed of the video frame pairs, and construct a detection network module by detecting the neural network with the preset information of the target person as output;
[0091] In one embodiment, the detection neural network is Faster R-CNN.
[0092] Step S6 takes the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtains the aggregate features corresponding to the video frame group composed of the video frame pairs, and uses the detection neural network to output the preset information of the target person. The specific steps of constructing the detection network module are as follows:
[0093] S61: Deformation feature f based on the output of the shape feature module t-τ→t 、f t+τ→t , the reference frame I output by the deep convolutional feature map extraction module t The corresponding feature map f t , and the weight coefficient ω corresponding to each deformation feature output by the weight coefficient calculation module t-τ ,ω t ,ω t+τ , according to the following formula, the support frame I is obtained t-τ , reference frame I t , support frame I t+τ Aggregate features J of the formed video frame group;
[0094] J = f t-τ→t ω t-τ +f t ω t +f t+τ→t ω t+τ
[0095] S62: Input the aggregated features into the detection neural network to obtain preset information of the target person.
[0096] S7. Taking the video frame sequence corresponding to the video containing the target person walking in real time as input and the preset information of the target person as output, a target person detection model to be trained is constructed based on the effective frame sequence extraction module, the deep convolution feature map extraction module, the optical flow information module, the deformation feature module and the weight coefficient calculation module. Based on the training of the video samples containing the target person walking, the target person detection model is obtained to complete the detection of the target person.
[0097] An embodiment of the present invention provides a task target detection system based on fragmented video information, including:
[0098] one or more processors;
[0099] A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations to obtain a target person detection model through the following steps, and then apply the target person detection model to complete the detection of a preset target person:
[0100] S1. Real-time acquisition of a video containing a target person walking, converting the video containing the target person walking into a time-sequentially arranged video frame sequence, extracting a preset number of consecutive video frames at a preset position in the video frame sequence as a valid frame sequence, taking the video frame sequence as input and the valid frame sequence as output, and constructing a valid frame sequence extraction module;
[0101] S2. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the deep convolutional neural network, taking each deep convolutional feature map corresponding to each video frame in the valid frame sequence as output, construct a deep convolutional feature map extraction module;
[0102] S3. Taking the effective frame sequence output by the effective frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, calculate the optical flow parameters of each group of video frame pairs as the motion information of the target person, and take the optical flow parameters of each group of video frame pairs in the effective frame sequence as output to construct an optical flow information module;
[0103] S4. Taking each deep convolution feature map output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear warping function, taking the deformation features of each group of video frame pairs as output, a deformation feature module is constructed;
[0104] S5. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence, and according to the fuzzy features and saliency features, and based on the softmax classification network, obtain the weight coefficient of each video frame in the valid frame sequence; taking the weight coefficient of each video frame in the valid frame sequence as output, construct a weight coefficient calculation module;
[0105] S6. Taking the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtain the aggregate features corresponding to the video frame group composed of the video frame pairs, and construct a detection network module by detecting the neural network with the preset information of the target person as output;
[0106] S7. Taking the video frame sequence corresponding to the video containing the target person walking in real time as input and the preset information of the target person as output, a target person detection model to be trained is constructed based on the effective frame sequence extraction module, the deep convolution feature map extraction module, the optical flow information module, the deformation feature module and the weight coefficient calculation module. Based on the training of the video samples containing the target person walking, the target person detection model is obtained to complete the detection of the target person.
[0107] A computer-readable medium storing software is provided in an embodiment of the present invention, wherein the computer-readable medium includes instructions that can be executed by one or more computers, and when the instructions are executed by the one or more computers, the operations of the task target detection method based on fragmented video information are performed.
[0108] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A task target detection method based on fragmented video information, characterized in that: Execute steps S1 to S7 according to a preset cycle to obtain a target person detection model, and then apply the target person detection model to complete the detection of the target person; S1. Real-time acquisition of a video containing a target person walking, converting the video containing the target person walking into a time-sequentially arranged video frame sequence, extracting a preset number of consecutive video frames at a preset position in the video frame sequence as a valid frame sequence, taking the video frame sequence as input and the valid frame sequence as output, and constructing a valid frame sequence extraction module; S2. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the deep convolutional neural network, taking each deep convolutional feature map corresponding to each video frame in the valid frame sequence as output, construct a deep convolutional feature map extraction module; S3. Taking the effective frame sequence output by the effective frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, calculate the optical flow parameters of each group of video frame pairs as the motion information of the target person, and take the optical flow parameters of each group of video frame pairs in the effective frame sequence as output to construct an optical flow information module; S4. Taking each deep convolution feature map output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear warping function, taking the deformation features of each group of video frame pairs as output, a deformation feature module is constructed; S5. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence, and according to the fuzzy features and saliency features, and based on the softmax classification network, obtain the weight coefficient of each video frame in the valid frame sequence; taking the weight coefficient of each video frame in the valid frame sequence as output, construct a weight coefficient calculation module; S6. Taking the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtain the aggregate features corresponding to the video frame group composed of the video frame pairs, and construct a detection network module by detecting the neural network with the preset information of the target person as output; S7. Taking the video frame sequence corresponding to the video containing the target person walking in real time as input and the preset information of the target person as output, a target person detection model to be trained is constructed based on the effective frame sequence extraction module, the deep convolution feature map extraction module, the optical flow information module, the deformation feature module and the weight coefficient calculation module. Based on the training of the video samples containing the target person walking, the target person detection model is obtained to complete the detection of the target person.
2. The task target detection method based on fragmented video information according to claim 1 is characterized in that: Step S3 takes the valid frame sequence output by the valid frame sequence extraction module as input, and based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the valid frame sequence, calculates the optical flow parameters of each group of video frame pairs as the motion information of the target person, and takes the optical flow parameters of each group of video frame pairs in the valid frame sequence as output. The specific steps of constructing the optical flow information module are as follows: S31: define the tth video frame I in the valid frame sequence t is the reference frame, the t-τth frame video frame I in the valid frame sequence t-τ , the t+τth frame video frame I t+τ As the supporting frame, the reference frame I t , Support frame I t-τ , Support frame I t+τ Input optical flow neural network; S32: Optical flow neural network includes convolutional layer, expansion layer, reference frame I t , Support frame I t-τ , Support frame I t+τ The reference frame I is obtained by the contraction part composed of the convolutional layer of the optical flow neural network t , Support frame I t-τ , Support frame I t+τ The corresponding feature maps respectively; S33: Reference Frame I t , Support frame I t-τ , Support frame I t+τ The corresponding feature maps are passed through the expansion layer of the optical flow neural network to obtain a reference frame I with a size expanded to the original image size. t , Support frame I t-τ , Support frame I t+τ The corresponding feature maps respectively; S34: Based on the reference frame I obtained in step S33 t , support frame I t-τ , support frame I t+τ The corresponding feature maps are used for optical flow prediction, respectively, with reference frame I t , Support frame I t-τ As a video frame pair, and the reference frame I t , Support frame I t+τ As a video frame pair, obtain the reference frame I t , support frame I t-T The optical flow parameters M between the corresponding feature maps t-τ→t , and reference frame I t , Support frame I t+τ The optical flow parameters M between the corresponding feature maps t+ τ →t As follows: M t-τ→t =FlowNet(I t-τ ,I t ) M t+τ→t =FlowNet(I t+τ ,I t ) Where M t-τ→t The reference frame I t , Support frame I t-τ The optical flow parameters between the corresponding feature maps, t-τ→t represents the reference frame I t , Support frame I t-τ The corresponding relationship, M t+τ→t The reference frame I t , Support frame I t+τ The optical flow parameters between the corresponding feature maps, t+τ→t represents the reference frame I t , Support frame I t+τ , FlowNet represents the optical flow neural network calculation.
3. The method for detecting a task target based on fragmented video information according to claim 2, characterized in that: In step S4, each deep convolution feature map output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module are used as input, and based on the bilinear warping function, the deformation features of each group of video frame pairs are used as output. The specific method of constructing the deformation feature module is as follows: f t-τ→t =W(f t-τ ,M t-τ→t ) f t+τ→t =W(f t+τ ,M t+τ→t ) In the formula, f t-τ→t The reference frame I t , Support frame I t-τ The deformation characteristics between t+τ→t The reference frame I t , Support frame I t+τ The deformation characteristics between them, W represents the calculation based on the bilinear distortion function, f t-τ The support frame I is output by the deep convolution feature map extraction module t-τ The corresponding feature map, f t+τ The support frame I is output by the deep convolution feature map extraction module t-τ The corresponding feature map.
4. The method for detecting a task target based on fragmented video information according to claim 3, characterized in that: Step S5 takes the valid frame sequence output by the valid frame sequence extraction module as input, obtains the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence based on the fuzzy mapping network and the saliency detection network, respectively, and obtains the weight coefficient of each video frame in the valid frame sequence based on the fuzzy features and the saliency features and the softmax classification network; takes the weight coefficient of each video frame in the valid frame sequence as output, and constructs the specific steps of the weight coefficient calculation module as follows: S51: inputting each video frame in the valid frame sequence into the fuzzy mapping network and the saliency detection network respectively, and obtaining the fuzzy features and saliency features corresponding to each video frame respectively; S52: Obtain the corrected blur map M corresponding to each video frame by multiplying the blur features and saliency features corresponding to each video frame obtained in step S51. blur-sali ; S53: Based on the step function with a threshold of 0.5, the corrected blur map M corresponding to each video frame is calculated. blur-sali Binarization is performed, where the step function is as follows: Where m is the corrected blur map M corresponding to each video frame blur-sali The value of u(m) is the corrected blur map M corresponding to each video frame after binarization. blur-sali The value of S54: For each video frame, all u(m) obtained in step S53 are added together to obtain the blur parameter Vcb of each video frame, and the blur parameter Vcb of each video frame is standardized. The standardization method is as follows: Where, Vcb i Indicates the blur parameter of video frame i, VcbNorm i represents the blur parameter of the standardized video frame i, and the value of i is {t-τ, t, t+τ}; S55: The blur parameter VcbNorm of each video frame obtained in step S54 after the normalization process is i Input to the softmax classification network to obtain the support frame I t-τ , reference frame I t , Support frame I t+τ The corresponding weight coefficients ω t-τ ,ω t ,ω t+τ .
5. The method for detecting a task target based on fragmented video information according to claim 4, characterized in that: Step S6 takes the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtains the aggregate features corresponding to the video frame group composed of the video frame pairs, and uses the detection neural network to output the preset information of the target person. The specific steps of constructing the detection network module are as follows: S61: Deformation feature f based on the output of the shape feature module t-τ→t 、f t+τ→t , the reference frame I output by the deep convolutional feature map extraction module t The corresponding feature map f t , and the weight coefficient ω corresponding to each deformation feature output by the weight coefficient calculation module t-τ ,ω t ,ω t+τ , according to the following formula, the support frame I is obtained t-τ , reference frame I t , Support frame I t+τ Aggregate features J of the formed video frame group; J=f t-τ→t oh t-τ +f t oh t +f t+τ→t oh t+τ S62: Input the aggregated features into the detection neural network to obtain preset information of the target person.
6. A task target detection system based on fragmented video information, characterized in that: include: one or more processors; A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations to obtain a target person detection model through the following steps, and then apply the target person detection model to complete the detection of a preset target person: S1. Real-time acquisition of a video containing a target person walking, converting the video containing the target person walking into a time-sequentially arranged video frame sequence, extracting a preset number of consecutive video frames at a preset position in the video frame sequence as a valid frame sequence, taking the video frame sequence as input and the valid frame sequence as output, and constructing a valid frame sequence extraction module; S2. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the deep convolutional neural network, taking each deep convolutional feature map corresponding to each video frame in the valid frame sequence as output, construct a deep convolutional feature map extraction module; S3. Taking the effective frame sequence output by the effective frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs consisting of two video frames separated by a preset number of frames in the effective frame sequence, calculate the optical flow parameters of each group of video frame pairs as the motion information of the target person, and take the optical flow parameters of each group of video frame pairs in the effective frame sequence as output to construct an optical flow information module; S4. Taking each deep convolution feature map output by the deep convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear warping function, taking the deformation features of each group of video frame pairs as output, a deformation feature module is constructed; S5. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence, and according to the fuzzy features and saliency features, and based on the softmax classification network, obtain the weight coefficient of each video frame in the valid frame sequence; taking the weight coefficient of each video frame in the valid frame sequence as output, construct a weight coefficient calculation module; S6. Taking the deformation features of each group of video frame pairs output by the deformation feature module, the deep convolution feature maps corresponding to each video frame output by the deep convolution feature map extraction module, and the weight coefficient of each video frame output by the weight coefficient calculation module as input, obtain the aggregate features corresponding to the video frame group composed of the video frame pairs, and construct a detection network module by detecting the neural network with the preset information of the target person as output; S7. Taking the video frame sequence corresponding to the video containing the target person walking in real time as input and the preset information of the target person as output, a target person detection model to be trained is constructed based on the effective frame sequence extraction module, the deep convolution feature map extraction module, the optical flow information module, the deformation feature module and the weight coefficient calculation module. Based on the training of the video samples containing the target person walking, the target person detection model is obtained to complete the detection of the target person.
7. A computer-readable medium storing software, characterized in that: The readable medium includes instructions that can be executed by one or more computers, and when the instructions are executed by the one or more computers, the operations of the task target detection method based on fragmented video information as described in any one of claims 1-5 are performed.
