A general visual model training method and system

Abstract

The application relates to a general visual model training method and system, and belongs to the technical field of computer vision. The method steps comprise the following: acquiring a video sample to be detected, extracting multi-level multi-scale features of continuous multiple frames of images in a video sequence by using a backbone network; predicting an optical flow field from a frame to be detected to a target frame by using an optical flow prediction network based on features of adjacent frames; calculating motion displacement ground truth by using ground truth frame information of the frame to be detected and the target frame, and supervising and constraining the optical flow field; twisting and aligning adjacent frame features by using the optical flow field, and fusing the adjacent frame features with features of the frame to be detected to generate enhanced features; performing target classification and positioning by using a target detection network based on the enhanced features; combining target detection loss and optical flow supervision loss to construct a total loss function, and performing end-to-end joint optimization training on the backbone network, the optical flow prediction network and the target detection network; and outputting a video monitoring result by using a trained visual model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a general visual model training method and system. Background Technology

[0002] Object detection is a classic research problem in computer vision. The basic task is to identify multiple objects in an image and locate the different objects (providing bounding boxes). Compared to classification problems, detection involves understanding the foreground and background of an image. We need to separate the target of interest from the background and determine its description (category and location). Therefore, the output of a detection model is a list, where each item uses a set of data to indicate the category of the detected target.

[0003] Optical flow estimation is a classic research problem in computer vision. In computer vision, this task defines the movement of an object in an image, which can be caused by camera movement or object movement. Specifically, it refers to the amount of movement of pixels representing the same object in one frame of a video image to the next frame, represented by a two-dimensional vector (representing movement in the x and y directions, respectively). Optical flow plays a crucial role in computer vision because it contains underlying motion information, and therefore has significant applications in video object detection, object segmentation, action recognition, object tracking, robot navigation, and shape information recovery.

[0004] The prior art CN116229336A, a video moving target recognition method, system, storage medium, and computer, discloses that: this scheme mainly relies on background subtraction and static feature extraction, lacking explicit modeling of the target's motion trajectory. Its background estimation module generates background frames by concatenating multi-level, multi-scale features, but does not consider the continuous motion relationship of the target between frames, resulting in a high false detection rate in fast-moving or complex dynamic scenes. Furthermore, directly performing channel stitching or averaging on multiple frames lacks adaptive correction for motion deformation, easily causing feature blurring or loss of detail. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a general visual model training method and system.

[0006] The technical solution of the present invention is as follows:

[0007] On the one hand, this invention proposes a general visual model training method, the specific steps of which include:

[0008] Acquire video samples to be detected, and use the backbone network to extract multi-level, multi-scale features of multiple consecutive frames in the video sequence.

[0009] Based on the multi-level, multi-scale features of adjacent frames, an optical flow prediction network is used to predict the optical flow field from the image to be detected to the target image.

[0010] Using the ground truth bounding box information in the frame image to be detected and the target frame image, the motion displacement ground truth value of the region corresponding to the ground truth bounding box is calculated, and the optical flow vector of the corresponding region in the predicted optical flow field is subject to supervised constraint based on the motion displacement ground truth value; the specific steps of performing supervised constraint are as follows:

[0011] For each ground truth box in the frame image to be detected, calculate the geometric center displacement between it and the corresponding ground truth box in the target frame image, and use it as the motion displacement ground truth value of the region corresponding to the ground truth box of the frame to be detected.

[0012] On the predicted optical flow field, the average optical flow vector of the region corresponding to the ground truth box of the frame to be detected is statistically analyzed;

[0013] The difference between the average optical flow vector and the true value of the motion displacement is calculated, and a preset difference threshold range is used as an optical flow supervision constraint.

[0014] Using the predicted optical flow field, multi-level, multi-scale features of adjacent frame images are distorted and aligned to the multi-level, multi-scale features of the frame image to be detected, and then fused with the same-level, same-scale features of the frame image to be detected to generate enhanced multi-level, multi-scale features of the frame image to be detected; the distortion and alignment step specifically includes:

[0015] Based on the optical flow field from the frame to be detected to the target frame, the multi-level, multi-scale features of the target frame are spatially aligned to the features of the corresponding scale of the frame to be detected through differentiable sampling transformation, generating a distorted feature map consistent with the feature space of the frame to be detected.

[0016] The differentiable transformation employs bilinear interpolation to resample the feature values.

[0017] The fusion step is specifically as follows:

[0018] For the distorted feature maps of adjacent frames and the feature map of the frame to be detected, a spatially adaptive weight map is learned through a convolutional neural network;

[0019] The weighted graph is used to perform weighted fusion of the distortion features of adjacent frame images and the same-scale features of the frame image to be detected;

[0020] Based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, the target is classified and located through the target detection network, and the detection result of the frame to be detected is output.

[0021] Based on the detection results and ground truth information of the frame image to be detected, a target detection loss function is constructed. Based on the ground truth information of the same instance in the frame to be detected and the target frame, an optical flow supervised loss function is constructed. The total loss function is constructed by combining the optical flow supervised loss function and the target detection loss function. The backbone network, optical flow prediction network and target detection network are jointly optimized and trained end-to-end. The trained visual model is used to output the video monitoring results.

[0022] In a preferred embodiment, the multi-level multi-scale features include shallow representation information and deep semantic information.

[0023] In a preferred embodiment, the backbone network adopts ResNet or VGG, and its output contains feature maps with multiple different receptive fields.

[0024] As a preferred embodiment, the optical flow prediction network employs the PWC-Net, FlowNet2.0, or LiteFlowNet algorithm.

[0025] In a preferred embodiment, during the pre-training process of the object detection network, Hungarian matching is used to allocate positive and negative samples. Each ground truth box target in the image to be detected is handled by only one object query. At the same time, the object detection network adopts a stepwise optimization strategy, that is, the object detection network is composed of multiple object detection modules stacked together.

[0026] In a preferred embodiment, the step of constructing the target detection loss function based on the detection result of the frame to be detected and the ground truth box information of the image of the frame to be detected specifically includes:

[0027] The detection result of the frame to be detected includes the category probability of the detected target and the bounding box coordinates;

[0028] The detection results are matched with the ground truth bounding boxes of the frame image to be detected, and the classification loss and regression loss are calculated respectively:

[0029] Construct a classification loss function to calculate the classification loss between the predicted class probability of the detected target and the true class label;

[0030] Construct a regression loss function to calculate the regression loss between the predicted bounding box coordinates and the true bounding box coordinates;

[0031] The classification loss function and the regression loss function are weighted and fused to obtain the final target detection loss function.

[0032] On the other hand, this invention proposes a general visual model training system, comprising:

[0033] The feature extraction module acquires video samples to be detected and uses the backbone network to extract multi-level, multi-scale features from multiple consecutive frames of images in the video sequence.

[0034] The optical flow prediction module predicts the optical flow field from the image to be detected to the target image through an optical flow prediction network, based on the multi-level, multi-scale features of adjacent frames.

[0035] The optical flow supervision module uses the ground truth box information in the frame image to be detected and the target frame image to calculate the ground truth value of the motion displacement of the region corresponding to the ground truth box, and performs supervision constraints on the optical flow vector of the corresponding region in the predicted optical flow field based on the ground truth value of the motion displacement; the specific steps of performing supervision constraints are as follows:

[0036] For each ground truth box in the frame image to be detected, calculate the geometric center displacement between it and the corresponding ground truth box in the target frame image, and use it as the motion displacement ground truth value of the region corresponding to the ground truth box of the frame to be detected.

[0037] On the predicted optical flow field, the average optical flow vector of the region corresponding to the ground truth box of the frame to be detected is statistically analyzed;

[0038] The difference between the average optical flow vector and the true value of the motion displacement is calculated, and a preset difference threshold range is used as an optical flow supervision constraint.

[0039] The feature fusion module utilizes the predicted optical flow field to distort and align multi-level, multi-scale features of adjacent frame images to the multi-level, multi-scale features of the frame image to be detected, and then fuses them with the same-scale features of the frame image to be detected, generating enhanced multi-level, multi-scale features of the frame image to be detected; the distortion and alignment step specifically involves:

[0040] Based on the optical flow field from the frame to be detected to the target frame, the multi-level, multi-scale features of the target frame are spatially aligned to the features of the corresponding scale of the frame to be detected through differentiable sampling transformation, generating a distorted feature map consistent with the feature space of the frame to be detected.

[0041] The differentiable transformation employs bilinear interpolation to resample the feature values.

[0042] The fusion step is specifically as follows:

[0043] For the distorted feature maps of adjacent frames and the feature map of the frame to be detected, a spatially adaptive weight map is learned through a convolutional neural network;

[0044] The weighted graph is used to perform weighted fusion of the distortion features of adjacent frame images and the same-scale features of the frame image to be detected;

[0045] Based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, the target is classified and located through the target detection network, and the detection result of the frame to be detected is output.

[0046] The target detection module, based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, performs target classification and localization through a target detection network and outputs the detection results of the frame to be detected.

[0047] The joint training module constructs a target detection loss function based on the detection results and ground truth information of the frame image to be detected, and constructs an optical flow supervision loss function based on the ground truth information of the same instance in the frame to be detected and the target frame. The total loss function is constructed by combining the optical flow supervision loss function and the target detection loss function. The backbone network, optical flow prediction network and target detection network are jointly optimized and trained end-to-end. The trained visual model is used to output the video monitoring results.

[0048] The present invention has the following beneficial effects:

[0049] 1. This invention uses optical flow field to distort and align features from adjacent frames and fuse them into the current frame. Essentially, it utilizes temporal context information to "repair" information loss caused by blurring or brief occlusion due to rapid motion in the current frame. Even if the target in the frame to be detected is unclear, the system can borrow clear features from adjacent frames to supplement it, thereby making a more reliable judgment.

[0050] 2. The feature fusion in this invention is not a simple addition of features, but rather a spatially adaptive weight map learned through a convolutional neural network. This enables the system to intelligently and dynamically select more reliable features from the reference frame or adjacent frames at each spatial location, generating higher-quality, more discriminative fused features.

[0051] 3. This invention employs an end-to-end training method that combines optical flow supervision loss and object detection loss, thereby unifying the optimization objectives of the two tasks. To reduce the total loss, the optical flow network learns to predict optical flow that is more conducive to feature alignment for the detection task; while the feature extraction network is forced to learn general features applicable to both optical flow and detection.

[0052] 4. Traditional high-precision optical flow estimation requires dense pixel-level annotation, which is extremely costly. This solution only requires object detection boxes to achieve effective and targeted supervision of optical flow, providing a new approach for training high-performance optical flow models on a wider range of datasets.

[0053] 5. This invention utilizes object query to remove redundant predicted target boxes and gradually improves the accuracy of predicted target boxes by stacking multiple target detection modules to build an target detection network. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the steps of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0057] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0058] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0059] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0060] Example 1:

[0061] See Figure 1 A general visual model training method, the specific steps of which include:

[0062] Acquire video samples to be detected, and use the backbone network to extract multi-level, multi-scale features of multiple consecutive frames in the video sequence.

[0063] Based on the multi-level, multi-scale features of adjacent frames, an optical flow prediction network is used to predict the optical flow field from the image to be detected to the target image.

[0064] Using the ground truth box information in the frame image to be detected and the target frame image, the motion displacement ground truth of the region corresponding to the ground truth box is calculated, and the optical flow vector of the corresponding region in the predicted optical flow field is supervised and constrained based on the motion displacement ground truth.

[0065] In this embodiment, the truth box information is a pre-prepared training dataset with precise annotations. These annotations are typically done manually by professional annotators using annotation tools. For example, for a video sequence, the annotator needs to draw bounding boxes and assign category labels (such as "person", "vehicle", etc.) to each target object in the first frame (the frame to be detected) and the second frame (the target frame) used for training. This process ensures the accuracy of the annotation information.

[0066] Commonly used video object detection or optical flow estimation datasets, such as ImageNet VID and KITTI, provide such frame-by-frame or sparse frame annotation information.

[0067] Using the predicted optical flow field, the multi-level, multi-scale features of adjacent frame images are distorted and aligned to the multi-level, multi-scale features of the frame image to be detected, and then fused with the same-scale features of the frame image to be detected to generate the enhanced multi-level, multi-scale features of the frame image to be detected.

[0068] Based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, the target is classified and located through the target detection network, and the detection result of the frame to be detected is output.

[0069] Based on the detection results and ground truth information of the frame image to be detected, a target detection loss function is constructed. Based on the ground truth information of the same instance in the frame to be detected and the target frame, an optical flow supervised loss function is constructed. The total loss function is constructed by combining the optical flow supervised loss function and the target detection loss function. The backbone network, optical flow prediction network and target detection network are jointly optimized and trained end-to-end. The trained visual model is used to output the video monitoring results.

[0070] In this embodiment, the step of constructing the total loss function by combining the optical flow supervision loss function and the target detection loss function adopts a weighted summation method, specifically as follows:

[0071]

[0072] Among them, the optical flow supervision loss and the target detection loss are loss values ​​calculated separately; and It is a hyperparameter, representing the weight coefficients of the two loss terms.

[0073] The weighting coefficients are used to balance the importance of the two tasks during training and their influence on the direction of model parameter updates. If α is set larger, training will focus more on improving the accuracy of optical flow estimation; if β is set larger, training will focus more on improving the performance of object detection.

[0074] The weighting coefficients can be fixed parameters, usually determined experimentally to obtain the best overall performance on the validation set, or they can be learnable parameters; in this scheme, fixed parameters are typically used.

[0075] Once the total loss function is calculated, backpropagation is performed. The gradients start from the total loss and propagate back through multiple trainable networks, including the optical flow supervision loss function, the object detection loss function, the feature fusion network, the optical flow prediction network, and the backbone network. The gradient of the object detection loss guides the object detection network and the feature fusion network to learn how to generate more accurate detection results. This, in turn, requires that the preceding optical flow prediction be sufficiently accurate to achieve effective feature alignment. The gradient of the optical flow supervision loss guides the fusion module in the optical flow prediction network to learn how to predict optical flow that better conforms to semantic motion patterns in moving target regions. The gradients of the two losses work together on the feature extraction module of the backbone network, enabling it to extract general features that are beneficial for both optical flow estimation and object detection.

[0076] End-to-end joint training refers to the entire system (from the input raw video frames to the final output object detection result) being a complete, differentiable learning model. During training, the input is the raw image data, and the final supervision signal (i.e., the total loss) can be directly backpropagated from the output (detection result) to the feature extraction layer at the front end of the model.

[0077] In a preferred embodiment of this invention, the multi-level multi-scale features include shallow representation information and deep semantic information.

[0078] In this embodiment, the shallow representation information mainly comes from the bottom or early layers of the feature extraction network, containing details and local features of the image:

[0079] Edge features: the boundary lines of an object's outline, such as the edges of a building or the outline of a vehicle;

[0080] Texture features: Repeating patterns on the surface of an object, such as brick wall texture, grass texture, clothing fibers, etc.

[0081] Color characteristics: Color distribution information of pixels, such as the blue gradient of the sky, the red of traffic signs, etc.

[0082] Corner features: Obvious corners or key points in an image, such as window corners, road intersections, etc.

[0083] Basic shapes: Simple geometric shapes, such as round wheels, rectangular doors and windows, etc.

[0084] Deep semantic information comes from high-level or late-level layers of the feature extraction network and is abstracted through multiple convolution and pooling operations:

[0085] Object category semantics: Recognizes high-level concepts such as "vehicles", "pedestrians", and "traffic signs";

[0086] Relationships between object components: Understand the compositional relationships of "a car includes components such as wheels, windows, and lights";

[0087] Scene context: Recognize the overall scene semantics, such as "urban road scene" and "indoor environment";

[0088] Functional semantics: Understanding functional concepts such as "drivable area" and "obstacle area";

[0089] Complex Patterns: Recognize complex visual patterns such as "traffic congestion" and "crowd gathering".

[0090] In a preferred embodiment of this invention, the backbone network adopts ResNet or VGG, and its output contains feature maps with multiple different receptive fields.

[0091] In a preferred embodiment of this invention, the optical flow prediction network employs the PWC-Net, FlowNet2.0, or LiteFlowNet algorithm.

[0092] In this embodiment, the PWC-Net algorithm is used as the optical flow prediction network. The pyramid processing framework used by WC-Net is as follows:

[0093] First, the feature extraction network generates a multi-scale feature pyramid for each of the two input images (Image1 and Image2). For example, the pyramid may contain features at multiple scales, from layer 4 (lowest resolution, such as 32x32 pixels) to layer 1 (highest resolution, such as 256x256 pixels).

[0094] The optical flow estimation process starts from the top layer (the 4th layer with the lowest resolution) and proceeds downwards layer by layer until the bottom layer (the 1st layer with the highest resolution).

[0095] At the top level of the pyramid (level 4), since this is the starting point for optical flow estimation, there is no prior optical flow information to refer to. Therefore, the "coarse optical flow" here is initialized as a vector field of all zeros. This means that the network assumes there is no relative motion between images when it starts at the top level.

[0096] In a preferred embodiment of this invention, the step of calculating the motion displacement truth value of the region corresponding to the truth value box using the truth value box information in the frame image to be detected and the target frame image, and performing supervised constraint on the optical flow vector of the corresponding region in the predicted optical flow field based on the motion displacement truth value, specifically includes:

[0097] For each ground truth box in the frame image to be detected, calculate the geometric center displacement between it and the corresponding ground truth box in the target frame image, and use it as the motion displacement ground truth value of the region corresponding to the ground truth box of the frame to be detected.

[0098] On the predicted optical flow field, the average optical flow vector of the region corresponding to the ground truth box of the frame to be detected is statistically analyzed;

[0099] The difference between the average optical flow vector and the true value of the motion displacement is calculated, and a preset difference threshold range is used as an optical flow supervision constraint.

[0100] In this embodiment, the above steps are explained in detail and analyzed with a set of examples:

[0101] The specific calculation process for the displacement of the geometric center:

[0102] In the frame to be detected (frame 1) and the target frame (frame 2), find two truth boxes that represent the same object;

[0103] Calculate the coordinates of the geometric center points of the two boxes separately, usually the center points of the boxes;

[0104] Subtracting the center point coordinates of the target frame from the center point coordinates of the detected frame yields a two-dimensional vector (Δx, Δy). This vector represents the overall motion vector of the object between the two frames on the image plane.

[0105] Example:

[0106] In the frame to be detected, the center of the detection box for a car is at (100, 200).

[0107] In the target frame, the center of the detection box for the same vehicle is at (110, 195).

[0108] Therefore, the geometric center displacement vector is (110-100, 195-200) = (10, -5).

[0109] This means that we consider the car to have moved 10 pixels to the right and 5 pixels up. This (10, -5) is the ground truth value for optical flow supervision generated for the "car" region.

[0110] The calculation process of the average optical flow vector:

[0111] On the dense optical flow map predicted by the optical flow network, find all pixels located within the ground truth box of the frame to be detected, and each pixel has an optical flow prediction vector.

[0112] The optical flow vectors of all these pixels are averaged (averaged in both the x and y directions) to obtain a single two-dimensional vector.

[0113] Example:

[0114] The optical flow network predicts the motion of every pixel in the entire image. We extract the predicted optical flow for each pixel within the vehicle frame (e.g., 2000 pixels).

[0115] These 2000 vectors may be different from each other, including a series of vectors such as (9,-4), (12,-7), (8,-3), etc.

[0116] We take the average of the x and y components of these 2000 vectors to get an average value, such as (10.2, -5.1). This (10.2, -5.1) is the average result of the network's prediction of the overall motion of the "car" region.

[0117] The specific steps for calculating the difference between the average optical flow vector and the geometric center displacement are as follows:

[0118] By calculating the Euclidean distance between the average optical flow vector and the geometric center displacement:

[0119]

[0120] In the formula, and The x and y coordinates of the displacement vector at the geometric center are... and Let x and y be the x and y coordinates of the average optical flow vector. Substituting the data from the previous example, we can obtain... This loss value of 0.22 reflects the deviation between the predicted average motion of the optical flow in the target area and the actual overall motion of the object.

[0121] Then set an allowable deviation range as the threshold range to ensure the accuracy of the results.

[0122] In a preferred embodiment of this example, the step of distorting and aligning the multi-level, multi-scale features of adjacent frame images to the multi-level, multi-scale features of the frame image to be detected using the predicted optical flow field specifically comprises:

[0123] Based on the optical flow field from the frame to be detected to the target frame, the multi-level, multi-scale features of the target frame are spatially aligned to the features of the corresponding scale of the frame to be detected through differentiable sampling transformation, generating a distorted feature map consistent with the feature space of the frame to be detected.

[0124] The differentiable transformation employs bilinear interpolation to resample the feature values.

[0125] In this embodiment, the detailed execution steps of the above steps are as follows:

[0126] The feature extraction network (backbone network) extracts a set of multi-level, multi-scale feature maps for the frame to be detected (Img1) and the target frame (Img2), such as a feature pyramid containing four scales: {P2, P3, P4, P5}, where P2 has the highest resolution and P5 has the lowest resolution.

[0127] The optical flow prediction network also operates in a pyramidal form. It estimates a set of multi-scale optical flow fields {Flow2, Flow3, Flow4, Flow5} from the frame to be detected to the target frame, with each scale of optical flow field having the same size as its corresponding feature map.

[0128] Before performing distortion alignment, it is necessary to ensure that the multi-level, multi-scale feature maps and the optical flow field are operated on at the same scale.

[0129] For a target pixel p = (x, y) on the feature map F1_p3 of the frame to be detected, the optical flow field Flow3 gives a displacement vector Δp ​​= (u, v) at that position. This vector indicates that the corresponding point of point p in the frame to be detected is located at p' = (x+u, y+v) on the feature map F2_p3 of the target frame, where P3 represents the scale and F represents the feature map number.

[0130] However, the coordinates of p' are often decimals. Therefore, it is necessary to use bilinear interpolation to calculate the eigenvalues ​​at that position based on the eigenvalues ​​of the four integer coordinate points surrounding p'.

[0131] This feature value, calculated through interpolation, is used as the value of the distorted feature map at position p in the frame to be detected.

[0132] The warp operation can be represented by a function called Warp:

[0133] F2_warped_p3 = Warp(F2_p3, Flow3)

[0134] Here, F2_warped_p3 is the warped and aligned target frame P3 scale feature. Now, its spatial layout has been aligned with the feature map F1_p3 of the frame to be detected based on motion information.

[0135] The above steps are performed independently at each scale. Finally, we obtain a set of warped multi-scale features of the target frame {F2_warped_p2, F2_warped_p3, F2_warped_p4, F2_warped_p5} aligned with the multi-scale feature space of the frame to be detected.

[0136] In a preferred embodiment of this invention, the step of distorting and aligning the multi-level, multi-scale features of adjacent frame images to the multi-level, multi-scale features of the frame image to be detected, and then fusing them with the same-scale features of the frame image to be detected, specifically comprises:

[0137] For the distorted feature maps of adjacent frames and the feature map of the frame to be detected, a spatially adaptive weight map is learned through a convolutional neural network;

[0138] The weighted graph is used to perform weighted fusion of the distortion features of adjacent frame images and the same-scale features of the frame image to be detected.

[0139] In this embodiment, the convolutional neural network is a lightweight sub-network with an encoder-decoder structure. Its design goal is to efficiently process local features and output a weight map that matches the size of the input feature map.

[0140] This small CNN takes as input distorted features from neighboring frames and features from the frame to be detected. Typically, these two types of features are concatenated along the channel dimension, providing the network with all the contextual information needed to make decisions. The network's output is a single-channel weight map with the same spatial dimensions (height and width) as the input feature map. The value of each pixel in the weight map is typically between 0 and 1 (achieved using the sigmoid activation function), representing the weight of that spatial location during feature fusion.

[0141] Obtain the weighted graph (After the weight map W is automatically learned by the network based on the input feature content), the final fused feature F_fused is obtained through the following calculation:

[0142]

[0143] In the formula, As a feature of fusion, Features of the frame to be detected This represents the distortion features of adjacent frame images.

[0144] When the weight W at a certain position is close to 1, the output feature mainly depends on the feature F_ref of the frame to be detected. This usually occurs in regions where the feature quality of the frame to be detected is high, while the distortion feature is unreliable due to optical flow estimation errors or occlusion.

[0145] When the weight at a certain location is close to 0, the output feature mainly depends on the warped frame feature F_warp. This usually occurs in regions where there is motion blur or occlusion in the frame to be detected, while adjacent frames can provide clearer information.

[0146] In a preferred embodiment of this invention, during the pre-training process of the target detection network, Hungarian matching is used to allocate positive and negative samples. Each ground truth box target in the image to be detected is handled by only one object query. At the same time, the target detection network adopts a stepwise optimization strategy, that is, the target detection network is composed of multiple target detection modules stacked together.

[0147] In this embodiment, in order to enable the model to run in an end-to-end manner, we use Hungarian matching to assign positive and negative samples during training, so that each ground truth bounding box is handled by only one object query. This way, no further post-processing is needed between object queries to remove redundant predicted bounding boxes during inference.

[0148] To obtain more accurate predicted detection boxes, we also adopted a stepwise optimization strategy, which involves stacking multiple object detection modules. The predicted detection boxes of the previous module are used as inputs to the next module. By continuously optimizing multiple modules, the accuracy of the predicted detection boxes is gradually improved.

[0149] In a preferred embodiment of this example, the step of constructing the target detection loss function based on the detection result of the frame to be detected and the ground truth box information of the image of the frame to be detected specifically includes:

[0150] The detection result of the frame to be detected includes the category probability of the detected target and the bounding box coordinates;

[0151] The detection results are matched with the ground truth bounding boxes of the frame image to be detected, and the classification loss and regression loss are calculated respectively:

[0152] Construct a classification loss function to calculate the classification loss between the predicted class probability of the detected target and the true class label;

[0153] Construct a regression loss function to calculate the regression loss between the predicted bounding box coordinates and the true bounding box coordinates;

[0154] The classification loss function and the regression loss function are weighted and fused to obtain the final target detection loss function.

[0155] In this embodiment, the step of matching the detection result with the ground truth box information of the frame image to be detected uses IoU calculation, the classification loss function uses the cross-entropy loss function, and the regression loss function uses the Smooth L1 loss function.

[0156] Example 2:

[0157] The feature extraction module acquires video samples to be detected and uses the backbone network to extract multi-level, multi-scale features from multiple consecutive frames of images in the video sequence.

[0158] The optical flow prediction module predicts the optical flow field from the image to be detected to the target image through an optical flow prediction network, based on the multi-level, multi-scale features of adjacent frames.

[0159] The optical flow supervision module uses the ground truth box information in the frame image to be detected and the target frame image to calculate the ground truth value of the motion displacement of the region corresponding to the ground truth box, and performs supervision constraints on the optical flow vector of the corresponding region in the predicted optical flow field based on the ground truth value of the motion displacement; the specific steps of performing supervision constraints are as follows:

[0160] For each ground truth box in the frame image to be detected, calculate the geometric center displacement between it and the corresponding ground truth box in the target frame image, and use it as the motion displacement ground truth value of the region corresponding to the ground truth box of the frame to be detected.

[0161] On the predicted optical flow field, the average optical flow vector of the region corresponding to the ground truth box of the frame to be detected is statistically analyzed;

[0162] The difference between the average optical flow vector and the true value of the motion displacement is calculated, and a preset difference threshold range is used as an optical flow supervision constraint.

[0163] The feature fusion module utilizes the predicted optical flow field to distort and align multi-level, multi-scale features of adjacent frame images to the multi-level, multi-scale features of the frame image to be detected, and then fuses them with the same-scale features of the frame image to be detected, generating enhanced multi-level, multi-scale features of the frame image to be detected; the distortion and alignment step specifically involves:

[0164] Based on the optical flow field from the frame to be detected to the target frame, the multi-level, multi-scale features of the target frame are spatially aligned to the features of the corresponding scale of the frame to be detected through differentiable sampling transformation, generating a distorted feature map consistent with the feature space of the frame to be detected.

[0165] The differentiable transformation employs bilinear interpolation to resample the feature values.

[0166] The fusion step is specifically as follows:

[0167] For the distorted feature maps of adjacent frames and the feature map of the frame to be detected, a spatially adaptive weight map is learned through a convolutional neural network;

[0168] The weighted graph is used to perform weighted fusion of the distortion features of adjacent frame images and the same-scale features of the frame image to be detected;

[0169] Based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, the target is classified and located through the target detection network, and the detection result of the frame to be detected is output.

[0170] The target detection module, based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, performs target classification and localization through a target detection network and outputs the detection results of the frame to be detected.

[0171] The joint training module constructs a target detection loss function based on the detection results and ground truth information of the frame image to be detected, and constructs an optical flow supervision loss function based on the ground truth information of the same instance in the frame to be detected and the target frame. The total loss function is constructed by combining the optical flow supervision loss function and the target detection loss function. The backbone network, optical flow prediction network and target detection network are jointly optimized and trained end-to-end. The trained visual model is used to output the video monitoring results.

[0172] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A general visual model training method, characterized in that, The specific steps include: Acquire video samples to be detected, and use the backbone network to extract multi-level, multi-scale features of multiple consecutive frames in the video sequence. Based on the multi-level, multi-scale features of adjacent frames, an optical flow prediction network is used to predict the optical flow field from the image to be detected to the target image. Using the ground truth bounding box information in the frame image to be detected and the target frame image, the motion displacement ground truth value of the region corresponding to the ground truth bounding box is calculated, and the optical flow vector of the corresponding region in the predicted optical flow field is subject to supervised constraint based on the motion displacement ground truth value; the specific steps of performing supervised constraint are as follows: For each ground truth box in the frame image to be detected, calculate the geometric center displacement between it and the corresponding ground truth box in the target frame image, and use it as the motion displacement ground truth value of the region corresponding to the ground truth box of the frame to be detected. On the predicted optical flow field, the average optical flow vector of the region corresponding to the ground truth box of the frame to be detected is statistically analyzed; The difference between the average optical flow vector and the true value of the motion displacement is calculated, and a preset difference threshold range is used as an optical flow supervision constraint. Using the predicted optical flow field, multi-level, multi-scale features of adjacent frame images are distorted and aligned to the multi-level, multi-scale features of the frame image to be detected, and then fused with the same-level, same-scale features of the frame image to be detected to generate enhanced multi-level, multi-scale features of the frame image to be detected; the distortion and alignment step specifically includes: Based on the optical flow field from the frame to be detected to the target frame, the multi-level, multi-scale features of the target frame are spatially aligned to the features of the corresponding scale of the frame to be detected through differentiable sampling transformation, generating a distorted feature map consistent with the feature space of the frame to be detected. The differentiable transformation employs bilinear interpolation to resample the feature values. The fusion step is specifically as follows: For the distorted feature maps of adjacent frames and the feature map of the frame to be detected, a spatially adaptive weight map is learned through a convolutional neural network; The weighted graph is used to perform weighted fusion of the distortion features of adjacent frame images and the same-scale features of the frame image to be detected; Based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, the target is classified and located through the target detection network, and the detection result of the frame to be detected is output. Based on the detection results and ground truth information of the frame image to be detected, a target detection loss function is constructed. Based on the ground truth information of the same instance in the frame to be detected and the target frame, an optical flow supervised loss function is constructed. The total loss function is constructed by combining the optical flow supervised loss function and the target detection loss function. The backbone network, optical flow prediction network and target detection network are jointly optimized and trained end-to-end. The trained visual model is used to output the video monitoring results.

2. The general visual model training method according to claim 1, characterized in that, The multi-level, multi-scale features include shallow representation information and deep semantic information.

3. The general visual model training method according to claim 1, characterized in that, The backbone network uses ResNet or VGG, and its output contains feature maps with multiple different receptive fields.

4. The general visual model training method according to claim 1, characterized in that, The optical flow prediction network uses the PWC-Net, FlowNet2.0, or LiteFlowNet algorithm.

5. The general visual model training method according to claim 1, characterized in that, During the pre-training process of the object detection network, Hungarian matching is used to assign positive and negative samples. Each ground truth box target in the image to be detected is handled by only one object query. At the same time, the object detection network adopts a stepwise optimization strategy, that is, the object detection network is composed of multiple object detection modules stacked together.

6. The general visual model training method according to claim 1, characterized in that, The specific steps for constructing the target detection loss function based on the detection result of the frame to be detected and the ground truth box information of the image of the frame to be detected are as follows: The detection result of the frame to be detected includes the category probability of the detected target and the bounding box coordinates; The detection results are matched with the ground truth bounding boxes of the frame image to be detected, and the classification loss and regression loss are calculated respectively: Construct a classification loss function to calculate the classification loss between the predicted class probability of the detected target and the true class label; Construct a regression loss function to calculate the regression loss between the predicted bounding box coordinates and the true bounding box coordinates; The classification loss function and the regression loss function are weighted and fused to obtain the final target detection loss function.

7. A general-purpose visual model training system, characterized in that, include: The feature extraction module acquires video samples to be detected and uses the backbone network to extract multi-level, multi-scale features from multiple consecutive frames of images in the video sequence. The optical flow prediction module predicts the optical flow field from the image to be detected to the target image through an optical flow prediction network, based on the multi-level, multi-scale features of adjacent frames. The optical flow supervision module uses the ground truth box information in the frame image to be detected and the target frame image to calculate the ground truth value of the motion displacement of the region corresponding to the ground truth box, and performs supervision constraints on the optical flow vector of the corresponding region in the predicted optical flow field based on the ground truth value of the motion displacement; the specific steps of performing supervision constraints are as follows: For each ground truth box in the frame image to be detected, calculate the geometric center displacement between it and the corresponding ground truth box in the target frame image, and use it as the motion displacement ground value of the target region of the frame to be detected. On the predicted optical flow field, the average optical flow vector of the target region of the frame to be detected is statistically analyzed; The difference between the average optical flow vector and the true value of the motion displacement is calculated, and a preset difference threshold range is used as an optical flow supervision constraint. The feature fusion module utilizes the predicted optical flow field to distort and align multi-level, multi-scale features of adjacent frame images to the multi-level, multi-scale features of the frame image to be detected, and then fuses them with the same-scale features of the frame image to be detected, generating enhanced multi-level, multi-scale features of the frame image to be detected; the distortion and alignment step specifically involves: Based on the optical flow field from the frame to be detected to the target frame, the multi-level, multi-scale features of the target frame are spatially aligned to the features of the corresponding scale of the frame to be detected through differentiable sampling transformation, generating a distorted feature map consistent with the feature space of the frame to be detected. The differentiable transformation employs bilinear interpolation to resample the feature values. The fusion step is specifically as follows: For the distorted feature maps of adjacent frames and the feature map of the frame to be detected, a spatially adaptive weight map is learned through a convolutional neural network; The weighted graph is used to perform weighted fusion of the distortion features of adjacent frame images and the same-scale features of the frame image to be detected; Based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, the target is classified and located through the target detection network, and the detection result of the frame to be detected is output. The target detection module, based on the multi-level and multi-scale features of the enhanced image of the frame to be detected, performs target classification and localization through a target detection network and outputs the detection results of the frame to be detected. The joint training module constructs a target detection loss function based on the detection results and ground truth information of the frame image to be detected, and constructs an optical flow supervision loss function based on the ground truth information of the same instance in the frame to be detected and the target frame. The total loss function is constructed by combining the optical flow supervision loss function and the target detection loss function. The backbone network, optical flow prediction network and target detection network are jointly optimized and trained end-to-end. The trained visual model is used to output the video monitoring results.

Images