A specific moving target intelligent recognition method and device based on weak supervision training

Abstract

The application discloses a specific moving target intelligent identification method and device based on weak supervision training, and the method comprises the following steps: constructing a basic network framework of specific moving target intelligent identification; obtaining a target response graph based on a dense contrast learning model, and generating a coarse segmentation mask of the corresponding target; in the training stage, the model is optimized by using a weighted cross-entropy loss based on the coarse segmentation mask guide and a density mapping loss. The device comprises a processor and a memory. By extracting the target response graph corresponding to each text description, the coarse segmentation mask of the target is generated, which is used to guide the weak supervision training based on the target frame; under the challenge environment of target movement, occlusion and appearance, the specific moving target intelligent identification model can be effectively optimized by the semantic information guide of the coarse segmentation mask.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition of specific moving targets, and more particularly to a method and apparatus for intelligent recognition of specific moving targets based on weakly supervised training. Background Technology

[0002] With the development of the internet, video has become one of the main ways people obtain information. Against this backdrop, video understanding technology is being applied more and more widely. For example, in the field of intelligent security, video understanding technology can help monitoring systems automatically identify abnormal behavior; in the field of intelligent transportation, video understanding technology can help autonomous vehicles better understand road conditions; and in the entertainment industry, video understanding technology can help game developers better design game scenes and character actions. The information contained in videos mainly manifests as the movement and behavior of targets; therefore, intelligent recognition of moving targets is a fundamental task of video understanding. Combining video with text information to achieve a more comprehensive and in-depth understanding of moving targets in videos has become a mainstream research direction.

[0003] Intelligent recognition of specific moving targets refers to the spatial segmentation of corresponding moving subjects in a video based on an input text query. Existing intelligent recognition methods for specific moving targets are all based on fully supervised learning, and their main processes include: video / text encoding, video-text correlation modeling, and segmentation / decoding. The most critical part of the intelligent recognition algorithm for specific moving targets is the video-text correlation modeling module. Therefore, based on different methods of video-text correlation modeling, intelligent recognition algorithms for specific moving targets can be divided into three main approaches: direct correlation modeling, correlation modeling based on mutual attention mechanisms, and correlation modeling based on Transformers. Methods based on direct correlation modeling extract multimodal correlations by directly adding, concatenating, or dynamically convolving video and text features. Representative algorithms include AVS. [1] CMDy [2] Simply concatenating or adding features from two different modalities cannot effectively extract their correlations. Therefore, many methods have begun to explore using various mutual attention mechanisms for cross-modal interaction, including: ACGA. [3] CSTM [4] CMPC-V [5] In recent years, Transformer-based model architectures have received widespread attention. Their powerful modeling capabilities for long-range and multimodal correlations make them well-suited for video-text correlation modeling tasks. Many studies have focused on combining Transformers with intelligent recognition tasks for specific moving targets; representative algorithms include MTTR. [6] ReferFormer [7],SgMg [8] And so on, and achieved good segmentation performance.

[0004] Although the field of intelligent recognition of specific moving targets has developed rapidly in recent years, its progress has been hampered by the fact that the annotation process is significantly more time-consuming and labor-intensive compared to other tasks. As a segmentation task, intelligent recognition of specific moving targets requires not only dense pixel-level annotation of the targets but also the extraction of motion information from each video segment for text annotation. Therefore, the annotation process is extremely cumbersome, and the annotation quality is often poor. Existing methods for intelligent recognition of specific moving targets are all based on fully supervised training, resulting in limited training data and performance highly dependent on the quality of the training annotations. Therefore, exploring effective weakly supervised methods for intelligent recognition of specific moving targets is essential. Currently, there are some weakly supervised training-based methods in image segmentation, such as BoxInstance. [9] BoxLevelSet

[10] BoxTeacher

[11] Image-based weak supervision methods often use bounding boxes as supervision. However, most image-based weak supervision methods rely on color and contour information for segmentation optimization. In videos, color and contour information often fail to provide a clear contrast with the background due to rapid motion and occlusion. Therefore, image-based weak supervision methods cannot be directly used to optimize video weak supervision segmentation algorithms. Summary of the Invention

[0005] This invention provides a method and apparatus for intelligent recognition of specific moving targets based on weakly supervised training. The invention generates a coarse segmentation mask for the target by extracting the target response map corresponding to each text description, which guides weakly supervised training based on the target bounding box. Under challenging environments such as target motion, occlusion, and appearance, this invention can effectively optimize the intelligent recognition model for specific moving targets by guiding the semantic information of the coarse segmentation mask. Details are described below:

[0006] A method for intelligent recognition of specific moving targets based on weakly supervised training, the method comprising:

[0007] Constructing a basic network framework for intelligent recognition of specific moving targets;

[0008] The target response map is obtained based on the dense contrastive learning model, and a coarse segmentation mask for the corresponding target is generated.

[0009] During the training phase, the model is optimized using weighted cross-entropy loss and density mapping loss guided by coarse segmentation masks.

[0010] Specifically, the step of obtaining the target response map based on the dense contrastive learning model and generating a coarse segmentation mask for the corresponding target involves:

[0011] The visual features of each frame are obtained by DenseCL. The feature vectors of the corresponding visual features inside the target box are extracted and cosine similarity is calculated with the original visual features to obtain multiple target response maps.

[0012] Then, the final target response map is selected by performing intersection and union comparison with the target bounding box; finally, for multiple targets in each frame, a coarse segmentation mask for each target is generated by comparing the response values ​​between multiple response maps and binarizing them.

[0013] Specifically, the optimization of the model using weighted cross-entropy loss and density mapping loss guided by coarse segmentation mask is as follows:

[0014] The obtained coarse segmentation mask is used as a guide, and the target's response map is used as the weight by the value of a nonlinear function, and the training is performed using a weighted cross-entropy function.

[0015] Based on the projection loss function, a mapping density loss term is added, which, together with the weighted cross-entropy, is used to jointly optimize the model.

[0016] The coarse segmentation mask is:

[0017]

[0018]

[0019] in, This represents the target response map generated by the k-th bounding box in the current frame. This represents the target response map generated by the nth bounding box in the current frame, where the point in the target response map has an x-coordinate and a y-coordinate. Let τ represent the set of integers from 1 to 1-K excluding k. Let τ be the threshold for judging the background ...

[0020] Furthermore, the weighted cross-entropy loss is:

[0021] The weighted graph generated by the weighting function is used as a guide to supervise the output segmentation graph, i.e.:

[0022]

[0023] Where M(x,y) is the generated coarse segmentation mask, P s (x,y) is the target segmentation prediction map output by the algorithm;

[0024]

[0025] Among them, H k W represents the weight map generated for the k-th target in the current frame. k This represents the target response map of the k-th target in the current frame, where Δ is a pre-set relaxation value.

[0026] Furthermore, the density mapping loss is:

[0027] L proj =Dice(max) x (B),max x (P))+Dice(max y (B),max y (P))+Dice(mean x (H),mean x (P))+Dice(mean y (H),mean y (P))

[0028] Where Dice(·) represents the Dice loss function, max x (·) and max y (·) represents the maximum value of the binary graph in the horizontal and vertical directions, mean. x (·) and mean y (·) represents the density values ​​of the binary image in the horizontal and vertical directions, H is the weighted image generated after binarization, P is the target segmentation prediction image output by the algorithm, and B is the binary image with the inside region of the target box as the foreground.

[0029] Specifically, the optimization of the model using weighted cross-entropy loss and density mapping loss guided by coarse segmentation mask is as follows:

[0030] The loss function for segmentation prediction is expressed as:

[0031] L seg =L wce +L proj

[0032] The final loss function during the model training phase is then expressed as:

[0033] L=αL cls +βL bbox +γL seg

[0034] Here, α, β, and γ are hyperparameters used to balance the weights among different types of loss functions.

[0035] A specific moving target intelligent recognition device based on weakly supervised training, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to execute any of the methods described above.

[0036] A computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform any of the methods described above.

[0037] The beneficial effects of the technical solution provided by this invention are:

[0038] 1. This invention uses self-supervised pre-trained DenseCL as a semantic feature extractor to extract the visual features of the current frame, obtain the target response map at multiple positions inside the target box through cosine similarity, and filter out the final target response map according to the intersection-union ratio with the target box, thereby obtaining a coarse segmentation mask; therefore, the coarse segmentation mask contains not only target color similarity information, but also target deep semantic similarity information.

[0039] 2. This invention uses SgMg as the benchmark intelligent recognition network for specific moving targets. By designing a weighted cross-entropy loss function and density mapping loss guided by a coarse segmentation mask, the network is trained. This can simultaneously incorporate the color and semantic information of the target to refine the segmentation prediction map. Through benchmark database testing, the weakly supervised training method proposed in this invention can effectively improve the model's segmentation performance, and the performance degradation is not significant compared with the fully supervised algorithm. Attached Figure Description

[0040] Figure 1 A flowchart of a method for intelligent recognition of specific moving targets based on weakly supervised training;

[0041] Figure 2 This is a flowchart of the coarse segmentation mask generation method;

[0042] Figure 3 The flowchart shows a weakly supervised training algorithm based on the SgMg benchmark network.

[0043] Figure 4 Image of the weight generation function used in the weighted cross-entropy loss guided by the coarse segmentation mask. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0045] Example 1

[0046] This invention provides a method for intelligent recognition of specific moving targets based on weakly supervised training. See [link to relevant documentation]. Figure 1 The method includes the following steps:

[0047] 101: Constructing a basic network framework for intelligent recognition of specific moving targets;

[0048] First, a spectrum-guided multi-granularity (SgMg) model is constructed. [8] This is a baseline intelligent network for recognizing specific moving targets. The network first encodes the input video frames and text queries using visual and text encoders, respectively, outputting corresponding visual and text features. Then, it uses a deformable detection-based Transformer (Deformable DETR) model.

[12] The network framework is used for multimodal correlation modeling. The output employs three different decoders (referred to as the target bounding box localization head, classification head, and segmentation head) to predict the spatial location, category, and corresponding fine-grained segmentation map of the target in the input video.

[0049] The SgMg network structure and the Deformable DETR module are well known to those skilled in the art, and will not be described in detail in the embodiments of the present invention.

[0050] 102: Obtain the target response map based on the Dense Contrastive Learning (DenseCL) model and generate a coarse segmentation mask for the corresponding target;

[0051] First, visual features for each frame are acquired using DenseCL. Feature vectors corresponding to the visual features within the target bounding box are extracted, and cosine similarity is calculated between these vectors and the original visual features to obtain multiple target response maps. Then, the final target response map is selected by performing an intersection-union comparison (IUCN) with the target bounding box. Finally, for multiple targets in each frame, a coarse segmentation mask for each target is generated by comparing and binarizing the response values ​​among the multiple response maps.

[0052] 103: During the training phase, the model is optimized using weighted cross-entropy loss and density mapping loss guided by coarse segmentation masks.

[0053] In this embodiment of the invention, the training process of the SgMg baseline network is redesigned. Due to the lack of fine-grained ground truth maps for target segmentation, this embodiment uses the obtained coarse segmentation mask as guidance and trains the network using a weighted cross-entropy function with the target's response map as weights and the values ​​of a nonlinear function. Simultaneously, in BoxInst... [9]Based on the projection loss function, a mapping density loss term is added, which, together with the weighted cross-entropy, is used to jointly optimize the model.

[0054] In summary, this invention presents a method for intelligent recognition of specific moving targets based on weakly supervised training, designed through steps 101 to 103. Based on the original SgMg network, the training of the segmentation branch is redesigned based on weak supervision of the target bounding box. By performing a cosine similarity operation on the visual features extracted by DenseCL, multiple target response maps are extracted for each frame, and a coarse segmentation mask for each target is obtained. Using the coarse segmentation mask as a guide, a weighted cross-entropy loss and a density mapping loss are designed to achieve weakly supervised training of the model. Since the coarse segmentation mask is generated based on the color and semantic information of the target in the current frame, it can provide effective prior knowledge for weakly supervised training. This invention extends the SgMg network in the weakly supervised domain, enabling it to cope with network training under conditions where dense segmentation annotations in the dataset are insufficient.

[0055] Example 2

[0056] The solution in Example 1 will be further described below:

[0057] 201: Target coarse segmentation mask generation;

[0058] Currently, there are many researches on weakly supervised segmentation algorithms based on bounding boxes, but most of these methods are for image segmentation. Compared to images, the biggest characteristic of video is that it contains motion information. In an input video, there may be multiple moving targets, which may appear, disappear, or be occluded by other targets. Furthermore, due to the motion of the targets, their boundaries may be blurred. These characteristics of targets in video make it impossible to directly train models using traditional weakly supervised methods, such as projection loss and color similarity loss. Therefore, it is necessary to design a method to extract coarse contour information of targets, and simultaneously perform coarse localization of targets based on color and semantic information within the bounding box, using this as coarse supervision to guide refined segmentation of the targets.

[0059] DenseCL is a benchmark model trained using a self-supervised training method. Because it is pre-trained using the similarity between color and deep features, it is very sensitive to regions with similar color and semantics. Therefore, visual information extraction based on DenseCL can effectively mine the color and semantic information of targets in visual features.

[0060] Based on the above analysis, the algorithm proposed in this embodiment of the invention uses DenseCL to generate a coarse segmentation mask for subsequent model training.

[0061] The process of generating a coarse segmentation mask based on DenseCL in the algorithm of this invention embodiment is as follows: Figure 2 As shown.

[0062] First, the DenseCL baseline model is pre-trained using a dense contrastive self-supervised learning method. In this embodiment of the invention, the DenseCL baseline model uses a 101-layer residual network as its main structure to encode the input video frames. This process can be represented as follows:

[0063] {F,F + ,F -} = DenseCL({I,I + ,I -})

[0064] Where I represents the current video frame, I + For the current video frame after data augmentation, I - This is another video frame that is different from the current frame. F, F + With F - These are the visual features of these three types of input video frames after DenseCL encoding.

[0065] Then, based on the above three types of output features, DenseCL is trained under self-supervised supervision. Because F... + Both F and F contain semantic information of the current video frame, while F - Since the current video frame does not contain semantic information, dense contrastive loss is used for self-supervised training, and its loss function can be expressed as:

[0066]

[0067] Where H and W represent the height and width of the output current frame features, f i f i + with f i - Let F and F represent visual features respectively. + With F - In this model, τ represents a temperature hyperparameter. By optimizing this loss function, the distance between features containing the same semantic content continuously decreases, while the distance between features containing different semantic content gradually increases. Therefore, the semantic similarity of targets in video frames can be effectively extracted.

[0068] In addition, DenseCL also introduces a global contrast loss, which, in conjunction with the dense contrast loss, is used to extract the global semantic similarity between frames, and can be expressed as:

[0069]

[0070] Among them, f, f + with f - Representing visual features F, F + With F - The result after averaging across spatial dimensions.

[0071] Therefore, the final DenseCL contrastive training loss is expressed as:

[0072] L contra =L dense +L global

[0073] After pre-training the DenseCL benchmark network, its model parameters are fixed, and visual features are extracted for each frame. The location of the target bounding box is found within the generated visual features, and the feature vector inside the target bounding box is used as the text query.

[0074]

[0075] Where, x bn With y bn Let q represent the x and y coordinates of the nth spatial location within the target bounding box, and B be the set of points within the target bounding box region. n Let be the feature vector of the nth spatial location.

[0076] Since the location of the target within the bounding box is unknown, cosine similarity is calculated between the feature vectors of all locations and the visual features to extract the target response map for each spatial location, and then filtered.

[0077]

[0078]

[0079] Among them, W bn The target response map represents the nth spatial location, IoU(·) represents the cross-union ratio between two binary masks, and W bn The final generated bounding box response map is represented by , where B represents a binary map with the area inside the bounding box as the foreground. This operation generates a corresponding bounding box response map for each bounding box in the input video frame.

[0080] Finally, the final coarse segmentation mask is generated based on the target response map, namely:

[0081]

[0082]

[0083] in, This represents the target response map generated by the k-th bounding box in the current frame. This represents the target response map generated by the nth bounding box in the current frame (where n≠k), where the point in the target response map has an x-coordinate and a y-coordinate. Let τ represent the set of integers from 1 to 1-K excluding k. Let τ be the threshold for judging background. For a point at position (x, y) in the k-th response map, if its response value is greater than the threshold τ and greater than the response values ​​at that position in all other response maps, the point is considered to be a foreground response of the k-th target and is assigned a foreground value of 1. Otherwise, the point is considered to be background and is assigned a background value of 0.

[0084] 202: Constructing an intelligent recognition network for specific moving targets in SgMg;

[0085] The intelligent recognition network process for specific moving targets in the algorithm of this invention embodiment is as follows: Figure 2 As shown.

[0086] The SgMg network uses a Deformable DETR as its backbone network. It encodes different modal inputs through two encoders and then outputs the final target localization and segmentation results through the Deformable DETR.

[0087] {P s ,P b ,P c}=Dec(q+F t ,Enc(Fuse(F v ,F t )))

[0088] Among them, P s P b and P c These represent segmentation prediction, bounding box prediction, and class prediction, respectively, where q is the random query input to the decoder, and F... v With F t These are the encoded visual and textual features, respectively. Fuse(•) is the multimodal pre-fusion process (using mutual attention mechanism), and Enc(•) and Dec(•) are the encoder and decoder of Deformable DETR, respectively.

[0089] In this embodiment of the invention, a weak supervision method based on target boxes is used. Therefore, for the network output P b Supervision is performed directly using the ground truth of the target bounding boxes, and the category output is P. c In this embodiment of the invention, the confidence level representing the existence of the target is always 1. For the segmentation output P... s The algorithm proposed in this embodiment of the invention is used for optimization.

[0090] 203: Weighted cross-entropy loss and density mapping loss guided by coarse segmentation mask;

[0091] The generated coarse segmentation mask is used as a guide, and cross-entropy loss is used for direct supervision. Since the coarse segmentation mask is inaccurate in extracting the target region, only revealing its approximate location and contour, a weight function is designed to assign a weight to each spatial location of the coarse segmentation mask. In this embodiment, the weight function is formally a piecewise function, namely:

[0092]

[0093] Among them, H k W represents the weight map generated for the k-th target in the current frame. k This represents the target response map of the k-th target in the current frame, where Δ is a pre-set relaxation value.

[0094] The graph of the weighting function is as follows Figure 4 As shown. Based on the weight function graph, the weight function has the following properties:

[0095] When the target response is greater than τ+Δ, the probability of considering the region as a foreground target is 1; when the target response is less than τ-Δ, the probability of considering the target as background is 1.

[0096] When the target response is between τ-Δ and τ+Δ, it is difficult to determine whether it is foreground or background. Therefore, the farther the target response is from the judgment threshold, the greater its weight.

[0097] At the threshold τ (i.e., the target edge), the weight H k It is very small because the edges of the coarse mask are very rough.

[0098] Finally, the weight graph generated by this weight function is used as a guide to supervise the output segmentation graph, that is:

[0099]

[0100] Where M(x,y) is the coarse segmentation mask generated in the embodiments of the present invention, and P s (x,y) is the target segmentation prediction map output by the algorithm.

[0101] In addition to supervision based on weighted cross-entropy, it is also necessary to make full use of the target boundary information provided by the bounding box. Since the four boundaries of the bounding box must overlap with the target segmentation mask, this embodiment of the invention borrows the projection loss function from BoxInst to restrict the spatial position of the target segmentation map. However, using only the projection loss can only obtain the spatial range of the target, but cannot reflect the density of the target's projection at each spatial position.

[0102] Therefore, this embodiment of the invention designs a density mapping loss based on the generated coarse segmentation mask. The final target box loss is the sum of the projection loss and the density mapping loss, that is:

[0103] L proj =Dice(max) x (B),max x (P))+Dice(max y (B),max y (P))+Dice(mean x (H),mean x (P))+Dice(mean y (H),mean y (P))

[0104] Where Dice(·) represents the Dice loss function, max x (·) and max y (·) represents the maximum value of the binary graph in the horizontal and vertical directions, mean. x (·) and mean y (·) represents the density values ​​of the binary image in the horizontal and vertical directions, H is the weighted image generated by the embodiment of the present invention after binarization, P is the target segmentation prediction image output by the algorithm, and B is the binary image with the inner region of the target box as the foreground.

[0105] 204: Model training and testing;

[0106] Based on the weakly supervised segmentation training strategy described above, the network is first trained. The SgMg baseline network outputs the target class P at the decoding end. c Target location box P b fine-grained segmentation mask P of the target s There are three different predictions, so they are supervised separately during the training phase.

[0107] For category prediction P c Since the dataset for this task often does not contain precise category labels for the targets, this embodiment of the invention defines category 0 as no target in the target bounding box and category 1 as a target in the bounding box, and uses negative log-likelihood loss to supervise the category prediction, that is:

[0108] L cls =-log(P c )

[0109] For target bounding box prediction P b It is represented as a four-dimensional vector, namely P. b= [x1, y1, x2, y2], where (x1, y1) and (x2, y2) are the coordinates of the top-left and bottom-right corners of the bounding box, respectively. Because bounding boxes are used for supervision, the dataset contains accurate bounding box annotations. The sum of the intersection-union loss and the generalized intersection-union loss is then used as the target box loss function, expressed as:

[0110]

[0111] Where IoU is the intersection-union ratio between the predicted bounding box and the labeled bounding box, i.e.:

[0112]

[0113] Where ∩ and ∪ represent the intersection and union of the areas within the boxes.

[0114] GIoU represents the generalized crossover ratio between the predicted bounding box and the labeled bounding box, i.e.:

[0115]

[0116] Where C represents the inclusion of P b and The minimum convex set.

[0117] For densely segmented mask prediction P s Since the dataset does not contain the ground truth of the segmentation mask, the weakly supervised loss proposed in this embodiment of the invention is used for supervision. The loss function for segmentation prediction is then expressed as:

[0118] L seg =L wce +L proj

[0119] The final loss function during the model training phase can then be expressed as:

[0120] L=αL cls +βL bbox +γL seg

[0121] Here, α, β, and γ are hyperparameters used to balance the weights among different types of loss functions.

[0122] During the testing phase, since the specific moving target intelligent recognition task only requires segmentation of the target in the video, only the dense segmentation mask prediction decoder in the model is enabled, and the output segmentation prediction P is used. s The result of binarization is used as the final recognition result.

[0123] Example 3

[0124] The effectiveness of the schemes in Examples 1 and 2 is evaluated below based on specific experimental data, as detailed in the following description:

[0125] 301: Data Composition;

[0126] The test set consists of all video sequences from the A2D Sentences dataset and the J-HMDB Sentences dataset. The A2D Sentences dataset includes 3782 video sequences and 6655 text queries; the J-HMDB Sentences dataset includes 928 video sequences and 21 action categories.

[0127] 302: Evaluation Criteria;

[0128] Three evaluation metrics from the segmentation task evaluation system were used on both datasets: Overall IoU, Mean IoU, and Precision@K (P@K). Overall IoU calculates the ratio of the cumulative intersection area to the cumulative union area between the predicted and actual values ​​for all test samples, while Mean IoU calculates the average IoU for all test samples. Precision@K calculates the percentage of test samples with an IoU higher than a predefined threshold K, where K is selected from [0.5, 0.6, 0.7, 0.8, 0.9]. Furthermore, the average precision (AP) on the [0.5:0.05:0.95] cross-section was also calculated.

[0129] In the segmentation task, IoU is calculated as the ratio of the overlapping area between different segmentation maps to the total area, i.e.:

[0130]

[0131] Among them, P s Let G represent the segmentation prediction graph, and G represent the segmentation ground truth graph.

[0132] 303: Comparison algorithm;

[0133] In the comparative experiment, the embodiments of the present invention were compared with six mainstream algorithms on the A2D Sentences test set and the J-HMDB Sentences dataset, including an algorithm based on dynamic convolution AVS[1], four algorithms based on attention mechanism ACGA[3], CSTM[4], CMPC-V[5], and ClawCrane

[13] , and an algorithm based on Transformer SgMg[8]. It is noted that existing intelligent recognition algorithms for specific moving targets are all based on fully supervised training. The method proposed in the embodiments of the present invention is the first algorithm to introduce weakly supervised training based on target boxes into the field of intelligent recognition of specific moving targets.

[0134] Tables 1 and 2 show the objective evaluation results of our method and related comparison algorithms on the A2D Sentences dataset and the J-HMDBSentences dataset, respectively. As shown in Table 1, on the A2D Sentences dataset, the weakly supervised algorithm proposed in this embodiment can even outperform some fully supervised algorithms (ACGA and AVS). Furthermore, this embodiment also notes that compared to the latest fully supervised algorithms, the weakly supervised algorithm proposed in this embodiment can achieve over 90% of the performance of fully supervised algorithms on most performance metrics, proving the effectiveness of the algorithm proposed in this embodiment. In addition, it can be found that for evaluation metrics with high IoU threshold requirements (such as P@0.9), the performance of the weakly supervised algorithm decreases significantly, while for metrics with low IoU threshold requirements (such as P@0.5), the performance decreases less. This reflects that the weakly supervised algorithm proposed in this embodiment can optimize the output predicted contour information to achieve an overlap rate of over 50% with the ground truth segmentation, but it cannot achieve the same refined segmentation effect as the fully supervised algorithm. This is determined by the blurring and occlusion characteristics of the target in the video dataset. As shown in Table 2, the weakly supervised algorithm proposed in this embodiment of the invention achieves good objective performance on the J-HMDB Sentences dataset, surpassing many fully supervised methods. Furthermore, unlike the A2D Sentences dataset, the ground truth annotations in the J-HMDB Sentences dataset are inherently transferred from the human backbone task, resulting in inaccurate segmentation annotations. Therefore, fully supervised algorithms cannot achieve high performance on evaluation metrics with high IoU threshold requirements. Thus, the weakly supervised algorithm proposed in this embodiment of the invention can achieve performance levels comparable to fully supervised algorithms.

[0135] Experiments on two databases have verified that the algorithm proposed in this invention can achieve performance comparable to, and even surpass, fully supervised intelligent moving target recognition algorithms under weak supervision, demonstrating that the algorithm proposed in this invention can effectively solve the weak supervision problem in intelligent moving target recognition tasks. Guided by a pre-generated coarse target segmentation mask at the color and semantic levels, the weak supervision method proposed in this invention can still perform good model optimization even in challenging scenarios such as blurred or occluded targets in videos.

[0136] Table 1. Performance comparison of the algorithm proposed in this invention on the A2D Sentences dataset.

[0137]

[0138]

[0139] Table 2 shows the performance comparison of the algorithm proposed in this invention on the J-HMDB Sentences dataset.

[0140]

[0141] Example 4

[0142] A specific moving target intelligent recognition device based on weakly supervised training, the device includes: a processor and a memory, the memory storing program instructions, and the processor calling the program instructions stored in the memory to cause the device to execute the following method steps in Embodiment 1:

[0143] Constructing a basic network framework for intelligent recognition of specific moving targets;

[0144] The target response map is obtained based on the dense contrastive learning model, and a coarse segmentation mask for the corresponding target is generated.

[0145] During the training phase, the model is optimized using weighted cross-entropy loss and density mapping loss guided by coarse segmentation masks.

[0146] Specifically, the process of obtaining the target response map based on a dense contrastive learning model and generating a coarse segmentation mask for the corresponding target involves:

[0147] The visual features of each frame are obtained by DenseCL. The feature vectors of the corresponding visual features inside the target box are extracted and cosine similarity is calculated with the original visual features to obtain multiple target response maps.

[0148] Then, the final target response map is selected by performing intersection and union comparison with the target bounding box; finally, for multiple targets in each frame, a coarse segmentation mask for each target is generated by comparing the response values ​​between multiple response maps and binarizing them.

[0149] Specifically, the optimization of the model using weighted cross-entropy loss and density mapping loss guided by coarse segmentation masks is as follows:

[0150] The obtained coarse segmentation mask is used as a guide, and the target's response map is used as the weight by the value of a nonlinear function, and the training is performed using a weighted cross-entropy function.

[0151] Based on the projection loss function, a mapping density loss term is added, which, together with the weighted cross-entropy, is used to jointly optimize the model.

[0152] The coarse segmentation mask is as follows:

[0153]

[0154]

[0155] in, This represents the target response map generated by the k-th bounding box in the current frame. This represents the target response map generated by the nth bounding box in the current frame, where the point in the target response map has an x-coordinate and a y-coordinate. Let τ represent the set of integers from 1 to 1-K excluding k. Let τ be the threshold for judging the background ...

[0156] Furthermore, the weighted cross-entropy loss is:

[0157] The weighted graph generated by the weighting function is used as a guide to supervise the output segmentation graph, i.e.:

[0158]

[0159] Where M(x,y) is the generated coarse segmentation mask, P s (x,y) is the target segmentation prediction map output by the algorithm;

[0160]

[0161] Among them, H k W represents the weight map generated for the k-th target in the current frame. k This represents the target response map of the k-th target in the current frame, where Δ is a pre-set relaxation value.

[0162] Furthermore, the density mapping loss is:

[0163] L proj =Dice(max) x (B),max x (P))+Dice(max y (B),max y (P))+Dice(mean x (H),mean x (P))+Dice(mean y (H),mean y (P))

[0164] Where Dice(·) represents the Dice loss function, max x (·) and max y (·) represents the maximum value of the binary graph in the horizontal and vertical directions, mean. x (·) and mean y(·) represents the density values ​​of the binary image in the horizontal and vertical directions, H is the weighted image generated after binarization, P is the target segmentation prediction image output by the algorithm, and B is the binary image with the inside region of the target box as the foreground.

[0165] Specifically, the optimization of the model using weighted cross-entropy loss and density mapping loss guided by coarse segmentation masks is as follows:

[0166] The loss function for segmentation prediction is expressed as:

[0167] L seg =L wce +L proj

[0168] The final loss function during the model training phase is then expressed as:

[0169] L=αL cls +βL bbox +γL seg

[0170] Here, α, β, and γ are hyperparameters used to balance the weights among different types of loss functions.

[0171] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0172] The execution entities of the processor 1 and memory 2 mentioned above can be devices with computing functions such as computers, microcontrollers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution entity and can select it according to the needs of actual application.

[0173] Data signals are transmitted between memory 2 and processor 1 via bus 3, which will not be described in detail in this embodiment of the invention.

[0174] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0175] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0176] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0177] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0178] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0179] References

[0180] [1]Gavrilyuk Kirill,Ghodrati Amir,Li Zhenyang,et al.Actor and actionvideo segmentation from a sentence[C] / / IEEE Conference on Computer Vision andPattern Recognition,2018:5958-5966.

[0181] [2]Wang Hao,Deng Cheng,Ma Fan,et al.Context Modulated DynamicNetworks for Actor andAction Video Segmentation with Language Queries[C] / / AAAI Conference on ArtificialIntelligence,2020:12152-12159.

[0182] [3]Wang Hao, Deng Cheng, Yan Junchi, et al. Asymmetric Cross-GuidedAttention Network for Actor and Action Video Segmentation From NaturalLanguage Query[C] / / International Conference on Computer Vision, 2019:3938-3947.

[0183] [4]Hui Tianrui,Huang Shaofei,Ding Zihan,et al.Collaborative Spatial-Temporal Modeling forLanguage-Queried Video Actor Segmentation[C] / / IEEEConference on Computer Vision andPattern Recognition,2021:4187-4196.

[0184] [5]Liu Si,Hui Tianrui,Huang Shaofei,et al.Cross-Modal ProgressiveComprehension forReferring Segmentation[J].IEEE Transactions on PatternAnalysis and Machine Intelligence,2022,44(9):4761-4775.

[0185] [6]Botach Adam,Zheltonozhskii Evgenii,Baskin Chaim.End-to-EndReferring Video ObjectSegmentation with Multimodal Transformers[C] / / IEEEConference on Computer Vision andPattern Recognition,2022:4975-4985.

[0186] [7]Wu Jiannan,Jiang Yi,Sun Peize,et al.Language as Queries forReferring Video ObjectSegmentation[C] / / IEEE Conference on Computer Vision andPattern Recognition,2022:4964-4974.

[0187] [8]Miao Bo,Bennamoun Mohammed,Gao Yongsheng,et al.Spectrum-guidedMulti-granularityReferring Video Object Segmentation[C] / / InternationalConference on Computer Vision,2023.[9]Tian Zhi,Shen Chunhua,Wang Xinlong,etal.BoxInst:High-performance instancesegmentation with box annotations[C] / / IEEE Conference on Computer Vision and PatternRecognition,2021:5443-5452.

[0188]

[10] Li Wentong,Liu Wenyu,Zhu Jianke,et al.Box-supervised instancesegmentation with levelset evolution[C] / / European Conference on ComputerVision,2022:1-18.

[0189]

[11] Chen Tianheng,Wang Xinggang,Chen Shaoyu,et al.BoxTeacher:Exploring high-qualitypseudo labels for weakly supervised instancesegmentation[C] / / IEEE Conference on ComputerVision and Pattern Recognition,2023.

[0190]

[12] Zhu Xizhou,Su weijie,Lu Lewei,et al.Deformable DETR:DeformableTransformers forEnd-to-End Object Detection[C] / / International Conference onLearning Representations,2021.

[13] Chen Liang,Wu Yu,Luo Yawei,etal.ClawCraneNet:Leveraging Object-level Relation for Text-based VideoSegmentation[J].CoRR,2021,abs / 2103.10702.

[0191] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0192] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent recognition of specific moving targets based on weakly supervised training, characterized in that, The method includes: Constructing a basic network framework for intelligent recognition of specific moving targets; The target response map is obtained based on the dense contrastive learning model, and a coarse segmentation mask for the corresponding target is generated. During the training phase, the model is optimized using weighted cross-entropy loss and density mapping loss guided by coarse segmentation mask. The specific steps for obtaining the target response map based on the dense contrastive learning model and generating the corresponding coarse segmentation mask for the target are as follows: The visual features of each frame are obtained by DenseCL. The feature vectors of the corresponding visual features inside the target box are extracted and cosine similarity is calculated with the original visual features to obtain multiple target response maps. Then, the final target response map is selected by performing intersection and union comparison with the target bounding box; finally, for multiple targets in each frame, a coarse segmentation mask for each target is generated by comparing the response values ​​between multiple response maps and binarizing them. The weighted cross-entropy loss is: The weighted graph generated by the weighting function is used as a guide to supervise the output segmentation graph, i.e.: in, For the generated coarse segmentation mask, The target segmentation prediction map output by the algorithm; in, Indicates the first in the current frame Weighted graphs generated for each target Indicates the first in the current frame Target response diagram for each target The preset relaxation value; The density mapping loss is: in, Represents the Dice loss function. and This represents the maximum value of the binary graph in the horizontal and vertical directions. and This represents the density values ​​in the horizontal and vertical directions of the binary image. This is the weight map generated after binarization. The target segmentation prediction map output by the algorithm. This is a binary image with the area inside the target bounding box as the foreground.

2. The method for intelligent recognition of specific moving targets based on weakly supervised training according to claim 1, characterized in that, The optimization of the model using weighted cross-entropy loss and density mapping loss guided by coarse segmentation mask is specifically as follows: The obtained coarse segmentation mask is used as a guide, and the target's response map is used as the weight by the value of a nonlinear function, and the training is performed using a weighted cross-entropy function. Based on the projection loss function, a mapping density loss term is added, which, together with the weighted cross-entropy, is used to jointly optimize the model.

3. The method for intelligent recognition of specific moving targets based on weakly supervised training according to claim 1, characterized in that, The coarse segmentation mask is: in, Indicates the current frame number The target response map generated by each box. Indicates the current frame number The target response map generated by the box has the horizontal axis as... The vertical axis is point, Indicates from 1 to And does not include A set of integers, The threshold for determining the background represents the threshold for the first... In the response diagram The point at the location, when its response value is greater than the judgment threshold. If the response value at that location is greater than the response value at that location in all other response maps, then that point is considered to belong to the [number]th [location]. For each target's foreground response, assign a foreground value of 1; otherwise, consider the location as background and assign a background value of 0.

4. The method for intelligent recognition of specific moving targets based on weakly supervised training according to claim 1, characterized in that, The optimization of the model using weighted cross-entropy loss and density mapping loss guided by coarse segmentation mask is specifically as follows: The loss function for segmentation prediction is expressed as: The final loss function during the model training phase is then expressed as: in, , and These are hyperparameters used to balance the weights among different types of loss functions.

5. A specific moving target intelligent recognition device based on weakly supervised training, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-4.

Images