Semi-automatic labeling method and system for rail transit engineering construction video images

Abstract

The invention discloses a semi-automatic labeling method and system for rail transit engineering construction video images, and the method comprises the steps: removing redundant frames from a video stream, extracting key frames, and forming a block index set; and driving the multi-modal large model to pre-annotate the image by using the constructed cue word, and carrying out binarization processing on an annotation result. Based on active learning, migrating unmarked samples to a marked set for multiple times and training a key sample classifier CD until the scale of the marked set reaches the standard; meanwhile, a confidence classifier CC is trained by comparing manual and pre-labeling results. And for residual samples in the unmarked set, after the residual samples reach the standard through a CC precision test, marking tasks are divided according to a threshold value theta: high-confidence samples are pre-marked, and low-confidence samples are manually marked. And finally, updating the set of the manually labeled samples, and determining whether to retrain the model or not according to category distribution. On the premise of ensuring the labeling quality, the blindness of manual labeling is effectively reduced, and the efficiency of the labeling process is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and pattern recognition, and more particularly relates to a semi-automatic labeling method and system for rail transit engineering construction video images. BACKGROUND

[0002] In the field of urban rail transit engineering construction, with the wide application of video monitoring technology, the labeling demand of massive video image data is growing. The traditional manual labeling method faces significant efficiency bottlenecks and cost pressures: on the one hand, the engineering construction scene is complex and diverse, containing personnel operation, equipment operation, environmental change and other multiple contents, and manual frame-by-frame labeling requires a lot of manpower and time, especially for long-period and large-scale construction projects, the labeling efficiency is difficult to meet the actual demand; on the other hand, the consistency and accuracy of manual labeling are easily affected by subjective factors, and for subtle features in complex scenes (such as safety hazard identification, equipment component state judgment, etc.), the labeling difficulty is large, and the quality is difficult to guarantee.

[0003] At the same time, the existing semi-automatic labeling technology has obvious limitations in the application of engineering construction field. Some labeling methods based on single modality are difficult to fully capture the complex information in video images, and the representation ability of multi-dimensional scene features is insufficient; the traditional active learning strategy lacks targeted optimization for the special nature of the engineering construction scene in sample screening, and is easy to fall into the problem of uneven class distribution, leading to model training bias. In addition, the key information in the engineering construction video is often distributed in specific frames, and the traditional method lacks an effective key frame extraction mechanism, resulting in redundant labeling workload and further reducing the labeling efficiency.

[0004] With the development of multi-modal large model technology, it shows advantages in cross-modal information understanding and feature extraction, but when directly applied to engineering construction video image labeling, it still faces the problem of field adaptability: the professional terms, scene features in the engineering construction field are different from the training data of general multi-modal models, and the accuracy of pre-labeling results needs to be optimized combined with domain knowledge. How to combine the pre-labeling ability of multi-modal large models with the particularity of engineering construction scenes and construct an efficient semi-automatic labeling process has become a key problem to improve the efficiency of rail transit engineering video image analysis SUMMARY

[0005] The present application aims to provide a semi-automatic labeling method and system for urban rail transit engineering construction video images. By combining multi-modal large model pre-labeling and active learning strategy, high-value samples are screened using classifier CD and confidence CC, reducing the workload of manual labeling and improving the labeling efficiency. At the same time, through key frame extraction, dynamic model optimization and other mechanisms, the labeling quality is guaranteed, solving the problems of low efficiency and high cost of traditional manual labeling and the lack of adaptability of existing semi-automatic technology, and the method is suitable for intelligent analysis of engineering construction video images.

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a semi-automatic labeling method for rail transit engineering construction video images, comprising:

[0007] S1. Extracting video key frames from continuous video streams by removing redundant frames with similar content to obtain a block index set VCI;

[0008] S2. Constructing prompt words of a multi-modal large model, and putting all images in the block index set VCI into the constructed prompt words respectively to obtain a result set Rma after pre-labeling of the large model, and performing binaryzation processing on all elements ma in Rma; i

[0009] S3. Based on the active learning strategy, selecting sample elements in the unlabeled sample set U to migrate to the labeled sample set L, retraining the classifier after each migration until the size of the labeled set L reaches the standard to end the process; training the confidence classifier CC by comparing the consistency of the manual labeling result and the multi-modal large model pre-labeling result of the labeled sample;

[0010] S4. Using the classifier CD to screen top-K negative samples from the unlabeled set U to form a subset S; testing the accuracy of the confidence classifier CC on each hazard category, scoring the samples in S after reaching the standard, combining the set threshold θ, using the pre-labeling result for samples with CC(x)≥θ; manually labeling samples with CC(x)<θ and putting them into the manual labeling set M; moving the samples in the set M into the labeled set L, and removing the corresponding samples from the unlabeled sample set U; and determining whether to retrain CD and CC according to the category distribution.

[0011] Further, the element ma in S2 i Specifically,

[0012] ma i =[r1,r2,...,r p ] is the image ci in the set CI i MLLM output result on all hazard categories, and p is the number of hazard categories to be identified;

[0013] ​The set CI E VCI, CI = {ci1, ci2,..., ci m+1}, the CI element ci i is extracted in S1 by eliminating similar redundant frames.

[0014] Further, the specific method for selecting sample elements in the selected unlabeled sample set U in S3 is:

[0015] A small number of samples in the set VCI are randomly selected for manual labeling to form a set L, and other unlabeled samples form a set U. The total amount of query labeling K and the total number of queries n are set, and the sample amount of each migration is K / n:

[0016] A multi-layer neural network structure is used as the basic framework of the classifier CD.

[0017] U is used as a negative sample, and L is used as a positive sample to train CD. The model distinguishes "whether it is a key labeled sample" by learning sample features. The "negative sample" required is K / n.

[0018] Further, the specific training method of the confidence classifier CC in S3 is:

[0019] When the size of the labeled sample set L reaches the standard, for each sample x E L, compare its manual labeling result y manual (x) with the pre-labeling result y model (x) of the multi-modal large model.

[0020] The indicator function δ(x) is defined as:

[0021]

[0022] where y manual (x) and y model (x) are both in vector form, representing the labeling results of sample x on each hazard category; the equation y manual (x) = y model (x) means that they are completely consistent on all hazard categories, and if there is any category that is not consistent, δ(x) = 0.

[0023] Take the sample x in the labeled set L as the input space, and the corresponding δ(x) value as the label space, to form a sample set D conf = {(x1, δ(x1)), (x2, δ(x2)),..., (x n , δ(x n ))}, and divide it into training set and test set.

[0024] A pre-defined multi-layer neural network structure is used as the basic framework of CC; stochastic gradient descent (SGD) is used as the optimizer, and the binary cross-entropy loss function L is minimized. BCE To optimize the model, train it until the loss function converges.

[0025] Furthermore, the binary cross-entropy loss function L BCE The calculation method is as follows:

[0026]

[0027] In the formula, m represents the total number of training samples, and y i The label representing the input sample. This represents the input result of the classifier.

[0028] Furthermore, the specific method for testing the accuracy of the confidence classifier CC on each hazard category in S4 is as follows:

[0029] From D conf Extract the required test set samples and divide them into subsets T according to the hazard category k. k Each subset contains sample pairs:

[0030]

[0031] Where, N k Representing subset T k The total number of elements in the middle; δ k (x i ) represents sample x i Consistency in labeling across category k;

[0032] For each test sample x i ∈T k Input CC to obtain its confidence score CC on category k. k (x i );

[0033] Set a decision threshold α, and set CC k (x i Samples with a value ≥ α are considered "reliable pre-labeled", meaning the predicted label is accurate. Conversely, it is judged as "unreliable," meaning the predicted label is not accurate.

[0034] Calculate the precision index for category k:

[0035]

[0036] Among them, TP k The number of true positive samples, i.e., δ k (x i ) = 1 and the total number of samples; FP k is the number of false positive samples, i.e. δ k (x i )=0 and the total number of samples;

[0037] The above calculation is repeated for all hazard categories k∈{1,2,…,K} to obtain a list of precisions for each hazard category [Precision1,Precision2,…,Precision k ].

[0038] Further, the threshold θ setting method in S4 is:

[0039] The confidence threshold θ is obtained by analyzing historical annotation data to count the consistency proportion of sample pre-annotation results and real manual annotation results under different confidence scores;

[0040] Suppose that the historical annotation data set contains N samples, for each sample i, the confidence score output by the confidence classifier is CC(x i ), and the consistency label of the real manual annotation result and the pre-annotation result is y i ; the value range [0,1] of the confidence score is divided into M intervals [b0,b1), [b1,b2), …, [b M-1 ,b M ], where b0=0,b M =1;

[0041] The number of samples n m-1 in each interval [[b m ,b m ] is counted, as well as the number of samples n m,y＝1 with consistent annotations, and the consistency proportion p m in each interval is calculated:

[0042]

[0043] The annotation accuracy is measured by the proportion of pre-annotation correctness in samples higher than the threshold θ, denoted as Precision(θ), assuming that θ falls in the interval [b k-1 ,b k ), then:

[0044]

[0045] The manual annotation workload is measured by the proportion of the number of samples lower than the threshold θ in the total number of samples, denoted as Workload(θ):

[0046]

[0047] In order to balance the labeling accuracy and the manual labeling workload, an optimization objective function J(θ) is constructed, and the accuracy and the workload are combined in a weighted manner:

[0048] J(θ) = ω·Precision(θ) - (1-ω)·Workload(θ)

[0049] Wherein, ω ∈ [0, 1] is a weight parameter, used to adjust the relative importance of accuracy and workload; the greater ω, the higher the importance of labeling accuracy; the smaller ω, the more attention to reducing the manual labeling workload;

[0050] By traversing all possible θ values, the corresponding J(θ) value is calculated, and the θ that maximizes J(θ) is selected as the optimal threshold:

[0051] θ * = argmax θ J(θ)

[0052] The optimal threshold θ is determined by the above quantitative method.

[0053] Further, the judgment method of S4 for determining whether to retrain CD and CC according to the category distribution is:

[0054] Let the labeled sample set L contain K hidden danger categories, and the number of samples of each category is n1, n2, …, n K , and the total number of samples The category frequency is defined as The balance index is: Wherein, is the uniform distribution frequency, and E ∈ [0, 1];

[0055] Let the number of samples of each category in the manual labeling set M be m1, m2, …, m K , and the offset is defined as:

[0056]

[0057] When the new balance index E ′ after labeling satisfies E ′ <E0 or the absolute value of the offset exceeds the threshold, that is , retraining is triggered; wherein, E0 is the initial balance threshold; γ is a preset offset threshold.

[0058] As a second aspect of the present application, a semi-automatic labeling system for rail transit engineering construction video images is also provided, comprising:

[0059] A video key frame acquisition unit is configured to extract video key frames from a continuous video stream by removing redundant frames with similar content to obtain a block index set VCI;

[0060] A multi-modal large model pre-labeling unit is configured to construct prompt words of a multi-modal large model, and place all images in the block index set VCI into the constructed prompt words respectively to obtain a result set Rma after pre-labeling of the large model, and determine all elements ma in Rma as a set of pre-labeled results. i and perform binaryzation processing on the image;

[0061] A classifier training unit is configured to select a sample element in the unlabeled sample set U to migrate to the labeled sample set L based on an active learning strategy, retrain the classifier after each migration, and end the process when the size of the labeled set L reaches a standard; and train a confidence classifier CC by comparing the consistency of the manual labeling result of the labeled sample and the pre-labeling result of the multi-modal large model.

[0062] A sample screening and labeling unit is configured to screen top-K negative samples from the unlabeled set U to form a subset S using the classifier CD; test the accuracy of the confidence classifier CC on each hazard category, score the samples in S after reaching the standard, combine the set threshold θ, and use the pre-labeling result for samples with CC(x)≥θ; manually label samples with CC(x)<θ and place them in the manually labeled set M; move the samples in the set M to the labeled set L, and remove the corresponding samples from the unlabeled sample set U; and determine whether to retrain CD and CC according to the category distribution.

[0063] As a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program executed by a processor to perform any step of the semi-automatic labeling method for rail transit engineering construction video images.

[0064] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0065] 1. The semi-automatic labeling method for rail transit engineering construction video images of the present application minimizes the difference between the labeled set and the unlabeled set to achieve efficient sample screening of rail transit engineering construction video images. The classifier is used to measure the difference between the labeled samples and the unlabeled samples, and the sample with the largest difference from the labeled sample in the unlabeled sample is selected as the object to be labeled, and the labeling resources are focused on the data that significantly improve the performance of the model through iterative migration and model retraining, which effectively reduces the blindness of manual labeling and improves the efficiency of the labeling process under the premise of ensuring labeling quality.

[0066] 2.The semi-automatic labeling method for rail transit engineering construction video images according to the present application, by training the confidence classifier, realizes the reliability evaluation of the pre-labeling result of the multi-modal large model. The consistency of the manual labeling and the pre-labeling result based on the labeled samples is used to construct the training data, and a specific neural network structure and an optimization method are used for model training, so that the confidence classifier can quantitatively output the reliability score of the pre-labeling result, providing a basis for subsequent labeling decision, and forming a scientific process of "automatically adopting high-confidence labeling and manually correcting low-confidence labeling".

[0067] 3.The semi-automatic labeling method for rail transit engineering construction video images according to the present application, by establishing a model dynamic retraining mechanism based on the category distribution, to adapt to the changes of data distribution. By quantitatively analyzing the category distribution of the labeled sample set, reasonable judgment indicators and trigger conditions are set, when the manual labeling samples cause the imbalance of the category distribution or the proportion of the key category samples does not meet the requirements, the retraining of the classifier is automatically triggered, to ensure that the model always maintains the accurate recognition ability of various hidden dangers, and maintains the stability and reliability of the labeling system. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 A semi-automatic labeling method for rail transit engineering construction video images according to an embodiment of the present application is provided.

[0069] Figure 2 A key sample classifier training flowchart according to an embodiment of the present application is provided.

[0070] Figure 3 A multi-layer neural network structure according to an embodiment of the present application is provided.

[0071] Figure 4 A sample labeling and screening method according to an embodiment of the present application is provided.

[0072] Figure 5 A system unit diagram according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0074] Embodiment 1

[0075] Please refer to Figure 1 Embodiment 1 provides a semi-automatic labeling method for rail transit engineering construction video images, comprising:

[0076] S1. Extracting video key frames from a continuous video stream by removing redundant frames with similar content to obtain a block index set VCI;

[0077] S2. Constructing a prompt word for a multi-modal large model, and placing all images in the block index set VCI into the constructed prompt word respectively to obtain a result set Rma after pre-labeling of the large model, and comparing all elements ma in Rma i and performing binaryzation processing;

[0078] S3. Based on an active learning strategy, selecting sample elements in the unlabeled sample set U to migrate to the labeled sample set L, retraining the classifier after each migration until the size of the labeled set L reaches the standard, and ending the process; training the confidence classifier CC by comparing the consistency of the manual labeling result of the labeled sample and the pre-labeling result of the multi-modal large model;

[0079] S4. Using the classifier CD to screen top-K negative samples from the unlabeled set U to form a subset S; testing the accuracy of the confidence classifier CC on each risk category, scoring the samples in S after reaching the standard, combining the set threshold θ, using the pre-labeling result for samples with CC(x)≥θ; manually labeling samples with CC(x)<θ and placing them in the manually labeled set M; moving the samples in set M to the labeled set L, while removing the corresponding samples from the unlabeled sample set U; and deciding whether to retrain CD and CC according to the category distribution.

[0080] This embodiment 1 further expands the above steps.

[0081] (1) Video key frame acquisition

[0082] For a video, calculate the structural similarity index (SSIM) of adjacent video frames in time sequence, which involves pixel mean, variance, and covariance. In a specific preferred embodiment, the stability constants are set to 6.5 and 58.5 respectively, and the calculation result is compared with the threshold value 0.75. If it is greater than the threshold value, the next video frame is removed. If it is less than the threshold value, the next video frame key frame is retained.

[0083] (2) Multi-modal large model pre-labeling

[0084] Constructing a prompt word for a multi-modal large model (MLLM). For example, using a multi-modal large model to determine whether a picture exists "fire operation point without fire extinguishing equipment" safety hazard, the prompt word is constructed as follows:

[0085] "<image_placeholder>

Fire operation safety hazard identification

[0086] This is a construction site scene picture, please analyze based on computer vision and safety specification knowledge base, execute according to the following steps:

[0087] 1. Position the fire operation point (welding, cutting, etc. open flame area);

[0088] 2. Check the effective fire extinguishing equipment (fire extinguisher, fire sand box, etc.) within 1 meter around the operation point;

[0089] 3. According to the "fire operation" clause in "Building Construction Safety Inspection Standard" JGJ59:

[0090] - No equipment or not up to standard → output

Hidden danger

[0091] - Equipment complete and compliant → output

No hidden danger

[0092] The above prompt words use <image_placeholder> to represent pictures that need to be identified for hidden dangers.

[0093] Refer to the prompt word format in S2, construct the prompt words of all hidden danger categories. Put all images in the block index set VCI into the above constructed prompt words respectively, and get the result set Rma after pre-labeling of the large model Rma={ma1,ma2,...,ma m+1}, where ma i =[r1,r2,...,r p ] is the image ci i in the set CI. The MLLM output result on all hidden danger categories, p is the number of hidden danger categories that need to be identified.

[0094] The set CI∈VCI, CI={ci1,ci2,...,ci m+1}, the CI element ci i is the key frame of the video extracted by removing similar content in S1.

[0095] All elements ma i in the set Rma are binarized, specifically, the elements r i =“Hidden danger” in ma i are replaced with the number 1, and r i is replaced with the number 0 except for “Hidden danger”.

[0096] Label example: Suppose you want to identify whether the image contains [no fire extinguishing equipment at the fire operation point, concentrated stacking of steel pipe frames on the fastener type steel pipe scaffold, etc. Heavy load, existence of civilian power strip, simultaneous operation of main and auxiliary hooks of crane] Four kinds of hidden dangers, get the label

[1010] indicating the existence of the first three kinds of hidden dangers and the absence of the second and fourth kinds of hidden dangers

[0097] (3) Classifier training

[0098] 3.1 Key sample classifier CD training

[0099] Please refer to Figure 2 , randomly select a small number of samples in the set VCI for manual marking to form the set L, and the other unmarked samples form the set U. Set the total number of queries K, that is, the total number of samples that need to be manually labeled and the total number of queries n, usually 10-20 times, then the sample size of each migration is K / n:

[0100] A multi-layer neural network structure is used, including 3x3 convolution, batch normalization, LeakyRelu / Sigmoid activation function, and fully connected layer as the basic architecture of the classifier CD.

[0101] Train CD with U as negative samples and L as positive samples. The model learns to distinguish "whether it is a key labeled sample" by learning sample features. Use CD to infer the samples in U and filter out "negative samples" with a classification probability close to 0, that is, the samples that the model considers most missing in L. The number of "negative samples" needed is K / n.

[0102] After finding the sample, remove it from set U and move it to set L. This finding and transferring of samples is repeated, and each outer loop is performed n times, and within each outer loop, it is repeated K / n times. When the outer loop is executed n times, the training process is complete, and the updated unmarked sample set U and labeled sample set L are output.

[0103] Please refer to Figure 3 , a multi-layer neural network structure as shown in the figure is used in the training process. In the figure, 3x3Conv represents a 3x3 convolution (the parameters oc in the parentheses represent the output channel number, and s represents the convolution step), BN (Batch Normalization) represents the batch normalization operation, LeakyRelu and Sigmoid are activation functions, and FullyConnected represents a fully connected layer. The entire addition process is completed in n times, and K / n samples are added to L from U each time. After each addition, a new classifier is trained on the new U and L. In a specific preferred embodiment, this process is repeated until the accuracy reaches 98% and stops. Generally, n is in the range of 10-20.

[0104] 3.2 Confidence classifier CC training

[0105] When the size of the labeled sample set L reaches the standard (usually more than 1000), for each sample x∈L, compare its manual labeling result y manual (x) with the pre-labeling result y model (x) of the multi-modal large model.

[0106] Define the indicator function δ(x) as:

[0107]

[0108] wherein, y manual (x) and y model (x) are both in vector form, representing the labeling results of sample x on each hazard category; the equation y manual (x) = y model (x) represents that they are completely consistent on all hazard categories, and if there is any category that is not consistent, then δ(x) = 0;

[0109] Take the sample x in the labeled set L as the input space, and the corresponding δ(x) value as the label space, to combine into a sample set D conf = {(x1, δ(x1)), (x2, δ(x2)), …, (x n , δ(x n ))}, and divide it into a training set and a test set; in a specific preferred embodiment, D conf is divided into a training set and a test set in a ratio of 8:2, the former is used for model parameter optimization, and the latter verifies the generalization ability;

[0110] A preset multi-layer neural network structure is adopted, such as containing 3x3 convolution, batch normalization, LeakyRelu / Sigmoid activation function and full connection layer as the basic framework of CC; ensure the joint modeling ability of the consistency of image features and labels. Take stochastic gradient descent (SGD) as the optimizer, and optimize the model by minimizing the binary cross-entropy loss function L BCE , and train until the loss function converges, such as the loss fluctuation of continuous iteration being less than a threshold value.

[0111] In a specific preferred embodiment, the binary cross-entropy loss function L BCE is calculated as follows:

[0112]

[0113] In the formula, m represents the total number of training samples, y i represents the label of the input sample, represents the input result of the classifier.

[0114] (4) Sample screening and labeling

[0115] Please refer to Figure 4 , and use the classifier CD trained by S3 to screen top-K samples from set U that meet the requirements to form a subset S of samples to be processed. The screening criteria are based on the classifier CD's measurement of sample diversity, such as the case where the sample is judged by the classifier as the most obvious negative sample feature.

[0116] The classifier CC in S3 is tested on the test set to determine whether the accuracy requirement is met in each hazard category. The accuracy requirement can be set according to the actual application scenario, for example, the accuracy requirement is required to be more than 95% in some key hazard categories, and the accuracy requirement is required to be 90% in other categories. If the accuracy requirement is not met in one or more hazard categories, the classifier CC is retrained when a new batch of samples is labeled. If the classifier CC meets the accuracy requirement in each hazard category, each sample x in the subset S of samples to be processed is input into the confidence classifier CC to obtain the confidence score CC(x).

[0117] In a specific preferred embodiment, the specific method for testing the accuracy of the confidence classifier CC in each hazard category is as follows:

[0118] The required test set samples are extracted from D conf , and are divided into subsets T k according to the hazard categories k, such as hot work operation and scaffold load.

[0119]

[0120] Wherein, N k represents the total number of elements in the subset T k ; δ k (x i ) represents the labeling consistency of the sample x i in the category k.

[0121] For each test sample x i ∈T k , input CC to obtain its confidence score CC k (x i ) in category k.

[0122] Set the decision threshold α, and determine the sample CC k (x i ) ≥ α as "pre-labeled reliable", that is, the predicted label , otherwise, determine it as "unreliable", that is, the predicted label

[0123] Calculate the accuracy index of category k:

[0124]

[0125] Wherein, TP k is the number of true positive samples, that is, the total number of samples δ k (x i ) = 1 and ; FP k is the number of false positive samples, that is, the total number of samples δ k(x i ) = 0 and The total number of samples;

[0126] Repeat the above calculation for all hazard categories k∈{1,2,…,K} to obtain a precision list for each hazard category [Precision1,Precision2,…,PrecisionK]. k ].

[0127] The confidence threshold θ is obtained by analyzing historical annotation data and statistically analyzing the consistency ratio between the sample pre-annotation results and the actual manual annotation results under different confidence scores.

[0128] Suppose that the historical labeled dataset contains N samples. For each sample i, the confidence score output by the confidence classifier is CC(x). i The consistency label between the actual manual annotation results and the pre-annotated results is y. i y i =1 indicates consistency, y i =0 indicates inconsistency. The confidence score range [0,1] is divided into M intervals [b0,b1), [b1,b2),...,[b... M-1 ,b M ], where b0=0, b M =1;

[0129] Statistical analysis of each interval [[b] m-1 ,b m The number of samples n in the range) m And the number of samples with consistent labeling, n. m,y＝1 Calculate the consistency ratio p within each interval. m :

[0130]

[0131] The accuracy of the annotation is measured by the proportion of correctly pre-labeled samples above a threshold θ, denoted as Precision(θ). It is assumed that θ falls within the interval [b...]. k-1 ,b k ),but:

[0132]

[0133] The workload of manual annotation is measured by the proportion of samples below a threshold θ to the total number of samples, denoted as Workload(θ).

[0134]

[0135] To balance the labeling accuracy and the workload of manual labeling, an optimization objective function J(θ) is constructed, and the accuracy and the workload are combined in a weighted manner:

[0136] J(θ) = ω·Precision(θ) - (1-ω)·Workload(θ)

[0137] where ω ∈ [0, 1] is a weight parameter for adjusting the relative importance of the accuracy and the workload; the greater ω is, the higher the importance of the labeling accuracy is; the smaller ω is, the more attention is paid to reducing the workload of manual labeling;

[0138] By traversing all possible θ values, i.e., each interval boundary b m , the corresponding J(θ) value is calculated, and the θ that maximizes J(θ) is selected as the optimal threshold:

[0139] θ * = argmax θ J(θ)

[0140] The optimal threshold θ is determined by the above quantification method.

[0141] For example, it is found through experiments that when θ = 0.8, the proportion of pre-labeled correct samples reaches 90%, and then θ can be set to 0.8. For a sample x, if CC(x) ≥ θ, the pre-labeled result of the multi-modal large model is directly taken as the true label of the sample; if CC(x) < θ, the sample is put into the manual labeling set M for manual labeling correction.

[0142] For the sample set M that is manually labeled, it is re-added to the labeled sample set L, and the corresponding samples are removed from the unlabeled sample set U. According to the new labeled sample set L and the unlabeled sample set U, it is considered whether the classifier CD and CC need to be retrained. If the number of manually labeled samples is large and is concentrated on certain categories, the retraining is performed.

[0143] In a preferred embodiment, the judgment method for deciding whether to retrain CD and CC according to the category distribution is:

[0144] Suppose that the labeled sample set L contains K hidden danger categories, and the number of samples of each category is n1, n2, …, n K , and the total number of samples is The category frequency is defined as The balance index is: wherein is the uniform distribution frequency, and E ∈ [0, 1]; the closer E is to 1, the more balanced the distribution is.

[0145] Suppose that the number of samples of each category in the manual labeling set M is m1, m2, …, mK , define the offset:

[0146]

[0147] When the new balance index E ′ satisfies E ′ <E0 or the sum of the absolute values of the offsets exceeds the threshold value, i.e. trigger retraining. Wherein E0 is the initial balance threshold; γ is a preset offset threshold.

[0148] Through the above detailed specific embodiments, the present application can effectively realize the semi-automatic labeling of urban rail transit engineering construction safety hazard images, reduce the workload of manual labeling, and ensure the accuracy and efficiency of labeling.

[0149] Embodiment 2

[0150] Please refer to Figure 5 , the embodiment 2 provides a kind of semi-automatic labeling system for rail transit engineering construction video image, comprising:

[0151] Video key frame acquisition unit is used to extract video key frame from continuous video stream by eliminating similar content Redundant frame to obtain block index set VCI;

[0152] Multi-modal large model pre-labeling unit is used to build the prompt word of multi-modal large model, and all images in block index set VCI are placed in the built prompt word respectively, to obtain the result set Rma after large model pre-labeling, all elements ma i In Rma, binary processing is carried out;

[0153] Classifier training unit is used to select sample element in unmarked sample set U based on active learning strategy and migrate to labeled sample set L, retrain classifier after each migration, until the size of labeled set L reaches the standard and end this process;By comparing the consistency of the manual labeling result of the labeled sample and the pre-labeling result of the multi-modal large model, the confidence classifier CC is trained;

[0154] Sample screening and labeling unit is used to screen top-K negative samples from unmarked set U to form subset S using classifier CD;Test the precision of confidence classifier CC on each hazard category, score the samples in S after reaching the standard, combine the set threshold θ, the sample of CC (x) ≥θ uses pre-labeling result;The sample of CC (x) <θ is manually labeled and placed in manually labeled set M;The samples of set M are moved to labeled set L, while the corresponding samples are removed from unmarked sample set U;And according to category distribution, decide whether to retrain CD and CC.

[0155] Embodiment 3

[0156] The embodiment 3 also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement any step of the semi-automatic labeling method for rail transit engineering construction video images.

[0157] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0158] For the computer readable storage medium provided in the present application, refer to the above method embodiments, and the present application will not be repeated here.

[0159] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A semi-automatic annotation method for video images of rail transit engineering construction, characterized in that, include: S1. Extract video keyframes from a continuous video stream by removing redundant frames with similar content to obtain a block index set (VCI); S2. Construct cue words for the multimodal large model, and put all images in the block index set VCI into the constructed cue words to obtain the pre-annotated result set Rma of the large model. Then, for all elements ma in Rma... i Perform binarization; S3. Based on the active learning strategy, select sample elements in the unlabeled sample set U and transfer them to the labeled sample set L. Retrain the classifier after each transfer until the size of the labeled set L reaches the target. Train the confidence classifier CC by comparing the consistency between the manually labeled results of the labeled samples and the pre-labeled results of the multimodal large model. S4. Use classifier CD to select the top-K negative samples from the unlabeled set U to form a subset S; test the accuracy of confidence classifier CC on each hazard category, and score the samples in S after the accuracy is achieved. Combined with the set threshold θ, use the pre-labeled results for samples CC(x)≥θ; manually label the samples CC(x)<θ and put them into the manually labeled set M; move the samples in set M into the labeled set L, and remove the corresponding samples from the unlabeled sample set U; and decide whether to retrain CD and CC based on the category distribution.

2. The semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, element ma in S2 i Specifically: ma i =[r1,r2,...,r p [Image ci in set CI] i The MLLM output for all hazard categories, where p is the number of hazard categories to be identified; The set CI ∈ VCI, CI = {ci1, ci2, ..., ci} m+1 The CI element ci i Extract video keyframes from S1 by removing redundant frames with similar content.

3. The semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The specific method for selecting sample elements in the unlabeled sample set U in S3 is as follows: A small number of samples from the set VCI are randomly selected and manually labeled to form set L, while the remaining unlabeled samples form set U. Given the total number of queried labeled samples K and the total number of queries n, the sample size for each migration is K / n. A multi-layer neural network structure is used as the basic architecture of the classifier CD. The model trains CD using U as a negative sample and L as a positive sample. The model learns the sample features to distinguish whether a sample is a key labeled sample. CD is used to infer the samples in U and filter out the "negative samples" with a classification probability close to 0. The number of "negative samples" required is K / n.

4. The semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The specific training method for the confidence classifier CC in S3 is as follows: Once the labeled sample set L reaches the required size, for each sample x∈L, compare it with its manually labeled result y. manual (x) and the pre-labeled results of the multimodal large model y model (x); Define the indicator function δ(x) as: Among them, y manual (x) and y model (x) are all in vector form, representing the labeling results of sample x in each hazard category; equation y manual (x)=y model δ(x) indicates that the two are completely consistent across all hazard categories. If there is any inconsistency in any category, then δ(x) = 0. Using samples x from the labeled set L as the input space and the corresponding δ(x) values ​​as the label space, a sample set D is formed. conf ={(x1,δ(x1)),(x2,δ(x2)),…,(x n ,δ(x n The dataset is divided into training and testing sets. A pre-defined multi-layer neural network structure is used as the basic framework of CC; stochastic gradient descent (SGD) is used as the optimizer, and the binary cross-entropy loss function L is minimized. BCE To optimize the model, train it until the loss function converges.

5. A semi-automatic annotation method for video images of rail transit engineering construction according to claim 4, characterized in that, The binary cross-entropy loss function L BCE The calculation method is as follows: In the formula, m represents the total number of training samples, and y i The label representing the input sample. This represents the input result of the classifier.

6. A semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The specific method for testing the accuracy of the confidence classifier CC in each hazard category in S4 is as follows: From D conf Extract the required test set samples and divide them into subsets T according to the hazard category k. k Each subset contains sample pairs: Where, N k Representing subset T k The total number of elements in the middle; δ k (x i ) represents sample x i Consistency in labeling across category k; For each test sample x i ∈T k Input CC to obtain its confidence score CC on category k. k (x i ); Set a decision threshold α, and set CC k (x i Samples with a value ≥ α are considered "reliable pre-labels", meaning the predicted labels are... Conversely, it is judged as "unreliable," meaning the predicted label is not accurate. Calculate the precision index for category k: Among them, TP k The number of true positive samples, i.e., δ k (x i ) = 1 and Total number of samples; FP k The number of false positive samples, i.e., δ k (x i ) = 0 and The total number of samples; Repeat the above calculation for all hazard categories k∈{1,2,…,K} to obtain a precision list for each hazard category [Precision1,Precision2,…,PrecisionK]. k ].

7. A semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The method for setting the threshold θ in S4 is as follows: The confidence threshold θ is obtained by analyzing historical annotation data and statistically analyzing the consistency ratio between the sample pre-annotation results and the actual manual annotation results under different confidence scores. Suppose that the historical labeled dataset contains N samples. For each sample i, the confidence score output by the confidence classifier is CC(x). i The consistency label between the actual manual annotation results and the pre-annotated results is y. i The confidence score range [0,1] is divided into M intervals [b0,b1), [b1,b2), ..., [b M-1 ,b M ], where b0=0, b M =1; Statistical analysis of each interval [[b] m-1 ,b m The number of samples n in the range) m And the number of samples with consistent labeling, n. m,y＝1 Calculate the consistency ratio p within each interval. m : The accuracy of the annotation is measured by the proportion of correctly pre-labeled samples above a threshold θ, denoted as Precision(θ). It is assumed that θ falls within the interval [b...]. k-1 ,b k ),but: The workload of manual annotation is measured by the proportion of samples below a threshold θ to the total number of samples, denoted as Workload(θ). To balance the accuracy of annotation and the workload of manual annotation, an optimization objective function J(θ) is constructed, and a weighted approach is used to combine accuracy and workload: J(θ)=ω·Precision(θ)-(1-ω)·Workload(θ) Where ω∈[0,1] is a weighting parameter used to adjust the relative importance of accuracy and workload; the larger ω is, the higher the importance attached to the accuracy of annotation; the smaller ω is, the more emphasis is placed on reducing the workload of manual annotation. By iterating through all possible values ​​of θ, calculating the corresponding J(θ) value, and selecting the θ that maximizes J(θ) as the optimal threshold: i * =argmax θ J(θ) The optimal threshold θ is determined using the quantization method described above.

8. A semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The method for determining whether to retrain CD and CC based on category distribution in S4 is as follows: Suppose that the labeled sample set L contains K hazard categories, and the number of samples in each category is n1, n2, ..., n. K Total number of samples Define category frequency as The equilibrium index is: in, For a uniformly distributed frequency, E∈[0,1]; Let the number of samples in each category of the manually labeled set M be m1, m2, ..., m. K Define the offset: When the new balance index E after annotation ′ satisfies E ′ < E0 or the sum of the absolute values of the offsets exceeds the threshold, that is re-training is triggered; where E0 is the initial balance threshold; γ is the preset offset threshold.

9. A semi-automatic annotation system for video images of rail transit engineering construction, characterized in that, include: The video keyframe acquisition unit is used to extract video keyframes from a continuous video stream by removing redundant frames with similar content to obtain a block index set (VCI). The multimodal large model pre-annotation unit is used to construct prompts for the multimodal large model. All images in the block index set VCI are placed into the constructed prompts, resulting in the pre-annotated large model set Rma. For each element ma in Rma... i Perform binarization; The classifier training unit is used to select sample elements from the unlabeled sample set U and transfer them to the labeled sample set L based on an active learning strategy. The classifier is retrained after each transfer until the size of the labeled set L reaches the target. The confidence classifier CC is trained by comparing the consistency between the manually labeled results of the labeled samples and the pre-labeled results of the multimodal large model. The sample selection and labeling unit is used to select the top-K negative samples from the unlabeled set U to form a subset S using classifier CD; test the accuracy of confidence classifier CC on each hazard category, and score the samples in S after reaching the target. Based on the set threshold θ, samples with CC(x)≥θ are pre-labeled; samples with CC(x)<θ are manually labeled and placed into the manually labeled set M; the samples in set M are moved into the labeled set L, and the corresponding samples are removed from the unlabeled sample set U; and the unit decides whether to retrain CD and CC based on the category distribution.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor according to any one of claims 1-8, which describes a semi-automatic annotation method for video images of rail transit engineering construction.

Images