Semi-automatic labeling method and system for rail transit engineering construction video images

Abstract

This invention discloses a semi-automatic annotation method and system for video images in rail transit engineering construction. The method includes: removing redundant frames from the video stream to extract keyframes and forming a block index set; using constructed prompt words to drive a multimodal large model to pre-annotate the images; and binarizing the annotation results. Based on active learning, unlabeled samples are repeatedly transferred to the labeled set, and a key sample classifier CD is trained until the labeled set reaches a certain size; simultaneously, a confidence classifier CC is trained by comparing manual and pre-annotated results. For the remaining samples in the unlabeled set, after the CC accuracy test meets the standard, annotation tasks are divided according to a threshold θ: high-confidence samples are pre-annotated, and low-confidence samples are manually annotated. Finally, the manually annotated samples update the set, and the model is retrained based on the category distribution. This method effectively reduces the blindness of manual annotation and improves the efficiency of the annotation process while ensuring annotation quality.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, and more specifically, relates to a semi-automatic annotation method and system for video images of rail transit engineering construction. Background Technology

[0002] In the field of urban rail transit engineering construction, with the widespread application of video surveillance technology, the demand for annotation of massive amounts of video image data is increasing daily. Traditional manual annotation methods face significant efficiency bottlenecks and cost pressures: on the one hand, engineering construction scenarios are complex and diverse, including personnel operations, equipment operation, environmental changes, and other diverse aspects. Manual frame-by-frame annotation requires a large amount of manpower and time, especially for long-cycle, large-scale construction projects, where the annotation efficiency cannot meet actual needs; on the other hand, the consistency and accuracy of manual annotation are easily affected by subjective factors, and the annotation of subtle features in complex scenarios (such as safety hazard identification and equipment component status judgment) is difficult and the quality is hard to guarantee.

[0003] Meanwhile, existing semi-automatic annotation technologies have significant limitations in the application of engineering construction. Some annotation methods based on a single modality struggle to fully capture the complex information in video images and lack the ability to represent multi-dimensional scene features. Traditional active learning strategies, when selecting samples, lack targeted optimization for the specific characteristics of engineering construction scenarios, easily falling into the problem of imbalanced class distribution, leading to model training bias. Furthermore, key information in engineering construction videos is often distributed in specific frames, and traditional methods lack effective keyframe extraction mechanisms, resulting in redundant annotation workload and further reducing annotation efficiency.

[0004] With the development of multimodal large model technology, it has shown advantages in cross-modal information understanding and feature extraction. However, when directly applied to the annotation of engineering construction video images, it still faces the problem of domain adaptability: there are differences between the professional terminology and scene characteristics of the engineering construction field and the training data of general multimodal models, and the accuracy of pre-annotation results needs to be optimized by combining domain knowledge. How to combine the pre-annotation capabilities of multimodal large models with the particularities of engineering construction scenarios and build an efficient semi-automatic annotation process has become a key issue in improving the efficiency of rail transit engineering video image analysis. Summary of the Invention

[0005] This invention aims to provide a semi-automatic annotation method and system for video images of urban rail transit engineering construction. By combining multimodal large model pre-annotation with active learning strategies, and utilizing classifier CD and confidence CC to screen high-value samples, the method reduces the workload of manual annotation and improves annotation efficiency. Simultaneously, through mechanisms such as keyframe extraction and dynamic model optimization, it ensures annotation quality, solving the problems of low efficiency, high cost, and insufficient adaptability of existing semi-automatic technologies in traditional manual annotation. This method is suitable for intelligent analysis of engineering construction video images.

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a semi-automatic annotation method for video images of rail transit engineering construction, comprising:

[0007] S1. Extract video keyframes from a continuous video stream by removing redundant frames with similar content to obtain a block index set (VCI);

[0008] S2. Construct cue words for the multimodal large model, and put all images in the block index set VCI into the constructed cue words to obtain the pre-annotated result set Rma of the large model. Then, for all elements ma in Rma... i Perform binarization;

[0009] S3. Based on the active learning strategy, select sample elements in the unlabeled sample set U and transfer them to the labeled sample set L. Retrain the classifier after each transfer until the size of the labeled set L reaches the target. Train the confidence classifier CC by comparing the consistency between the manually labeled results of the labeled samples and the pre-labeled results of the multimodal large model.

[0010] S4. Use classifier CD to select the top-K negative samples from the unlabeled set U to form a subset S; test the accuracy of confidence classifier CC on each hazard category, and score the samples in S after the accuracy is achieved. Combined with the set threshold θ, use the pre-labeled results for samples CC(x)≥θ; manually label the samples CC(x)<θ and put them into the manually labeled set M; move the samples in set M into the labeled set L, and remove the corresponding samples from the unlabeled sample set U; and decide whether to retrain CD and CC based on the category distribution.

[0011] Furthermore, in S2, element ma i Specifically:

[0012] ma i =[r1,r2,...,r p [Image ci in set CI] i The MLLM output for all hazard categories, where p is the number of hazard categories to be identified;

[0013] The set CI ∈ VCI, CI = {ci1, ci2, ..., ci} m+1 The CI element ci i Extract video keyframes from S1 by removing redundant frames with similar content.

[0014] Furthermore, the specific method for selecting sample elements in the unlabeled sample set U in S3 is as follows:

[0015] A small number of samples from the set VCI are randomly selected and manually labeled to form set L, while the remaining unlabeled samples form set U. Given the total number of queried labeled samples K and the total number of queries n, the sample size for each migration is K / n.

[0016] A multi-layer neural network structure is used as the basic architecture of the classifier CD.

[0017] The model trains CD using U as a negative sample and L as a positive sample. It learns the sample features to distinguish whether a sample is a key labeled sample. The model then uses CD to infer the samples in U and selects the "negative samples" with a classification probability close to 0. The number of "negative samples" required is K / n.

[0018] Furthermore, the specific training method for the confidence classifier CC in S3 is as follows:

[0019] Once the labeled sample set L reaches the required size, for each sample x∈L, compare it with its manually labeled result y. manual (x) and the pre-labeled results of the multimodal large model y model (x);

[0020] Define the indicator function δ(x) as:

[0021]

[0022] Among them, y manual (x) and y model (x) are all in vector form, representing the labeling results of sample x in each hazard category; equation y manual (x)=y model δ(x) indicates that the two are completely consistent across all hazard categories. If there is any inconsistency in any category, then δ(x) = 0.

[0023] Using samples x from the labeled set L as the input space and the corresponding δ(x) values ​​as the label space, a sample set D is formed. conf ={(x1,δ(x1)),(x2,δ(x2)),…,(x n ,δ(x n The dataset is divided into training and testing sets.

[0024] A pre-defined multi-layer neural network structure is used as the basic framework of CC; stochastic gradient descent (SGD) is used as the optimizer, and the binary cross-entropy loss function L is minimized. BCE To optimize the model, train it until the loss function converges.

[0025] Furthermore, the binary cross-entropy loss function L BCE The calculation method is as follows:

[0026]

[0027] In the formula, m represents the total number of training samples, and y i The label representing the input sample. This represents the input result of the classifier.

[0028] Furthermore, the specific method for testing the accuracy of the confidence classifier CC on each hazard category in S4 is as follows:

[0029] From D conf Extract the required test set samples and divide them into subsets T according to the hazard category k. k Each subset contains sample pairs:

[0030]

[0031] Where, N k Representing subset T k The total number of elements in the middle; δ k (x i ) represents sample x i Consistency in labeling across category k;

[0032] For each test sample x i ∈T k Input CC to obtain its confidence score CC on category k. k (x i );

[0033] Set a decision threshold α, and set CC k (x i Samples with a value ≥ α are considered "reliable pre-labeled", meaning the predicted label is accurate. Conversely, it is judged as "unreliable," meaning the predicted label is not accurate.

[0034] Calculate the precision index for category k:

[0035]

[0036] Among them, TP k The number of true positive samples, i.e., δ k (x i ) = 1 and Total number of samples; FP k The number of false positive samples, i.e., δ k (x i ) = 0 and The total number of samples;

[0037] Repeat the above calculation for all hazard categories k∈{1,2,…,K} to obtain a precision list for each hazard category [Precision1,Precision2,…,PrecisionK]. k ].

[0038] Furthermore, the method for setting the threshold θ in S4 is as follows:

[0039] The confidence threshold θ is obtained by analyzing historical annotation data and statistically analyzing the consistency ratio between the sample pre-annotation results and the actual manual annotation results under different confidence scores.

[0040] Suppose that the historical labeled dataset contains N samples. For each sample i, the confidence score output by the confidence classifier is CC(x). i The consistency label between the actual manual annotation results and the pre-annotated results is y. i The confidence score range [0,1] is divided into M intervals [b0,b1), [b1,b2), ..., [b M-1 ,b M ], where b0=0, b M =1;

[0041] Statistical analysis of each interval [[b] m-1 ,b m The number of samples n in the range) m And the number of samples with consistent labeling, n. m,y＝1 Calculate the consistency ratio p within each interval. m :

[0042]

[0043] The accuracy of the annotation is measured by the proportion of correctly pre-labeled samples above a threshold θ, denoted as Precision(θ). It is assumed that θ falls within the interval [b...]. k-1 ,b k ),but:

[0044]

[0045] The workload of manual annotation is measured by the proportion of samples below a threshold θ to the total number of samples, denoted as Workload(θ).

[0046]

[0047] To balance the annotation accuracy and the manual annotation workload, an optimization objective function J(θ) is constructed. By using a weighted method, the accuracy and the workload are combined:

[0048] J(θ) = ω·Precision(θ) - (1 - ω)·Workload(θ)

[0049] where ω ∈ [0, 1] is a weight parameter used to adjust the relative importance of the accuracy and the workload. The larger ω is, the higher the degree of emphasis on the annotation accuracy; the smaller ω is, the more attention is paid to reducing the manual annotation workload.

[0050] By traversing all possible θ values, calculating the corresponding J(θ) values, and selecting the θ that maximizes J(θ) as the optimal threshold:

[0051] θ * = argmax θ J(θ)

[0052] The optimal threshold θ is determined by the above quantization method.

[0053] Furthermore, the judgment method for deciding whether to retrain CD and CC according to the category distribution in S4 is as follows:

[0054] Let the labeled sample set L contain K hidden danger categories, and the number of samples in each category be n1, n2,..., n K , the total number of samples Define the category frequency as The balance index is: where is the uniform distribution frequency, E ∈ [0, 1];

[0055] Let the number of samples in each category in the manual annotation set M be m1, m2,..., m K , define the offset:

[0056]

[0057] When the new balance index E ′ satisfies E ′ < E0 or the sum of the absolute values of the offsets exceeds the threshold, that is , retraining is triggered; where E0 is the initial balance threshold; γ is the preset offset threshold.

[0058] As the second aspect of the present invention, a semi-automatic annotation system for video images of rail transit engineering construction is further provided, including:

[0059] The video keyframe acquisition unit is used to extract video keyframes from a continuous video stream by removing redundant frames with similar content to obtain a block index set (VCI).

[0060] The multimodal large model pre-annotation unit is used to construct prompts for the multimodal large model. All images in the block index set VCI are placed into the constructed prompts, resulting in the pre-annotated large model set Rma. For each element ma in Rma... i Perform binarization;

[0061] The classifier training unit is used to select sample elements from the unlabeled sample set U and transfer them to the labeled sample set L based on an active learning strategy. The classifier is retrained after each transfer until the size of the labeled set L reaches the target. The confidence classifier CC is trained by comparing the consistency between the manually labeled results of the labeled samples and the pre-labeled results of the multimodal large model.

[0062] The sample selection and labeling unit is used to select the top-K negative samples from the unlabeled set U to form a subset S using classifier CD; test the accuracy of confidence classifier CC on each hazard category, and score the samples in S after reaching the target. Based on the set threshold θ, samples with CC(x)≥θ are pre-labeled; samples with CC(x)<θ are manually labeled and placed into the manually labeled set M; the samples in set M are moved into the labeled set L, and the corresponding samples are removed from the unlabeled sample set U; and the unit decides whether to retrain CD and CC based on the category distribution.

[0063] As a third aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor of any step of the semi-automatic annotation method for video images of rail transit engineering construction.

[0064] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0065] 1. The semi-automatic annotation method for video images of rail transit engineering construction of the present invention achieves efficient sample selection of video images of rail transit engineering construction by minimizing the difference between the labeled set and the unlabeled set. A classifier is used to measure the difference between labeled and unlabeled samples, and samples with the greatest difference from labeled samples among the unlabeled samples are preferentially selected as the objects to be annotated. Through iterative transfer learning and model retraining, annotation resources are focused on data that significantly improves model performance. While ensuring annotation quality, this method effectively reduces the blindness of manual annotation and improves the efficiency of the annotation process.

[0066] 2. The semi-automatic annotation method for video images of rail transit engineering construction of the present invention achieves reliability assessment of pre-annotation results of multimodal large models by training a confidence classifier. Training data is constructed based on the consistency between manually annotated samples and pre-annotation results. A specific neural network structure and optimization method are used for model training, enabling the confidence classifier to quantify the reliability score of the pre-annotation results, providing a basis for subsequent annotation decisions and forming a scientific process of "automatically adopting high-confidence annotations and manually correcting low-confidence annotations."

[0067] 3. The semi-automatic annotation method for video images of rail transit engineering construction of the present invention establishes a dynamic retraining mechanism for the model based on category distribution to adapt to changes in data distribution. By quantitatively analyzing the category distribution of the labeled sample set and setting reasonable judgment indicators and triggering conditions, when manually annotated samples lead to an imbalance in category distribution or the proportion of key category samples does not meet the requirements, the retraining of the classifier is automatically triggered to ensure that the model always maintains its ability to accurately identify various hidden dangers and maintain the stability and reliability of the annotation system. Attached Figure Description

[0068] Figure 1 This is a flowchart of a semi-automatic annotation method for video images of rail transit engineering construction according to an embodiment of the present invention;

[0069] Figure 2 This is a flowchart illustrating the training process of the key sample classifier in an embodiment of the present invention.

[0070] Figure 3 This is a diagram of a multilayer neural network structure according to an embodiment of the present invention;

[0071] Figure 4 This is a sample labeling and screening method according to an embodiment of the present invention;

[0072] Figure 5 This is a system unit diagram of an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0074] Example 1

[0075] Please refer to Figure 1 This embodiment 1 provides a semi-automatic annotation method for video images of rail transit engineering construction, including:

[0076] S1. Extract video keyframes from a continuous video stream by removing redundant frames with similar content to obtain a block index set (VCI);

[0077] S2. Construct cue words for the multimodal large model, and put all images in the block index set VCI into the constructed cue words to obtain the pre-annotated result set Rma of the large model. Then, for all elements ma in Rma... i Perform binarization;

[0078] S3. Based on the active learning strategy, select sample elements in the unlabeled sample set U and transfer them to the labeled sample set L. Retrain the classifier after each transfer until the size of the labeled set L reaches the target. Train the confidence classifier CC by comparing the consistency between the manually labeled results of the labeled samples and the pre-labeled results of the multimodal large model.

[0079] S4. Use classifier CD to select the top-K negative samples from the unlabeled set U to form a subset S; test the accuracy of confidence classifier CC on each hazard category, and score the samples in S after the accuracy is achieved. Combined with the set threshold θ, use the pre-labeled results for samples CC(x)≥θ; manually label the samples CC(x)<θ and put them into the manually labeled set M; move the samples in set M into the labeled set L, and remove the corresponding samples from the unlabeled sample set U; and decide whether to retrain CD and CC based on the category distribution.

[0080] This embodiment 1 further elaborates on the above steps.

[0081] (1) Video keyframe acquisition

[0082] For a video, the structural similarity index (SSIM) of two adjacent video frames is calculated in chronological order, which involves pixel mean, variance, and covariance. In a preferred embodiment, the stability constants are set to 6.5 and 58.5, respectively, and the calculation results are compared with a threshold of 0.75. If the result is greater than the threshold, the next video frame is removed; if the result is less than the threshold, the keyframe of the next video frame is retained.

[0083] (2) Pre-annotation of multimodal large models

[0084] Hint words for constructing a multimodal large model (MLLM). For example, when using a multimodal large model to determine if an image contains a safety hazard such as "no fire extinguishing equipment at the hot work site," the hint words would be as follows:

[0085] “<image_placeholder> [Identification of Safety Hazards During Hot Work]

[0086] This is a picture of a construction site. Please follow these steps based on the Computer Vision Analysis and Safety Standards Knowledge Base:

[0087] 1. Locate the hot work area (areas with open flames such as welding and cutting);

[0088] 2. Verify the availability of fire extinguishing equipment (fire extinguishers, fire sandboxes, etc.) within 1 meter of the work site;

[0089] 3. Based on the "Hot Work" clause in the "Standard for Safety Inspection of Building Construction" JGJ59:

[0090] - Lack of or substandard equipment → Output [Potential risks]

[0091] -Equipment is complete and compliant → Output is [without hidden dangers]

[0092] In the above prompt words<image_placeholder> Images used to represent potential hazards that need to be identified.

[0093] Referring to the prompt word format in S2, construct prompt words for all hazard categories. Place all images from the block index set VCI into the constructed prompt words to obtain the pre-annotated result set Rma = {ma1, ma2, ..., ma...}. m+1}, where ma i =[r1,r2,...,r p [Image ci in set CI] i The MLLM output results for all hazard categories, where p is the number of hazard categories to be identified.

[0094] The set CI ∈ VCI, CI = {ci1, ci2, ..., ci} m+1 The CI element ci i Extract video keyframes from S1 by removing redundant frames with similar content.

[0095] All elements ma in set Rma i Binarization is performed, specifically in the matrix... i Chinese r i = Elements with "potential risks" are replaced with the number 1, r i Replace all elements except "potentially problematic" with the number 0.

[0096] Example of a label: Suppose we want to identify whether an image contains the following four hazards: [no fire extinguishers at the hot work site, heavy loads such as steel bars and steel pipes piled up on the scaffolding, civilian power strips present, and crane main and auxiliary hooks operating simultaneously]. If we get the label

[1010] , it means that the first and third hazards exist, and the second and fourth hazards do not exist.

[0097] (3) Classifier training

[0098] 3.1 Training of Key Sample Classifier CD

[0099] Please refer to Figure 2 A small number of samples from the VCI set are randomly selected and manually labeled to form set L, while the remaining unlabeled samples form set U. The total number of queried labels is set to K, which represents the final total number of samples requiring manual labeling and the total number of queries n, typically 10-20. Therefore, the sample size for each migration is K / n.

[0100] A multi-layer neural network structure is adopted, including 3×3 convolution, batch normalization, LeakyRelu / Sigmoid activation function and fully connected layers as the basic architecture of classifier CD.

[0101] The model trains CD with U as negative samples and L as positive samples. The model learns the features of the samples to distinguish whether they are key labeled samples. The model uses CD to infer the samples in U and selects the "negative samples" with a classification probability close to 0. These are the samples that the model considers to be the most missing in L. The number of "negative samples" required is K / n.

[0102] Once a sample is found, it is removed from set U and moved to set L. This process of finding and moving samples is repeated cyclically. Each outer loop executes n times, and within each outer loop, it executes K / n times. After the outer loop has executed n times, the entire training process ends, and the updated unlabeled sample set U and labeled sample set L are output.

[0103] Please refer to Figure 3 During training, a multi-layer neural network structure as shown in the figure is used. In the figure, 3×3Conv represents a 3×3 convolution (the parameter oc in parentheses represents the number of output channels, and s represents the stride of the convolution), BN (Batch Normalization) represents batch normalization, LeakyReLU and Sigmoid are activation functions, and FullyConnected represents a fully connected layer. The entire addition process is divided into n steps, with K / n samples taken from U and added to L each time. After each addition, a new classifier is trained on the new U and L. In a preferred embodiment, this process is repeated until the accuracy reaches 98%. Generally, n is between 10 and 20.

[0104] 3.2 Training of the Confidence Classifier CC

[0105] Once the labeled sample set L reaches a certain size (generally more than 1000 images), for each sample x∈L, compare it with its manually labeled result y. manual (x) and the pre-labeled results of the multimodal large model y model (x);

[0106] Define the indicator function δ(x) as:

[0107]

[0108] Among them, y manual (x) and y model (x) are all in vector form, representing the labeling results of sample x in each hazard category; equation y manual (x)=y model δ(x) indicates that the two are completely consistent across all hazard categories. If there is any inconsistency in any category, then δ(x) = 0.

[0109] Using samples x from the labeled set L as the input space and the corresponding δ(x) values ​​as the label space, a sample set D is formed. conf ={(x1,δ(x1)),(x2,δ(x2)),…,(x n ,δ(x n ))}, and divide it into training set and test set; in a specific preferred embodiment, D conf The model is divided into a training set and a test set in an 8:2 ratio. The former is used to optimize model parameters, while the latter is used to verify generalization ability.

[0110] A pre-defined multi-layer neural network structure, including 3×3 convolutions, batch normalization, LeakyReLU / Sigmoid activation functions, and fully connected layers, serves as the basic framework for CC, ensuring joint modeling capabilities that align with image features and annotations. Stochastic gradient descent (SGD) is used as the optimizer, minimizing the binary cross-entropy loss function L... BCE To optimize the model, train it until the loss function converges, such as when the loss fluctuation is less than a threshold during continuous iterations.

[0111] In a preferred embodiment, the binary cross-entropy loss function L BCE The calculation method is as follows:

[0112]

[0113] In the formula, m represents the total number of training samples, and y i The label representing the input sample. This represents the input result of the classifier.

[0114] (4) Sample screening and labeling

[0115] Please refer to Figure 4 The classifier CD, trained using S3, selects the top-K samples from set U that meet the requirements, forming a subset S of samples to be processed. The selection criteria are based on the classifier CD's measure of sample dissimilarity, such as the case where the sample is judged by the classifier as having the most obvious negative sample features.

[0116] The classifier CC in S3 is tested on the test set to determine whether it meets the accuracy requirements for each hazard category. The accuracy requirements can be set according to the actual application scenario; for example, an accuracy of over 95% is required for certain key hazard categories, and 90% for other categories. If the accuracy requirements are not met for one or more hazard categories, the classifier CC is retrained while waiting for a new batch of samples to be labeled. If the classifier CC meets the accuracy requirements for all hazard categories, each sample x in the subset S to be processed is input into the confidence classifier CC to obtain the confidence score CC(x).

[0117] In a specific preferred embodiment, the method for testing the accuracy of the confidence classifier CC on each hazard category is as follows:

[0118] From D conf Extract the required test set samples and divide them into subsets T according to the hazard category k, such as hot work, scaffolding load, etc. k Each subset contains sample pairs:

[0119]

[0120] Where, N k Representing subset T k The total number of elements in the middle; δ k (x i ) represents sample x i Consistency in labeling across category k;

[0121] For each test sample x i ∈T k Input CC to obtain its confidence score CC on category k. k (x i );

[0122] Set a decision threshold α, and set CC k (x i Samples with a value ≥ α are considered "reliable pre-labeled", meaning the predicted label is accurate. Conversely, it is judged as "unreliable," meaning the predicted label is not accurate.

[0123] Calculate the precision index for category k:

[0124]

[0125] Among them, TP k The number of true positive samples, i.e., δ k (x i ) = 1 and Total number of samples; FP k The number of false positive samples, i.e., δ k(x i ) = 0 and The total number of samples;

[0126] Repeat the above calculation for all hazard categories k∈{1,2,…,K} to obtain a precision list for each hazard category [Precision1,Precision2,…,PrecisionK]. k ].

[0127] The confidence threshold θ is obtained by analyzing historical annotation data and statistically analyzing the consistency ratio between the sample pre-annotation results and the actual manual annotation results under different confidence scores.

[0128] Suppose that the historical labeled dataset contains N samples. For each sample i, the confidence score output by the confidence classifier is CC(x). i The consistency label between the actual manual annotation results and the pre-annotated results is y. i y i =1 indicates consistency, y i =0 indicates inconsistency. The confidence score range [0,1] is divided into M intervals [b0,b1), [b1,b2),...,[b... M-1 ,b M ], where b0=0, b M =1;

[0129] Statistical analysis of each interval [[b] m-1 ,b m The number of samples n in the range) m And the number of samples with consistent labeling, n. m,y＝1 Calculate the consistency ratio p within each interval. m :

[0130]

[0131] The accuracy of the annotation is measured by the proportion of correctly pre-labeled samples above a threshold θ, denoted as Precision(θ). It is assumed that θ falls within the interval [b...]. k-1 ,b k ),but:

[0132]

[0133] The workload of manual annotation is measured by the proportion of samples below a threshold θ to the total number of samples, denoted as Workload(θ).

[0134]

[0135] To balance the accuracy of annotation and the workload of manual annotation, an optimization objective function J(θ) is constructed, and a weighted approach is used to combine accuracy and workload:

[0136] J(θ)=ω·Precision(θ)-(1-ω)·Workload(θ)

[0137] Where ω∈[0,1] is a weighting parameter used to adjust the relative importance of accuracy and workload; the larger ω is, the higher the importance attached to the accuracy of annotation; the smaller ω is, the more emphasis is placed on reducing the workload of manual annotation.

[0138] By iterating through all possible values ​​of θ, i.e., each interval boundary b m Calculate the corresponding J(θ) value, and select the θ that maximizes J(θ) as the optimal threshold:

[0139] θ * =argmax θ J(θ)

[0140] The optimal threshold θ is determined using the quantization method described above.

[0141] For example, if experiments show that 90% of the pre-labeled samples are correctly labeled when θ = 0.8, then θ can be set to 0.8. For a sample x, if CC(x) ≥ θ, then the pre-labeling result of the multimodal large model is directly used as the true label of the sample; if CC(x) < θ, then the sample is placed into the manually labeled set M for manual labeling correction.

[0142] For the manually labeled sample set M, add it back to the labeled sample set L, and remove the corresponding samples from the unlabeled sample set U. Based on the new labeled sample set L and the unlabeled sample set U, consider whether classifiers CD and CC need to be retrained. If the number of manually labeled samples is large and their distribution is concentrated in certain categories, then retraining is performed.

[0143] In a preferred embodiment, the method for determining whether to retrain CD and CC based on category distribution is as follows:

[0144] Suppose that the labeled sample set L contains K hazard categories, and the number of samples in each category is n1, n2, ..., n. K Total number of samples Define category frequency as The equilibrium index is: in, For a uniformly distributed frequency, E∈[0,1]; the closer E is to 1, the more balanced the distribution.

[0145] Let the number of samples in each category of the manually labeled set M be m1, m2, ..., m.K , define the offset:

[0146]

[0147] When the new balance index E after annotation ′ satisfies E ′ < E0 or the sum of the absolute values of the offsets exceeds the threshold, that is trigger retraining. Among them, E0 is the initial balance threshold; γ is the preset offset threshold.

[0148] Through the above detailed specific implementation manners, the present invention can effectively achieve semi-automatic annotation of images of potential safety hazards in urban rail transit engineering construction, while reducing the manual annotation workload and ensuring the accuracy and efficiency of annotation.

[0149] Embodiment 2 <0​​​​​​​​​​​​​​​​​​​​​

[0156] This embodiment 3 also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement any step of the semi-automatic annotation method for video images of rail transit engineering construction.

[0157] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0159] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semi-automatic annotation method for video images of rail transit engineering construction, characterized in that, include: S1. Extract video keyframes from a continuous video stream by removing redundant frames with similar content to obtain a block index set. ; S2. Construct prompts for a multimodal large model and set up a block index set. All images are placed into the constructed prompt words to obtain the result set after pre-annotation of the large model. ,right All elements Perform binarization; S3. Based on the active learning strategy, select the unlabeled sample set. The sample elements in the sample set are migrated to the labeled sample set. The classifier is retrained after each transfer. until the set of tags Once the target scale is reached, this process ends; a confidence classifier is trained by comparing the consistency between the manually labeled results of the labeled samples and the pre-labeled results of the multimodal large model. ; S4. Using a classifier Unlabeled collection Screening A subset of negative samples Test confidence classifier Accuracy in each hazard category, after meeting the standard The mid-sample score, combined with the set threshold. ,Will The samples used were pre-labeled results; The samples were manually labeled and placed into a manual labeling set. ; set The sample is moved into the label set. Meanwhile, the unlabeled sample set Remove the corresponding samples; and decide whether to retrain based on the class distribution. and ; The selected unlabeled sample set in S3 The specific method for selecting sample elements is as follows: Randomly select a set A small number of samples were manually labeled to form an aggregator. Other unlabeled samples form a set. Set the total number of queried labels. Total number of queries Then the sample size for each migration is : A multi-layer neural network structure is used as the classifier. The basic infrastructure; Will As a negative sample, Used as positive samples for training The model learns sample features to distinguish whether a sample is a key-labeled sample; it utilizes... right Using the medium sample for inference, the number of "negative samples" with a classification probability close to 0 is required to filter them out. ; The The selection criteria for negative samples are based on the classifier. The measure of sample difference is based on the sample that is judged as a negative sample by the classifier with the most obvious features. The sample with the greatest difference from the labeled sample among the unlabeled samples is selected as the object to be labeled.

2. The semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The elements in S2 Specifically: For set Images in MLLM output results for all hazard categories, The number of hazard categories that need to be identified; The set , The element Extract video keyframes from S1 by removing redundant frames with similar content.

3. The semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The confidence classifier in S3 The specific training method is as follows: When the sample set is labeled Once the target size is reached, for each sample Compare with its manually labeled results Compared with pre-labeled results of multimodal large models ; Define indicator functions for: ， in, and All are in vector form, representing the labeling results of sample x in each hazard category; equations This indicates that the two are completely consistent across all hazard categories. If there is any inconsistency in any category, then... ; With the labeled set Samples in As the input space, the corresponding Values ​​are used as the label space and combined to form the sample set. , and divided into training set and test set; Using a pre-defined multi-layer neural network structure as The basic framework; based on stochastic gradient descent The optimizer minimizes the binary cross-entropy loss function. To optimize the model, train it until the loss function converges.

4. The semi-automatic annotation method for video images of rail transit engineering construction according to claim 3, characterized in that, The binary cross-entropy loss function The calculation method is as follows: ， In the formula, This represents the total number of training samples. The label representing the input sample. This represents the input result of the classifier.

5. A semi-automatic annotation method for video images of rail transit engineering construction according to claim 3, characterized in that, The test confidence classifier in S4 The specific method for determining the accuracy for each hazard category is as follows: from Extract the required test set samples and categorize them by hazard type. partitioning subsets Each subset contains sample pairs: ， in, Representing a subset The total number of elements in the text; Indicates sample In category Consistency of annotations; For each test sample ,enter Get its category Confidence score ; Set decision threshold ,Will The samples were judged as "reliable pre-labeling", meaning the predicted labels were reliable. Conversely, it is judged as "unreliable," meaning the predicted label is not accurate. ; Calculate the precision index for category k: ， in, The number of true positive samples, i.e. The total number of samples; The number of false positive samples, i.e. The total number of samples; For all hazard categories Repeat the above calculations to obtain a list of the accuracy levels for each hazard category. .

6. A semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, The threshold in S4 The setting method is as follows: Confidence threshold The consistency ratio between the pre-annotated sample results and the actual manual annotation results under different confidence scores was obtained by analyzing historical annotation data. Suppose that the historical annotation dataset contains For each sample The confidence score output by the confidence classifier is The consistency label between the actual manual annotation results and the pre-annotation results is: The range of confidence scores Divided into each interval ,in ; Statistics for each interval Number of samples within and the number of samples with consistent labeling. Calculate the consistency ratio within each interval. : ， Above the threshold The accuracy of labeling is measured by the proportion of correctly pre-labeled samples, denoted as . Assuming Falling in the range ,but: ， Below the threshold The workload of manual annotation is measured by the proportion of the sample size to the total sample size, denoted as . : ， To balance annotation accuracy and manual annotation workload, an optimization objective function is constructed. A weighted approach is used to combine accuracy and workload: ， in, This is a weighting parameter used to adjust the relative importance of accuracy and workload; The larger the value, the higher the importance attached to the accuracy of the annotation; The smaller the size, the more emphasis is placed on reducing the amount of manual labeling work; By iterating through all possible... Value, calculate the corresponding Value, select to make Maximize As the optimal threshold: ， The optimal threshold is determined using the quantization method described above. .

7. A semi-automatic annotation method for video images of rail transit engineering construction according to claim 1, characterized in that, In step S4, the decision on whether to retrain is based on the category distribution. and The method for determining this is as follows: Let the labeled sample set be... Include There are 10 hazard categories, and the sample size for each category is... Total number of samples Define category frequency as The equilibrium index is: ,in, For uniformly distributed frequencies, ; Let the number of samples in each category of the manually labeled set M be... Define the offset: ， The new equilibrium index after labeling satisfy Or the sum of the absolute values ​​of the offsets exceeds the threshold. Retraining is triggered at certain times; among them, This is the initial equalization threshold; This is the preset offset threshold.

8. A semi-automatic annotation system for video images of rail transit engineering construction, characterized in that, include: The video keyframe acquisition unit is used to extract video keyframes from a continuous video stream by removing redundant frames with similar content to obtain a block index set. ; Multimodal large model pre-annotation units are used to construct prompt words for multimodal large models and to set up block indexes. All images are placed into the constructed prompt words to obtain the result set after pre-annotation of the large model. ,right All elements Perform binarization; The classifier training unit is used to select a set of unlabeled samples based on an active learning strategy. The sample elements in the sample set are migrated to the labeled sample set. The classifier is retrained after each transfer. until the set of tags Once the target scale is reached, this process ends; a confidence classifier is trained by comparing the consistency between the manually labeled results of the labeled samples and the pre-labeled results of the multimodal large model. ; The sample selection and labeling unit is used to employ a classifier. Unlabeled collection Screening A subset of negative samples Test confidence classifier Accuracy in each hazard category, after meeting the standard The mid-sample score, combined with the set threshold. ,Will The samples used were pre-labeled results; The samples were manually labeled and placed into a manual labeling set. ; set The sample is moved into the label set. Meanwhile, the unlabeled sample set Remove the corresponding samples; and decide whether to retrain based on the class distribution. and ; The classifier training unit selects a set of unlabeled samples. The specific method for selecting sample elements is as follows: Randomly select a set A small number of samples were manually labeled to form an aggregator. Other unlabeled samples form a set. Set the total number of queried labels. Total number of queries Then the sample size for each migration is : A multi-layer neural network structure is used as the classifier. The basic infrastructure; Will As a negative sample, Used as positive samples for training The model learns sample features to distinguish whether a sample is a key-labeled sample; it utilizes... right Using the medium sample for inference, the number of "negative samples" with a classification probability close to 0 is required to filter them out. ; The The selection criteria for negative samples are based on the classifier. The measure of sample difference is based on the sample that is judged as a negative sample by the classifier with the most obvious features. The sample with the greatest difference from the labeled sample among the unlabeled samples is selected as the object to be labeled.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor according to any one of claims 1-7, which describes a semi-automatic annotation method for video images of rail transit engineering construction.

Images