Detection and identification method for elevator inspection video, electronic equipment and storage medium

Abstract

The invention discloses a detection and identification method for an elevator inspection video, electronic equipment and a storage medium, and relates to the field of elevator video quality detection. The method comprises the following steps: predicting unlabeled data through an initial first detection identification model to generate a pseudo label sample; performing training set updating on the first training set according to the pseudo tag sample and the calculated dynamic confidence threshold, and performing model updating training processing on the first detection and recognition model based on the updated second training set to obtain an updated second detection and recognition model; calculating a dual-task joint loss value of the second detection and recognition model; performing model iteration based on the second detection and recognition model until the dual-task joint loss value converges to obtain a target detection and recognition model; and detecting and identifying the to-be-detected elevator inspection video through the target detection and identification model, and outputting a comprehensive auditing result. The manual labeling cost can be reduced by introducing a pseudo label mechanism, and the auditing efficiency and the auditing accuracy of the elevator inspection video are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of elevator video quality detection, and in particular to a method for detecting and identifying elevator inspection videos, an electronic device, and a storage medium. BACKGROUND

[0002] With the increasingly stringent requirements for elevator safety management, video records formed during elevator inspection processes have gradually become important evidence for supervision and traceability. Existing compliance determination of elevator inspection videos mostly relies on manual auditing, such as determining whether a video is abnormal or whether a video is continuous and has not been lost. However, the manual auditing method is inefficient, highly subjective, and low in accuracy, and is difficult to meet large-scale application requirements. In related technologies, image recognition means are attempted to extract key elements in elevator inspection videos, but due to high labeling costs of training data, model performance is often limited by the size and quality of the initial training set, and it is difficult to adapt to complex and variable actual inspection environments, and the recognition accuracy is limited. How to realize a low-cost, high-efficiency, and high-accuracy video auditing method is a technical problem to be solved. SUMMARY

[0003] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a method for detecting and identifying elevator inspection videos, an electronic device, and a storage medium, which reduces the cost of manual labeling by introducing a pseudo-label mechanism, and improves the auditing efficiency and accuracy of elevator inspection videos.

[0004] In a first aspect, an embodiment of the present application provides a method for detecting and identifying elevator inspection videos, comprising:

[0005] predicting and processing unlabeled data by an initial first detection and identification model to generate pseudo-label samples; wherein the first detection and identification model is obtained by model training based on an initial first training set;

[0006] updating the first training set based on the pseudo-label samples and a calculated dynamic confidence threshold to obtain an updated second training set;

[0007] updating and training the first detection and identification model based on the second training set to obtain an updated second detection and identification model;

[0008] calculating a loss value of the second detection and identification model to obtain a double-task joint loss value;

[0009] performing model iteration based on the second detection and identification model until the double-task joint loss value converges to obtain a target detection and identification model; the model iteration includes the prediction and processing, the training set updating, the model updating and training, and the loss value calculation.

[0010] The target detection and recognition model is used to detect and recognize the elevator inspection video under test, and output a comprehensive review result.

[0011] In a second aspect, embodiments of this application provide an electronic device, including at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the detection and recognition method for elevator inspection videos as described in any of the embodiments of the first aspect.

[0012] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the detection and recognition method for elevator inspection videos as described in any of the embodiments of the first aspect.

[0013] This application embodiment includes: in the process of reviewing elevator inspection videos generated during elevator inspection, firstly, a first detection and recognition model is used to predict unlabeled data to generate pseudo-label samples; wherein, the first detection and recognition model is obtained by model training based on an initial first training set; secondly, the first training set is updated based on the pseudo-label samples and the calculated dynamic confidence threshold to obtain an updated second training set; updating the training set through the introduced pseudo-label mechanism reduces the cost of manual labeling; then, the first detection and recognition model is updated and trained based on the second training set to obtain an updated second detection and recognition model; then, the second detection and recognition model is processed to calculate a loss value to obtain a dual-task joint loss value; then, the model is iterated based on the second detection and recognition model until the dual-task joint loss value converges to obtain a target detection and recognition model; the model iteration process includes: the prediction process, the training set update process, the model update training process, and the loss value calculation process; a reliable target detection and recognition model is obtained through iterative training based on the continuously updated training set; finally, the target detection and recognition model is used to detect and recognize the elevator inspection video to be tested, and a comprehensive review result is output. This object detection and recognition model is used to perform content detection and recognition on elevator inspection videos, thereby improving the efficiency and accuracy of elevator inspection video review. In other words, this application embodiment can reduce manual annotation costs and improve the efficiency and accuracy of elevator inspection video review by introducing a pseudo-label mechanism. Attached Figure Description

[0014] Figure 1This is a flowchart illustrating the steps of a method for detecting and recognizing elevator inspection videos provided in one embodiment of this application;

[0015] Figure 2 This is provided in one embodiment of the present application. Figure 1 A flowchart illustrating the specific steps of step S160.

[0016] Figure 3 This is a schematic diagram of the specific structure of a target digit recognition model provided in one embodiment of this application;

[0017] Figure 4 This is a schematic diagram of the overall process of a method for detecting and recognizing elevator inspection videos provided in one embodiment of this application;

[0018] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0020] It should be noted that although a logical order is shown in the flowcharts in this application, in some cases, the steps shown or described may be performed in a different order than that shown in the flowcharts. In the description of this application, "several" means one or more, and "more" means two or more. The terms "first" and "second" are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order in which the technical features are indicated.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] First, let me explain some of the terms used in this application:

[0023] Monte Carlo dropout: Monte Carlo dropout method; Monte Carlo refers to a probabilistic statistical method based on random sampling.

[0024] This application provides a method and electronic device, as well as a computer-readable storage medium, for detecting and recognizing elevator inspection videos, relating to the field of elevator video quality inspection. The method includes: generating pseudo-label samples by predicting unlabeled data using an initial first detection and recognition model; updating a first training set based on the pseudo-label samples and a calculated dynamic confidence threshold; updating and training the first detection and recognition model based on the updated second training set to obtain an updated second detection and recognition model; calculating the dual-task joint loss value of the second detection and recognition model; iterating the model based on the second detection and recognition model until the dual-task joint loss value converges to obtain a target detection and recognition model; and using the target detection and recognition model to detect and recognize the elevator inspection video under test and output a comprehensive review result. This method can reduce manual labeling costs by introducing a pseudo-label mechanism and improve the efficiency and accuracy of elevator inspection video review.

[0025] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0026] Firstly, such as Figure 1 As shown, the detection and recognition method for elevator inspection videos may include, but is not limited to, steps S110 to S160.

[0027] Step S110: Generate pseudo-label samples by predicting the unlabeled data using the initial first detection and recognition model; wherein, the first detection and recognition model is obtained by training the model based on the initial first training set.

[0028] Step S120: Update the first training set based on the pseudo-label samples and the calculated dynamic confidence threshold to obtain the updated second training set.

[0029] Step S130: Based on the second training set, perform model update training on the first detection and recognition model to obtain the updated second detection and recognition model.

[0030] Step S140: Perform loss value calculation on the second detection and recognition model to obtain the joint loss value of the two tasks.

[0031] Step S150: Perform model iteration processing based on the second detection and recognition model until the joint loss value of the two tasks converges to obtain the target detection and recognition model; the model iteration processing includes: prediction processing, training set update processing, model update training processing and loss value calculation processing.

[0032] Step S160: The target detection and recognition model is used to detect and recognize the elevator inspection video under test, and the comprehensive audit results are output.

[0033] To further explain step S110, specifically, the initial first detection and recognition model includes: a first content detection model and a first digit recognition model. The initial first training set includes: a small number of labeled samples.

[0034] In other words, before step S110, the initial training phase uses a small number of manually labeled samples to train an initial first content detection model and an initial first digit recognition model. Then, based on the initial first content detection model and the initial first digit recognition model, step S110 is executed to predict unlabeled data and generate pseudo-label samples, laying the data foundation for subsequent updates to the training set. It is important to emphasize that, under the initial condition of only a small number of manually labeled samples, this application uses the initial first detection and recognition model to predict unlabeled data and generate pseudo-labels. A high-quality sample set is constructed by combining confidence screening and manual correction, and the screened pseudo-labels are continuously incorporated into the training set for iterative training.

[0035] Specifically, the second detection and recognition model includes: an updated second content detection model and an updated second digit recognition model.

[0036] Specifically, the object detection and recognition model includes: a target content detection model trained at the end and a target digit recognition model trained at the end. The target digit recognition model achieves dual-task fusion detection, where the dual tasks refer to: content detection of elevator nameplates and timestamps, and content parsing and recognition of timestamps.

[0037] Specifically, the specific process of the first detection and recognition model predicting the unlabeled data in step S110 (such as frame extraction, multi-category content detection, image preprocessing, and digit recognition) is the same as the specific process of the target detection and recognition model detecting and recognizing the elevator inspection video to be tested in step S160. It will not be repeated here. Please refer to the further explanation of step S160 for details.

[0038] Specifically, the first content detection model is based on the YOLO v11n architecture; similarly, the second content detection model and the target content detection model are also based on the YOLO v11n architecture.

[0039] In the process of reviewing the elevator inspection video generated during the elevator inspection process, through steps S110 to S160, firstly, the unlabeled data is predicted using an initial first detection and recognition model to generate pseudo-label samples; the first detection and recognition model is trained based on the initial first training set. Secondly, the first training set is updated based on the pseudo-label samples and the calculated dynamic confidence threshold to obtain an updated second training set; the pseudo-label mechanism reduces the cost of manual labeling by updating the training set. Next, the first detection and recognition model is updated and trained based on the second training set to obtain an updated second detection and recognition model. Then, the loss value is calculated for the second detection and recognition model to obtain a joint loss value for both tasks. Next, the model is iterated based on the second detection and recognition model until the joint loss value for both tasks converges to obtain the target detection and recognition model; the model iteration process includes prediction processing, training set update processing, model update training processing, and loss value calculation processing; a reliable target detection and recognition model is obtained through iterative training based on the continuously updated training set. Finally, the target detection and recognition model is used to detect and recognize the elevator inspection video under test, and a comprehensive review result is output. This object detection and recognition model is used to perform content detection and recognition on elevator inspection videos, thereby improving the efficiency and accuracy of elevator inspection video review. In other words, this application embodiment can reduce manual annotation costs and improve the efficiency and accuracy of elevator inspection video review by introducing a pseudo-label mechanism.

[0040] It is important to emphasize that this application focuses on the dual-task scenario of target detection and digital recognition in elevator inspection videos. It proposes a model iterative optimization mechanism based on pseudo-labels to obtain a target detection and recognition model with stronger detection and recognition capabilities. In the elevator video compliance judgment scenario, a digital recognition model is introduced. By automatically cropping the timestamp area and performing image preprocessing, combined with convolutional neural networks to extract digital features, the structured recognition of time formats and compliance judgment are achieved.

[0041] In some embodiments, pseudo-label samples include, but are not limited to: target location, category label, and sample confidence level.

[0042] According to some embodiments of this application, the pseudo-label sample includes: sample confidence; further, step S120 is described as follows: the first training set is updated based on the pseudo-label sample and the calculated dynamic confidence threshold to obtain an updated second training set, including but not limited to steps S121 to S124.

[0043] Step S121: Calculate the dynamic confidence threshold based on the preset confidence threshold calculation formula.

[0044] Step S122: Compare the dynamic confidence threshold and the sample confidence to obtain the comparison result.

[0045] Step S123: Based on the comparison results, the pseudo-label samples are screened and new samples are determined to obtain the target new samples.

[0046] Step S124: Add the newly added target sample to the first training set to obtain the updated second training set.

[0047] Specifically, in step S121, the preset reliability threshold calculation formula is as follows:

[0048] θ(t)=θ min +(θ max -θ min )·e -kt ;

[0049] Where t is the training round; θ(t) is the dynamic confidence threshold used to filter pseudo-labeled samples in the t-th training round; θ max θ is the preset upper limit threshold; min e is the lower limit of the preset threshold. -kt It is a decay function with base e as the base of the natural logarithm; k is the decay coefficient, and k is greater than 0.

[0050] Understandably, the dynamic confidence threshold used in each round of model iteration is different, so that suitable target new samples can be better selected from pseudo-labeled samples based on the dynamic confidence threshold.

[0051] Specifically, in step S122, the comparison results include: the sample confidence level is greater than or equal to the preset confidence threshold, and the sample confidence level is less than the preset confidence threshold.

[0052] Specifically, steps S123 and S124 are further explained. The target new samples include: a first new sample and a second new sample. Step S123: Based on the comparison results, the pseudo-label samples are screened and new samples are determined to obtain the target new samples, including but not limited to the following steps: samples with a confidence level greater than or equal to a preset confidence threshold are determined as the first new samples; samples with a confidence level less than the preset confidence threshold are determined as samples to be labeled; and samples to be labeled with a probability greater than a preset correction probability are corrected to obtain the second new sample. Then, step S124 is executed to add the first new sample and / or the second new sample to the first training set to obtain the updated second training set.

[0053] Specifically, let's take an example to illustrate the specific process of training set update.

[0054] First, using the current detection and recognition model f θ(·) For unlabeled data x j Generate pseudo-labels and calculate the dynamic confidence threshold θ(t). Classify the labels according to the following rules to obtain updated labels.

[0055] Wherein, softmax(f θ (x j )) refers to the detection and recognition model f θ (·) For unlabeled data x j The confidence level of the generated pseudo-label samples.

[0056] Then, for the labels that need to be manually labeled Calculate the uncertainty score. The formula for calculating the uncertainty score is:

[0057]

[0058] Where, U(x) j ) is the sample x j The uncertainty score is given by K, where K is the number of Monte Carlo dropout samples and k is the k-th training iteration. For the Kth sampling, the model is applied to sample x. j The prediction results are given by mode(·), which is the mode function that returns the class that appears most frequently in the K predictions, and I(·), which is the indicator function that returns 1 when the condition in parentheses is true, and 0 otherwise.

[0059] Next, samples with confidence levels lower than a preset confidence threshold are identified as samples to be labeled (i.e., pseudo-labeled samples to be labeled). When correcting samples to be labeled, a manual correction probability p is defined. h (x j ):

[0060] Where λ is a pre-configured sensitivity adjustment parameter; when p h (x j Manual correction is triggered when p > 0.5; otherwise, p h (x j If the value is less than or equal to 0.5, then manual correction will not be triggered for the sample to be labeled.

[0061] Through steps S121 to S124, unlabeled data is predicted and pseudo-labels are generated. These pseudo-labels are then automatically filtered using an established dynamic confidence threshold filtering mechanism, reducing manual annotation costs and constructing a high-quality label sample set. Furthermore, manual correction is incorporated to further enrich the high-quality label sample set. The training set is then updated using this high-quality label sample set, enriching the training data and facilitating further improvement in model accuracy. Therefore, this application reduces manual annotation costs by introducing a pseudo-label mechanism to update the training set. By adding the filtered pseudo-labels to the current training set to obtain an updated training set, this application uses an incremental training method to iteratively optimize the detection and recognition model, thereby improving the model's detection and recognition capabilities.

[0062] According to some embodiments of this application, the second detection and recognition model includes: a second content detection model and a second digit recognition model. Further explanation of step S140: Step S140: Perform loss value calculation processing on the second detection and recognition model to obtain a joint loss value for both tasks, including but not limited to steps S141 to S143.

[0063] Step S141: Calculate the detection loss value of the second content detection model based on the first preset loss function.

[0064] Step S142: The recognition loss value of the second digit recognition model is calculated by using a second preset loss function that minimizes the cross-entropy loss function based on supervised learning.

[0065] Step S143: Multiply the detection loss value by the preset first training task weight coefficient to obtain the first product value; multiply the recognition loss value by the preset second training task weight coefficient to obtain the second product value; add the first product value and the second product value to obtain the dual-task joint loss value.

[0066] Specifically, the preset loss function in step S141 is an improved Focal Loss calculation function, and the specific expression of the first preset loss function is as follows:

[0067]

[0068] Among them, L det γ is the detection loss value of the second content detection model; γ is the hard sample adjustment factor (γ≥2), s i To predict confidence levels, M i y represents the total number of candidate boxes in a single batch of training samples. i Let be the true label of the i-th candidate box.

[0069] Specifically, the expression for the second preset loss function in step S142 is as follows:

[0070]

[0071] Among them, L rec y represents the recognition loss value of the second digit recognition model. i For real labels, Let N be the predicted probability of the current detection and recognition model for the i-th type of content, and N be the total number of categories of the content to be detected.

[0072] Specifically, the joint loss function for the two tasks used in step S143 is:

[0073] L=α·L de t(θ d )+β·L rec (θ r );

[0074] Where, θ d θ represents the detection network parameters of the content detection model. r L represents the recognition network parameters of the digit recognition model. det (θ d L refers to the detection loss value of the content detection model under the current detection network parameters; rec (θ r ) refers to the recognition loss value of the digit recognition model under the current recognition network parameters; α is the preset weight coefficient of the first training task; β is the preset weight coefficient of the second training task; and α+β=1.

[0075] Through steps S141 to S143, the joint loss value of the two tasks is calculated, which quantifies the deviation between the model prediction result and the true value, thereby guiding the optimization direction of the model parameters; and providing a reference for ending the model iteration to obtain the target detection and recognition model.

[0076] Further explanation of step S150: Specifically, firstly, based on the second detection and recognition model, the prediction processing shown in step S110 is performed to generate pseudo-label samples; then, the training set update processing shown in step S120 is performed to obtain an updated third training set; next, based on the updated third training set, the model update training processing shown in step S130 is performed to obtain an updated third detection and recognition model, and the loss value calculation processing in step S150 is performed to calculate the joint loss value of the third detection and recognition model for both tasks; thus, one model iteration is completed. This process is repeated multiple times until the joint loss value for both tasks converges, ending the model iteration and outputting the target detection and recognition model (including: target content detection model and target digit recognition model); model iteration based on continuously updated training sets improves the accuracy of the target detection and recognition model in reviewing applications.

[0077] According to some embodiments of this application, the target detection and recognition model includes: a target content detection model and a target digit detection model; combined with Figure 2 Further explanation of step S160: Step S160: The target detection and recognition model is used to detect and recognize the elevator inspection video to be tested, and output the comprehensive review result, including but not limited to steps S210 to 250.

[0078] Step S210: Perform frame extraction processing on the elevator inspection video to be tested to obtain multiple frame images to be tested.

[0079] Step S220: Input each frame image to be tested into the target content detection model for multi-class content detection processing, and output the content detection results including multi-class target content; the content detection results include: a set of timestamp regions and a set of nameplate regions in the frame image to be tested; wherein, the set of timestamp regions includes multiple timestamp region images; the set of nameplate regions includes multiple nameplate region images.

[0080] Step S230: Perform image preprocessing on the cropped timestamp region image to obtain a preprocessed enhanced timestamp region image.

[0081] Step S240: Input the enhanced timestamp region images of multiple preprocessed frames into the target digit detection model, perform continuous frame timestamp recognition processing, and obtain the timestamp verification result.

[0082] Step S250: Generate and output the comprehensive review result based on the content detection result and the timestamp verification result.

[0083] Specifically, step S210 further includes: extracting frames from the video to be detected according to a preset frame extraction interval, assuming the original video frame rate is F. v (frames / second), preset frame extraction interval is T. s (seconds), then the set of extracted frame indices is: Where N is the total number of frames in the elevator inspection video to be tested. f is a function that rounds down; i To extract the i-th frame image to be tested.

[0084] According to some embodiments of this application, step S220 is further described. Step S220: Each frame image to be tested is input into the target content detection model for multi-type content detection processing, and the content detection results of multi-type targets are output, including but not limited to steps S221 to S224.

[0085] Step S221: Input each test frame image into the pre-trained target detection model for multi-class detection processing, and output an initial candidate box set for all target regions in each test frame image; wherein, the initial candidate box set includes multiple candidate boxes, the coordinate information corresponding to the candidate boxes, the category label, and the confidence score; the category label is the nameplate label and the timestamp label.

[0086] Step S222: Filter all candidate boxes according to the preset minimum confidence threshold, remove candidate boxes with confidence scores less than the preset minimum confidence threshold, and generate a target candidate box set based on the remaining candidate boxes.

[0087] Step S223: Classify and store the remaining candidate boxes in the target candidate box set according to the category label to obtain the set of all detected timestamp regions and nameplate regions in the frame image.

[0088] Step S224: When the timestamp region set and the nameplate region set are detected, generate the content detection result based on the timestamp region set and the nameplate region set.

[0089] Further explaining step S221, specifically, step S221 includes: obtaining the extracted test frame image f i ∈R H ×W×3 Then, where R refers to the set of real numbers, H refers to the height of the frame image to be tested, and W refers to the width of the frame image to be tested; the frame image to be tested f i The input is fed into the target content detection model, which performs multi-class target detection and outputs an initial candidate box set for all target regions in each frame of the test image; the initial candidate box set is denoted as:

[0090]

[0091] Among them, (b) j c j s j ) refers to the j-th candidate box; b j This refers to the coordinate information corresponding to the j-th candidate box. Specifically, Represents the coordinates of the top-left corner of the j-th candidate box: Bottom right corner coordinates: Among them, c j This refers to the category label corresponding to the j-th candidate box; specifically, c j ∈C; category set C = {"nameplate", "timestamp"}; s j This refers to the confidence score corresponding to the j-th candidate box; s j ∈[0,1] represents the confidence score of the corresponding category.

[0092] Further explaining step S222, specifically, step S222 includes: First, setting a minimum confidence threshold S0 for all candidate box sets. After initial filtering, a set of candidate boxes should be retained. To save subsequent computing resources; M i is the total number of candidate boxes detected in the i-th frame.

[0093] Further explaining step S223, specifically, step S223 includes: storing the remaining candidate boxes by category to obtain a set of all detected timestamp regions in the frame image. Collection of nameplate areas Right now:

[0094]

[0095] Among them, f i [b j ] indicates that in the frame image f to be tested i The remaining candidate box coordinate information b j The image of the region cropped out by the boundary.

[0096] To further explain step S224, the detected timestamp region set and nameplate region set are output, that is, the content detection result is generated and output.

[0097] Through steps S221 to S224, content detection results for multiple target types can be output, thus completing the content detection task.

[0098] According to some embodiments of this application, step S230 is further described. Step S230: Image preprocessing is performed on the cropped timestamp region image to obtain a preprocessed enhanced timestamp region image, including but not limited to steps S231 to S232.

[0099] Step S231: Perform grayscale processing on the timestamp region image to obtain a grayscale timestamp region image.

[0100] Step S232: Perform noise reduction and enhancement processing on the grayscale timestamp region image based on the median filtering algorithm to obtain the enhanced timestamp region image.

[0101] Specifically, step S231 further explains that the coordinate information of the candidate bounding boxes in the timestamp region output by the target content detection model is obtained. Then, the cropped timestamp region image I tsPreprocessing is performed to improve the accuracy of digital recognition. The timestamp region image Its is sequentially processed by grayscale conversion and noise reduction / enhancement. For grayscale conversion, the formula used to convert the color image to a grayscale image P is: P = 0.299R + 0.587G + 0.114B; where R is the red channel value of a pixel in the color image, G is the green channel value of a pixel in the color image, and B is the blue channel value of a pixel in the color image. For noise reduction / enhancement, a median filtering algorithm is used. Assuming the filtering window size is k*k, for a pixel at coordinates (x, y) in the image, its filtered value is the median of the grayscale values ​​of all pixels within the window. Where median{·} is used for median calculation.

[0102] Steps S231 to S232 are used to perform grayscale processing and filtering enhancement on the cropped timestamp region image to obtain an enhanced timestamp region image, which improves the accuracy of subsequent digital sequence recognition.

[0103] According to some embodiments of this application, step S240 is further described. Step S240: The preprocessed and enhanced timestamp region image is input into the target digit detection model for continuous frame timestamp recognition processing to obtain timestamp verification results, including but not limited to steps S241 to S243.

[0104] Step S241: Input the multi-frame enhanced timestamp region image into the target digit detection model for digit sequence format recognition processing to obtain multiple time digit sequences.

[0105] Step S242: Perform format verification on each time number sequence to obtain the format verification result; wherein, the format verification result is: the timestamp format is correct or the timestamp format is incorrect.

[0106] Step S243: If no correctly formatted timestamps are detected within a preset number of consecutive frames, the timestamp verification result is determined to be: there are no consecutive qualified timestamps in the elevator inspection video to be tested; if correctly formatted timestamps are detected within a preset number of consecutive frames, the timestamp verification result is determined to be: there are consecutive qualified timestamps in the elevator inspection video to be tested.

[0107] Specifically, the target digit recognition model M involved in step S240 will be further explained. rec The target digit recognition model M rec Based on a Convolutional Neural Network (CNN) architecture, it consists of multiple convolutional layers and pooling layers, used to extract local features of an image. Specifically, such as... Figure 3 As shown, the target digit recognition model M recAn enhanced convolutional neural network structure is employed, consisting of three convolutional modules. Each module includes a 3×3 convolutional layer, a batch normalization layer, a ReLU activation function layer, and a 2×2 max pooling layer. The number of output channels for the convolutional layers in the three modules are 64, 128, and 256, respectively. After convolutional feature extraction, a spatial attention mechanism is introduced to enhance the model's ability to focus on digit regions. Finally, the recognition result is output through a fully connected layer.

[0108] Further explanation of steps S241 to S243. The preprocessed timestamp region image... Input to digit recognition model M rec First, the time-digit sequence is extracted and its format is checked; simultaneously, continuous frame detection is performed; finally, the timestamp verification result is output. Where 0 represents that the frame image does not have a valid timestamp with a correct format (i.e., the timestamp format is incorrect), and 1 represents that the frame image has a valid timestamp with a correct format.

[0109] It is understandable that a fault-tolerant mechanism based on continuous frame detection results is introduced to enhance the system's fault tolerance. Let the sequence of continuously extracted keyframes from the elevator inspection video be:

[0110] Where T is the total number of extracted keyframes. If no matching timestamp is detected in any of the n consecutive frames, the timestamp information of the video segment is deemed non-compliant, expressed using the indicator function I(·):

[0111] The indicator function I(·) returns 1 if the condition within the parentheses is true, and 0 otherwise. When N = 0, it means that no matching timestamps have been detected in n consecutive frames; when N is not equal to 0, it means that a matching timestamp has been detected in n consecutive frames.

[0112] Understandably, if the timestamps meet the requirements (correct format and continuous), the video capture is considered compliant; if the timestamps do not meet the requirements, the video capture is considered non-compliant.

[0113] Through steps S210 to S250, the area corresponding to the timestamp is cropped and then input into the digital recognition model to extract the time digital sequence, determine whether it meets a specific format, and whether a correctly formatted timestamp is detected within a preset number of consecutive frames, so as to assist in determining whether the video capture is compliant; and complete the task of parsing and recognizing the content of the timestamp.

[0114] Combination Figure 4 The following example illustrates the overall process and effect of the elevator inspection video detection and recognition method provided in this application. It should be noted that... Figure 4This paper mainly demonstrates the process of optimizing and iteratively updating the model based on pseudo-label samples and fusing dual-task detection in the elevator inspection video detection and recognition method proposed in this application.

[0115] Step S401, initial training phase, using the first training set (including a small number of manually labeled samples) to train the model to obtain the first detection model and the first digit recognition model.

[0116] Step S402: The first detection model and the first digit recognition model are used to predict the unlabeled keyframe image and automatically generate pseudo-label samples including target location, category label and confidence level.

[0117] Step S403: The accuracy of pseudo-label samples is screened and new samples are determined by the calculated dynamic confidence threshold. Only pseudo-labels with high confidence or after correction are retained as target new samples.

[0118] Step S404: The target is to add the new sample to the first training set to obtain the updated second training set.

[0119] Step S405: Train the first detection model and the first digit recognition model based on the second training set to obtain the second detection model and the second digit recognition model; perform one model iteration through steps S402 to S405; then perform multiple model iterations based on the second detection model and the second digit recognition model (repeating steps S402 to S405 multiple times). When the joint loss value of the two tasks converges, output the finally trained object detection and recognition module (including: object detection model and object digit recognition model). It can be understood that in each round of model iteration, the model is retrained using the updated training set to continuously improve the model's detection and recognition capabilities in complex scenarios.

[0120] Step S406: Obtain the elevator inspection video to be tested.

[0121] Step S407: The elevator inspection video to be tested is input into the target detection and recognition module. Specifically, in the target detection and recognition module, firstly, the first detection model performs multi-class detection processing on the elevator inspection video to obtain a set of timestamp regions and a set of nameplate regions. The timestamp region image is cropped and preprocessed to obtain a preprocessed timestamp region image. The preprocessed timestamp region image is input into the target digit recognition model, and a timestamp verification result is obtained through continuous frame timestamp recognition processing. Based on the timestamp verification result and the nameplate region set, a comprehensive review result is output. Therefore, ultimately, the trained target detection model is used to determine whether the video contains nameplate and timestamp regions. Combined with the timestamp content output by the target digit recognition model, the compliance of the timestamp format is judged, thereby achieving compliance determination of the elevator inspection video in terms of both image elements and time information, ensuring that the video meets the inspection standards and evidence preservation requirements.

[0122] As can be seen, this application adds the filtered pseudo-labels to the current training set to obtain an updated training set, and iteratively optimizes the detection and recognition model based on the updated training set. The final model outputs a comprehensive result indicating whether the elevator inspection video has a nameplate and timestamp, and whether the time format is compliant.

[0123] The model training process in this application is carried out iteratively. In each iteration, the model absorbs new pseudo-label information, gradually improving its ability to detect nameplate and timestamp areas in complex environments, as well as the recognition accuracy of timestamp digital sequences, and finally achieving a highly robust judgment on the compliance of elevator inspection video images.

[0124] This method jointly models an object detection model and an object digit recognition model, enabling simultaneous completion of two tasks: locating elevator nameplates and timestamps, detecting content, and parsing and recognizing the content of the timestamps. Furthermore, it introduces a pseudo-label mechanism to reduce manual annotation costs. By training the model with a small number of manually labeled samples in the initial stage, combined with pseudo-label generation and confidence filtering strategies, the training data scale can be efficiently expanded, significantly reducing the workload of manual annotation. It supports iterative optimization and format judgment, improving system robustness. By continuously adding filtered pseudo-labels to the training set, the detection and recognition models are iteratively optimized, allowing the model to continuously improve its performance through multiple rounds of training.

[0125] like Figure 5 As shown, the present invention also provides an electronic device, comprising:

[0126] The processor 501 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0127] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 to execute the elevator inspection video detection and recognition method of the embodiments of this application.

[0128] The input / output interface 503 is used to implement information input and output;

[0129] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0130] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0131] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0132] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method for detecting and recognizing elevator inspection videos.

[0133] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0135] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by this application.

Claims

1. A method for detecting and recognizing elevator inspection videos, characterized in that, include: The initial first detection and recognition model is used to predict and generate pseudo-label samples from unlabeled data; wherein, the first detection and recognition model is obtained by training the model based on the initial first training set; The first training set is updated based on the pseudo-labeled samples and the calculated dynamic confidence threshold to obtain the updated second training set. Based on the second training set, the first detection and recognition model is updated and trained to obtain the updated second detection and recognition model. The second detection and recognition model is processed to calculate the loss value, and the joint loss value of the two tasks is obtained. Based on the second detection and recognition model, model iteration processing is performed until the joint loss value of the two tasks converges to obtain the target detection and recognition model; the model iteration processing includes: the prediction processing, the training set update processing, the model update training processing, and the loss value calculation processing; The target detection and recognition model is used to detect and recognize the elevator inspection video under test, and output a comprehensive review result.

2. The method for detecting and recognizing elevator inspection videos according to claim 1, characterized in that, The pseudo-label samples include: sample confidence levels; The step of updating the first training set based on the pseudo-labeled samples and the calculated dynamic confidence threshold to obtain the updated second training set includes: The dynamic confidence threshold is calculated based on a pre-set confidence threshold calculation formula. The comparison result is obtained by comparing the dynamic confidence threshold and the sample confidence. Based on the comparison results, the pseudo-label samples are screened and new samples are determined to obtain the target new samples. The newly added target samples are added to the first training set to obtain the updated second training set.

3. The method for detecting and recognizing elevator inspection videos according to claim 2, characterized in that, The formula for calculating the preset confidence threshold is: θ(t)=θ min +(θ max -θ min )·e -kt ; Where t is the training round; θ(t) is the dynamic confidence threshold used to filter the pseudo-labeled samples in the t-th training round; θ max θ is the preset upper limit threshold; min The lower limit of the preset threshold; e -kt It is a decay function with base e as the base of the natural logarithm; k is the decay coefficient, and k is greater than 0.

4. The method for detecting and recognizing elevator inspection videos according to claim 1, characterized in that, The second detection and recognition model includes: a second content detection model and a second digit recognition model; The step of calculating the loss value of the second detection and recognition model to obtain the joint loss value of the two tasks includes: Based on the first preset loss function, the detection loss value of the second content detection model is calculated; The recognition loss value of the second digit recognition model is calculated by using a second preset loss function that minimizes the cross-entropy loss function based on supervised learning. The detection loss value is multiplied by a preset first training task weight coefficient to obtain a first product value; the recognition loss value is multiplied by a preset second training task weight coefficient to obtain a second product value; the first product value and the second product value are added together to obtain the dual-task joint loss value.

5. The method for detecting and recognizing elevator inspection videos according to claim 1, characterized in that, The target detection and recognition model includes: a target content detection model and a target digit detection model; The process involves using the target detection and recognition model to detect and identify the elevator inspection video under test, and outputting a comprehensive review result, including: The elevator inspection video to be tested is subjected to frame extraction processing to obtain multiple test frame images; Each of the test frame images is input into the target content detection model for multi-category content detection processing, and the output includes content detection results of multiple types of target content; the content detection results include: a set of timestamp regions and a set of nameplate regions in the test frame image; wherein, the set of timestamp regions includes multiple timestamp region images; the set of nameplate regions includes multiple nameplate region images; The cropped timestamp region image is preprocessed to obtain a preprocessed enhanced timestamp region image. The enhanced timestamp region image after multi-frame preprocessing is input into the target digit detection model to perform continuous frame timestamp recognition processing to obtain timestamp verification results. Based on the content detection results and the timestamp verification results, the comprehensive review results are generated and output.

6. The method for detecting and recognizing elevator inspection videos according to claim 5, characterized in that, The step of inputting each of the test frame images into the target content detection model for multi-class content detection processing and outputting multi-class target content detection results includes: Each of the test frame images is input into a pre-trained target detection model for multi-class detection processing, and an initial candidate box set for all target regions in each test frame image is output; wherein, the initial candidate box set includes multiple candidate boxes, the coordinate information corresponding to the candidate boxes, category labels, and confidence scores; the category labels are nameplate labels and timestamp labels; All candidate boxes are filtered according to a preset minimum confidence threshold, and candidate boxes with confidence scores lower than the preset minimum confidence threshold are removed. A target candidate box set is generated based on the remaining candidate boxes. The remaining candidate boxes in the target candidate box set are classified and stored according to the category label to obtain the set of all detected timestamp regions and the set of nameplate regions in the frame image; When the timestamp region set and the nameplate region set are detected, the content detection result is generated based on the timestamp region set and the nameplate region set.

7. The method for detecting and recognizing elevator inspection videos according to claim 5, characterized in that, The step of inputting the preprocessed and enhanced timestamp region image into the target digit detection model for continuous frame timestamp recognition processing to obtain timestamp verification results includes: Multiple frames of the enhanced timestamp region images are input into the target digit detection model for digit sequence format recognition processing to obtain multiple time digit sequences; Each of the aforementioned time-digit sequences is subjected to format verification to obtain a format verification result; wherein, the format verification result is: the timestamp format is correct or the timestamp format is incorrect; If no correctly formatted timestamp is detected within a preset number of consecutive frames, the timestamp verification result is determined to be: there are no consecutive qualified timestamps in the elevator inspection video to be tested; if correctly formatted timestamps are detected within the preset number of consecutive frames, the timestamp verification result is determined to be: there are consecutive qualified timestamps in the elevator inspection video to be tested.

8. The method for detecting and recognizing elevator inspection videos according to claim 5, characterized in that, The step of preprocessing the cropped timestamp region image to obtain a preprocessed enhanced timestamp region image includes: The timestamp region image is converted to grayscale to obtain a grayscale timestamp region image; The grayscale timestamp region image is subjected to noise reduction and enhancement processing based on the median filtering algorithm to obtain an enhanced timestamp region image.

9. An electronic device, characterized in that, It includes at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the detection and identification method for elevator inspection videos as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the detection and identification method for elevator inspection videos as described in any one of claims 1 to 8.

Images