Human time auditing method, device and equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a man-hour auditing method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a to-be-audited video; inputting each to-be-audited image into a target detection model, and obtaining a safety helmet detection result of each to-be-audited image and a personnel detection result of each to-be-audited image output by the target detection model; determining the number of auditing personnel based on each safety helmet detection result and each personnel detection result; based on each to-be-audited image, determining the earliest time of the to-be-audited video and the latest time of the to-be-audited video through an optical character recognition model; and determining the auditing duration based on the earliest time and the latest time. Through the mode, human errors can be prevented from being introduced in the auditing process, the accuracy of the auditing result can be effectively improved, the time cost and the labor cost are reduced, and the auditing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a human-time auditing method, apparatus, equipment, storage medium, and program product. Background Technology

[0002] For maintenance work orders categorized into five scenarios—project on-site support, line engineering acceptance, special monitoring, line protection publicity, and emergency communication support—all require a video receipt attached to the maintenance return form to prove the number of audit personnel participating in the task, as well as the start and end times of the task. For these personnel-time-based work orders, the return video must be taken and uploaded at the start of the task, and the video must include the faces of all maintenance personnel involved to facilitate headcount identification and determine the audit duration.

[0003] Existing methods for auditing personnel and time mostly rely on manual identification of the number of people being audited and the duration of audits in videos. Each video requires manual identification and recording, and if omissions occur, it is difficult to accurately count personnel and durations. In addition, if the video duration is too long, it requires huge time and manpower costs.

[0004] Therefore, the existing human time auditing methods have low accuracy and low efficiency. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for auditing personnel and time, in order to solve the technical problems of low accuracy and low efficiency of existing auditing methods for personnel and time.

[0006] In a first aspect, embodiments of this application provide a method for auditing personnel and time, comprising: acquiring a video to be audited; the video to be audited is obtained by filming audit personnel, and the video to be audited includes multiple images to be audited; inputting each image to be audited into a target detection model to obtain the safety helmet detection result and the personnel detection result of each image to be audited output by the target detection model; determining the number of audit personnel based on each safety helmet detection result and each personnel detection result; determining the earliest time and the latest time of the video to be audited based on each image to be audited using an optical character recognition model; and determining the audit duration based on the earliest time and the latest time.

[0007] In one embodiment, before inputting each image to be audited into the target detection model to obtain the helmet detection result and personnel detection result of each image to be audited, the method further includes: acquiring multiple images to be labeled; the images to be labeled are photos of personnel at the work site; personnel bounding box annotation and personnel category label annotation are performed on the personnel in each image to be labeled, and helmet bounding box annotation and helmet category label annotation are performed on the helmets in each image to be labeled, to obtain multiple sample images, sample personnel detection results corresponding to each sample image, and sample helmet detection results corresponding to each sample image; based on the multiple sample images, the sample personnel detection results corresponding to each sample image, and the sample helmet detection results corresponding to each sample image, the initial model built based on the improved Yolov10 algorithm is trained to obtain the target detection model.

[0008] In one embodiment, acquiring multiple images to be labeled includes: acquiring a sample video sequence; the sample video sequence is obtained by filming personnel at a work site, and the sample video sequence includes multiple image frames; determining multiple inter-frame similarities based on the multiple image frames; each inter-frame similarity is determined based on the distance between any two image frames in the sample video sequence; clustering the multiple image frames based on the multiple inter-frame similarities to obtain a first clustering result; the first clustering result includes multiple first clusters, and each first cluster includes at least one image frame; determining the first cluster center of each first cluster, and iteratively optimizing each first cluster center based on a genetic algorithm to obtain multiple optimized cluster centers; calculating the distance between each image frame and each optimized cluster center, and clustering the multiple image frames based on the minimum distance criterion to obtain a second clustering result; the second clustering result includes multiple second clusters, and each second cluster includes at least one image frame; determining the target image frame of each second cluster, and using the multiple target image frames as multiple images to be labeled; each target image frame is the image frame in each second cluster that is closest to the second cluster center of the second cluster.

[0009] In one embodiment, the helmet detection result includes the helmet bounding box coordinates, and the personnel detection result includes the personnel bounding box coordinates. Based on each helmet detection result and each personnel detection result, determining the number of auditors includes: determining the intersection-union ratio (IU) of helmets and personnel in each image to be audited, and the relative positions between helmets and personnel in each image to be audited, based on the helmet bounding box coordinates and personnel bounding box coordinates of each image to be audited; determining the auditors in each image to be audited based on the IU and the relative positions between helmets and personnel in each image to be audited; and determining the number of auditors based on the auditors in each image to be audited.

[0010] In one embodiment, based on each image to be audited, the earliest time and latest time of the video to be audited are determined using an optical character recognition (OCR) model. This includes: preprocessing each image to be audited to obtain multiple images to be recognized; inputting each image to be recognized into the OCR model to obtain the text recognition result of each image output by the OCR model; performing matching filtering on each text recognition result based on regular expressions to obtain target strings that conform to the time format in each image to be recognized; and determining the earliest time and latest time of the video to be audited based on each target string. The OCR model is a model built based on the ChineseOCR algorithm.

[0011] In one embodiment, image preprocessing is performed on each image to be audited to obtain multiple images to be identified, including: performing grayscale processing on each image to be audited to obtain multiple grayscale images; performing binarization processing on each grayscale image to obtain multiple black and white images; performing denoising processing on each black and white image to obtain multiple denoised images; performing tilt correction processing on each denoised image to obtain multiple corrected images; performing size normalization processing on each corrected image to obtain multiple standard images; and performing contrast enhancement processing on each standard image to obtain multiple images to be identified.

[0012] Secondly, embodiments of this application provide a personnel auditing device, comprising: an acquisition module for acquiring a video to be audited; the video to be audited is obtained by filming auditing personnel, and the video to be audited includes multiple images to be audited; a target detection module for inputting each image to be audited into a target detection model to obtain the safety helmet detection result and the personnel detection result of each image to be audited output by the target detection model; a personnel quantity determination module for determining the number of auditing personnel based on each safety helmet detection result and each personnel detection result; a text character recognition module for determining the earliest time and the latest time of the video to be audited based on each image to be audited using an optical character recognition model; and an audit duration determination module for determining the audit duration based on the earliest time and the latest time.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described human auditing methods.

[0014] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the aforementioned real-time auditing methods.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described human auditing methods.

[0016] The human-time auditing method, apparatus, equipment, storage medium, and program products provided in this application embodiment eliminate the need for manual identification of auditors in videos. Instead, they use a target detection model to identify each image to be audited in the video, obtaining the safety helmet detection result and personnel detection result for each image. Based on the safety helmet detection result and personnel detection result for each image, the number of auditors is determined, avoiding the introduction of human error in the auditing process and effectively improving the accuracy of the audit results. While using the target detection model to determine the number of auditors, the audit duration is determined using an optical character recognition model, eliminating the need for manual duration statistics, reducing time and labor costs, and improving audit efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the human time auditing method provided in the embodiments of this application.

[0019] Figure 2 This is the second flowchart of the personnel-time audit method provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram of the calculation process for the intersection-union ratio provided in the embodiments of this application.

[0021] Figure 4 This is a flowchart illustrating the audit duration calculation method provided in the embodiments of this application.

[0022] Figure 5 This is a schematic diagram of the structure of the human time audit device provided in the embodiments of this application.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Please see Figure 1 , Figure 1 This is one of the flowcharts illustrating the personnel and time auditing method provided in this application embodiment. In this application embodiment, the personnel and time auditing method includes steps S110 to S150, each step being as follows: S110: Obtain the video to be audited.

[0026] The video to be audited is taken by the auditors and includes multiple images to be audited.

[0027] Please see Figure 2 , Figure 2 This is the second flowchart of the personnel-time audit method provided in the embodiments of this application.

[0028] Specifically, such as Figure 2 As shown, before conducting a manpower and time task audit, it is necessary to distribute manpower and time-related tasks: the maintenance supervisor can distribute manpower and time-related tasks through the system. When distributing a task, information such as task title, task description, task specialty, task type, task content, required number of people, and time is required.

[0029] The maintenance supervisor can assign tasks according to five types of work orders (project follow-up and coordination, line project acceptance, dedicated line monitoring, line protection publicity and emergency communication support) corresponding to different scenarios and specialties (transmission lines, indoor distribution, base stations, group customer dedicated lines and home broadband). The assigned specialties and task types are shown in Table 1.

[0030] Table 1

[0031] Furthermore, obtain the audit videos corresponding to the issued tasks.

[0032] Specifically, the APP can be used to collect and process on-site audit videos, which can ensure the authenticity of on-site personnel and their work.

[0033] S120: Input each image to be audited into the target detection model to obtain the safety helmet detection result and the personnel detection result of each image to be audited.

[0034] Specifically, each image to be audited is input into a pre-trained target detection model, which performs target detection on each image and outputs the safety helmet detection result and the personnel detection result for each image.

[0035] The results of the safety helmet inspection include the coordinates of the safety helmet bounding box, and the results of the personnel inspection include the coordinates of the personnel bounding box.

[0036] Preferably, the target detection model can output the center point coordinates, height, width, and confidence score of each helmet bounding box, as well as the center point coordinates, height, width, and confidence score of each person bounding box. For each helmet or person bounding box, it is determined whether its confidence score is greater than or equal to a preset threshold (e.g., 0.92). If its confidence score is greater than or equal to the preset threshold, the bounding box is considered reliable, and the prediction result can be retained; if its confidence score is less than the preset threshold, the bounding box is considered unreliable, and the prediction result is not retained. Filtering the detection results based on confidence score ensures high recall and accuracy of the model.

[0037] Optionally, the object detection model is a model built based on the improved Yolov10 algorithm / SSD algorithm / Faster R-CNN algorithm.

[0038] S130: Determine the number of auditors based on the inspection results of each safety helmet and each person.

[0039] Please see Figure 3 , Figure 3 This is a schematic diagram of the calculation process for the intersection-union ratio provided in the embodiments of this application.

[0040] Specifically, for each image to be audited, the bounding box coordinates of the safety helmet and the bounding box coordinates of the personnel in the image to be audited can be predicted using the target detection model. Then, based on the bounding box coordinates of the safety helmet and the bounding box coordinates of the personnel, the intersection-union ratio (IOU) of the safety helmet and the personnel in the image to be audited can be calculated, and the relative positions between the safety helmet and the personnel in the image to be audited can be determined.

[0041] like Figure 3 As shown, assuming that the safety helmet bounding box A and the personnel bounding box B are both rectangles, A∩B represents the intersection of the safety helmet bounding box A and the personnel bounding box B, and A∪B represents the union of the safety helmet bounding box A and the personnel bounding box B. Then, IOU is the intersection area divided by the union area.

[0042] Furthermore, based on the crossover ratio of the safety helmet and the person in the image to be audited, and the relative position between the safety helmet and the person in the image to be audited, the audit personnel in each image to be audited are determined.

[0043] Specifically, the correlation between the safety helmet and the person is determined based on the intersection-union ratio (IOU) and relative position: if the IOU exceeds a set threshold (e.g., 80%), and the center point of the safety helmet (i.e., the coordinates of the safety helmet bounding box) is above the center point of the person (i.e., the coordinates of the person bounding box), then the person is considered to be wearing a safety helmet, and the person is the auditor, that is, the auditor is the person wearing a safety helmet.

[0044] For each image to be audited, the auditors in that image can be identified using the methods described above.

[0045] Furthermore, the number of auditors is determined based on the auditors in each image to be audited.

[0046] Specifically, the number of auditors is determined by counting the number of people wearing safety helmets identified in each image to be audited.

[0047] S140: Based on each image to be audited, determine the earliest time and the latest time of the video to be audited using an optical character recognition model.

[0048] Understandably, each frame of the video to be audited contains text (such as video recording time, playback time, location, watermark, etc.).

[0049] At this point, for each image in the video to be audited, text detection and recognition can be performed using an optical character recognition model to extract the earliest time (i.e., video start time) and the latest time (i.e., video end time) of the video to be audited.

[0050] S150: Determine the audit duration based on the earliest and latest times.

[0051] Specifically, the audit duration is calculated using the following formula: Duration = EndTime - StartTime; Where Duration represents the audit duration; EndTime is the latest time (i.e., video end time) of all videos to be audited in an audit work order; StartTime is the earliest time (i.e., video start time) of all videos to be audited in an audit work order.

[0052] The human-time auditing method provided in this application does not require manual identification of auditors in videos. Instead, it uses a target detection model to identify each image to be audited in the video, obtaining the safety helmet detection result and personnel detection result for each image. Based on the safety helmet detection result and personnel detection result for each image, the number of auditors is determined, which avoids introducing human error into the auditing process and effectively improves the accuracy of the audit results. While using the target detection model to determine the number of auditors, the audit duration is determined using an optical character recognition model, eliminating the need for manual duration statistics, reducing time and labor costs, and improving audit efficiency.

[0053] In some embodiments, before inputting each image to be audited into the target detection model to obtain the helmet detection result and personnel detection result of each image to be audited, the method further includes: acquiring multiple images to be labeled; the images to be labeled are images of personnel at the work site; personnel bounding box annotation and personnel category label annotation are performed on the personnel in each image to be labeled, and helmet bounding box annotation and helmet category label annotation are performed on the helmets in each image to be labeled, to obtain multiple sample images, sample personnel detection results corresponding to each sample image, and sample helmet detection results corresponding to each sample image; based on the multiple sample images, the sample personnel detection results corresponding to each sample image, and the sample helmet detection results corresponding to each sample image, the initial model constructed based on the improved Yolov10 algorithm is trained to obtain the target detection model.

[0054] Understandably, such as Figure 2 As shown, before using the object detection model to detect the audit image, it is necessary to label training samples in order to build and train the object detection model.

[0055] Specifically, the app is used to film personnel at the actual work site, obtaining sample video sequences, each consisting of multiple image frames. Since these sample video sequences may contain numerous identical or similar frames, a video clustering algorithm can be used to extract keyframes as images to be labeled. Identical or similar frames are then removed, providing the necessary image data for subsequent steps while reducing the collection and statistics of invalid samples, thus decreasing the model's data processing load.

[0056] Furthermore, for each person in the image to be labeled, personnel bounding boxes and personnel category labels are annotated, and for each safety helmet in the image to be labeled, safety helmet bounding boxes and safety helmet category labels are annotated, to obtain multiple sample images, the sample personnel detection results corresponding to each sample image, and the sample safety helmet detection results corresponding to each sample image.

[0057] In this embodiment, for each image to be labeled, bounding boxes can be used to label the positions of the person and the safety helmet in the image, and a corresponding category label can be assigned to each bounding box. Then, a deep learning model is used for training to learn the relationship between the person and the safety helmet.

[0058] Specifically, for each image to be labeled, the image is loaded and displayed in the labeling tool (LabelImg); the drawing tools in the labeling tool (such as rectangles or polygons) are used to label the personnel bounding boxes in the image to determine the location of the personnel. During the labeling process, the size and position of the boxes can be adjusted as needed to accurately surround the personnel. For the labeled personnel bounding boxes, the corresponding category label (e.g., person) is assigned. Similarly, after completing the personnel bounding box labeling and personnel category labeling, the same method is used to label the safety helmet bounding boxes and safety helmet category labels: the drawing tools are used to label the safety helmet bounding boxes in the image to determine the location of the safety helmets. For the labeled safety helmet bounding boxes, the corresponding category label (e.g., hat) is assigned.

[0059] For each image to be labeled, the above method can be used to label the samples, thereby obtaining multiple sample images, the sample personnel detection results corresponding to each sample image, and the sample safety helmet detection results corresponding to each sample image.

[0060] Optionally, for the image to be labeled in a complex scene, there may be multiple people and safety helmets, in which case different colors or numbers can be used to distinguish them.

[0061] Alternatively, during the annotation process, shortcut keys or functions provided by tools can be used to improve annotation efficiency, such as copying and pasting annotated boxes, adjusting the size and position of boxes, etc.

[0062] Optionally, for consecutive frames in a sample video sequence, each frame can be selectively labeled, or some key frames can be selected for labeling as needed, and interpolation or tracking algorithms can be used to estimate the labeling results of the remaining frames.

[0063] Optionally, after annotation is completed, the annotated samples are saved in a standard data format, such as XML, JSON, or CSV, for subsequent model training and evaluation.

[0064] Optionally, when annotating, it is necessary to ensure that the annotators have good visual judgment and annotation accuracy; during the annotation process, consistent annotation specifications and standards should be followed to ensure the consistency and reliability of the annotation results; for complex scenes or ambiguous situations, annotations or explanations can be added in the annotation tool to facilitate subsequent data analysis and model training; in addition, the quality of annotation results can be checked and verified regularly to ensure the accuracy and consistency of the annotation.

[0065] Furthermore, based on multiple sample images, the detection results of the sample personnel corresponding to each sample image, and the detection results of the sample safety helmet corresponding to each sample image, the initial model built based on the improved Yolov10 algorithm is trained to learn the relationship between the personnel and the safety helmet, thereby obtaining the target detection model.

[0066] During training, the object detection model learns to extract image features and predict the category and location of each bounding box. When inference is needed on new video images, the trained object detection model can be used to identify all people and determine whether they are wearing helmets: the video can be broken down into frames, and the trained object detection model can be applied to each frame for inference. The object detection model will output the category label and confidence score for each bounding box, and the person wearing the helmet can be determined based on the confidence threshold.

[0067] In this embodiment, an initial model built based on the improved Yolov10 algorithm needs to be trained using processed training samples and labels to achieve object detection. By constructing object detection models for safety helmets and people, the bounding box coordinates of the safety helmets and people in each image to be audited are obtained. The correlation between the safety helmets and people is determined by combining the IOU algorithm and relative positional relationships, thereby identifying the audit personnel in the audit photos and outputting the corresponding number of audit personnel.

[0068] The initial model built on the improved Yolov10 algorithm introduces a new feature extraction structure, RGELAN. RGELAN replaces the C2f structure in the original Yolov10 algorithm with lightweight convolutions (convt-tiny) to reduce computational complexity. To compensate for the potential limitations in feature extraction capabilities caused by lightweight convolutions, a model parameterization (RepConv) technique is introduced. RepConv uses three branches during training to capture features from different receptive domains, while reducing parameters and computation during inference through reparameterization. During training, RepConv includes an identity branch, a 1×1 convolution, and a 3×3 convolution to learn features from different receptive domains, enriching the extracted feature information. During inference, RepConv reparameterizes by converting the identity branch and the 1×1 convolution into a 3×3 convolution, then merges the 3×3 convolution branches to produce a single branch, integrating features from different input branches with the parameter count of a standard 3×3 convolution.

[0069] Specifically, the initial model built on the improved Yolov10 algorithm does not require non-maximum suppression (NMS) during training to improve inference efficiency; in this model, a consistent dual assignment strategy is adopted to achieve training without NMS through dual label assignment and consistent matching metric.

[0070] In addition, the anchor size of the model needs to be adjusted to fit the size of the helmet and the person. The base size is set to 128, and the size scaling factor [0.5, 2] and aspect ratio [0.6, 3] are used to generate a suitable anchor.

[0071] In some embodiments, acquiring multiple images to be labeled includes: acquiring a sample video sequence; the sample video sequence is obtained by filming personnel at a work site, and the sample video sequence includes multiple image frames; determining multiple inter-frame similarities based on the multiple image frames; each inter-frame similarity is determined based on the distance between any two image frames in the sample video sequence; clustering the multiple image frames based on the multiple inter-frame similarities to obtain a first clustering result; the first clustering result includes multiple first clusters, and each first cluster includes at least one image frame; determining the first cluster center of each first cluster, and iteratively optimizing each first cluster center based on a genetic algorithm to obtain multiple optimized cluster centers; calculating the distance between each image frame and each optimized cluster center, and clustering the multiple image frames based on the minimum distance criterion to obtain a second clustering result; the second clustering result includes multiple second clusters, and each second cluster includes at least one image frame; determining the target image frame of each second cluster, and using the multiple target image frames as multiple images to be labeled; each target image frame is the image frame in each second cluster that is closest to the second cluster center of the second cluster.

[0072] After obtaining a sample video sequence containing multiple image frames, it is necessary to calculate the inter-frame similarity in order to perform video clustering.

[0073] Specifically, the sample video sequence includes multiple image frames. For any two image frames in the sample video sequence, the Euclidean distance between the two image frames can be calculated. The Euclidean distance is used to represent the inter-frame similarity between the two image frames. The smaller the Euclidean distance, the higher the inter-frame similarity, and the larger the Euclidean distance, the lower the inter-frame similarity.

[0074] The Euclidean distance between two image frames is defined as follows: ; Where A represents image frame A; B represents image frame B; This represents the inter-frame similarity between image frame A and image frame B; Indicates the row number of pixels in an image frame; This indicates the number of columns of pixels in an image frame; Indicates the first image frame A Line number The number of pixels in a column; Indicates the first image frame B Line number The number of pixels in a column.

[0075] In the above formula, This represents the square of the difference in position between two pixels in space. This represents the sum of squared differences in the positions of all pixels; This represents the square root of the sum of the squared differences in the positions of all pixels, i.e., the Euclidean distance. The summation range can be defined as the entire image frame as above, or it can be defined as a region of the same size within the image frame or a single point. The latter two represent the Euclidean distance under a local feature metric.

[0076] Similar frames can be determined by calculating the similarity value of all pixels in an image frame. The threshold for the similarity value MSE can be set to 10. If it is greater than 10, the two image frames are considered dissimilar. If more than 90% of the pixels in an image frame are similar, the two image frames are considered similar.

[0077] For each image frame in the sample video sequence, the inter-frame similarity between that image frame and other image frames can be calculated in the above manner, thereby obtaining multiple inter-frame similarities.

[0078] Furthermore, based on multiple inter-frame similarities, multiple image frames are clustered using a hierarchical clustering algorithm to obtain a first clustering result. The first clustering result includes multiple first clusters, and each first cluster includes at least one image frame.

[0079] Specifically, the agglomerative hierarchical clustering algorithm is used to perform initial clustering of image frames in the sample video sequence. The main idea is to first treat each sample to be clustered as a class, and then merge similar classes into one class based on the size of the inter-frame similarity, until the termination condition is met.

[0080] To facilitate understanding of the video clustering method in this embodiment, we will use 100 image frames as an example: Assuming there are 1000 image frames in the sample video sequence, and each image frame can be considered as a sample to be clustered, then there are 1000 samples to be clustered. The basic steps of clustering are as follows: (1) Determine the termination condition: Assume that the number of keyframes to be extracted is 10, that is, the termination condition is that the number of clusters is 10.

[0081] In this clustering process, the mean of the feature vectors is denoted as M, the variance of the feature vectors is denoted as V, and the keyframe is one of the image frames among all similar image frames, D={d i |i=1,2,3,…,N} represents the distance vector. In this embodiment, the final value is equal to the inter-frame distance d(Fa,Fb)>M+2*number of image frames.

[0082] (2) Initialization: Each sample to be clustered is assigned to a separate class, and the inter-frame similarity between each two classes (i.e., two samples to be clustered) is calculated and obtained in turn.

[0083] Frame similarity is represented by Euclidean distance.

[0084] (3) Find the two samples to be clustered with the closest Euclidean distance (i.e., find the two samples to be clustered with the highest inter-frame similarity) and classify the two samples to be clustered into one class.

[0085] (4) Recalculate the inter-frame similarity between the newly generated class and each of the old classes.

[0086] (5) Repeat steps (3) and (4) until the number of clusters is 10, obtain the first clustering result, and end the clustering.

[0087] At this point, the first clustering result contains 10 first clusters, and each first cluster includes at least one image frame.

[0088] Furthermore, the first cluster center of each first cluster is determined, and each first cluster center is iteratively optimized based on the genetic algorithm and the K-Means clustering algorithm to obtain multiple optimized cluster centers. Then, the distance between each image frame and each optimized cluster center is calculated, and multiple image frames are clustered based on the minimum distance criterion to obtain the second clustering result. The second clustering result includes multiple second clusters, and each second cluster includes at least one image frame.

[0089] Furthermore, the target image frame for each second cluster is determined, and multiple target image frames are treated as multiple images to be labeled.

[0090] Each target image frame is the image frame in each second cluster that is closest to the center of the second cluster.

[0091] Specifically, after obtaining 10 first clusters, the initial cluster centers of the 10 first clusters are optimized using a genetic algorithm and used as input for the K-means clustering algorithm. This avoids the influence of randomly generated initial cluster centers on the clustering results of the K-means clustering algorithm, as well as the overfitting phenomenon caused by incompatibility between the clustering results of hierarchical clustering and the K-means clustering algorithm.

[0092] Continuing with the example of the first clustering result containing 10 clusters, the specific steps for optimizing the clustering are as follows: (1) Initialize the population based on the hierarchical clustering results (i.e., the 10 first clusters).

[0093] (2) Define the total internal sum of squares (SSE) fitness function to evaluate the quality of each chromosome (first cluster center).

[0094] (3) Based on the genetic algorithm, each chromosome (first cluster center) is iteratively optimized to obtain multiple optimized cluster centers. The iterative process is as follows: (a) Selection: Select chromosomes based on fitness.

[0095] (b) Crossover: Randomly pair chromosomes and exchange information.

[0096] (c) Variation: Random variation of chromosomes.

[0097] (d) Evaluation: Calculate the fitness of the new population.

[0098] (4) Check whether the maximum number of iterations (e.g., 500) and the fitness change threshold are less than 0.001 as termination conditions to obtain multiple optimized cluster centers.

[0099] (5) After the iterative optimization is completed, select the chromosome with the highest fitness and decode it as the final cluster center.

[0100] (6) Use the cluster centers generated in step (5) as the initial cluster centroids of the K-means clustering algorithm.

[0101] (7) Calculate the distance between each image frame and each optimized cluster center, and re-cluster multiple image frames based on the minimum distance criterion.

[0102] (8) Recalculate the cluster centers for each class.

[0103] (9) Repeat steps (7) and (8) until the objects in each second cluster no longer change.

[0104] (10) Calculate the second cluster center of each second cluster and output the frame closest to the second cluster center as the key frame (i.e. the target image frame).

[0105] (11) The 10 target image frames are obtained through the previous steps and are used as 10 images to be labeled.

[0106] In some embodiments, the helmet detection result includes the helmet bounding box coordinates, and the personnel detection result includes the personnel bounding box coordinates. Based on each helmet detection result and each personnel detection result, determining the number of auditors includes: determining the intersection-union ratio (IU) of helmets and personnel in each image to be audited, and the relative positions between helmets and personnel in each image to be audited, based on the helmet bounding box coordinates and personnel bounding box coordinates of each image to be audited; determining the auditors in each image to be audited based on the IU and the relative positions between helmets and personnel in each image to be audited; and determining the number of auditors based on the auditors in each image to be audited.

[0107] Specifically, for each image to be audited, the bounding box coordinates of the safety helmet and the bounding box coordinates of the personnel in the image to be audited can be predicted using the target detection model. Then, based on the bounding box coordinates of the safety helmet and the bounding box coordinates of the personnel, the intersection-union ratio (IOU) of the safety helmet and the personnel in the image to be audited can be calculated, and the relative positions between the safety helmet and the personnel in the image to be audited can be determined.

[0108] like Figure 3 As shown, assuming that the safety helmet bounding box A and the personnel bounding box B are both rectangles, A∩B represents the intersection of the safety helmet bounding box A and the personnel bounding box B, and A∪B represents the union of the safety helmet bounding box A and the personnel bounding box B. Then, IOU is the intersection area divided by the union area.

[0109] Furthermore, based on the crossover ratio of the safety helmet and the person in the image to be audited, and the relative position between the safety helmet and the person in the image to be audited, the audit personnel in each image to be audited are determined.

[0110] Specifically, the correlation between the safety helmet and the person is determined based on the intersection-union ratio (IOU) and relative position: if the IOU exceeds a set threshold (e.g., 80%), and the center point of the safety helmet (i.e., the coordinates of the safety helmet bounding box) is above the center point of the person (i.e., the coordinates of the person bounding box), then the person is considered to be wearing a safety helmet, and the person is the auditor, that is, the auditor is the person wearing a safety helmet.

[0111] For each image to be audited, the auditors in that image can be identified using the methods described above.

[0112] Furthermore, the number of auditors is determined based on the auditors in each image to be audited.

[0113] Specifically, the number of auditors is determined by counting the number of people wearing safety helmets identified in each image to be audited.

[0114] In some embodiments, based on each image to be audited, the earliest time and latest time of the video to be audited are determined using an optical character recognition (OCR) model. This includes: preprocessing each image to be audited to obtain multiple images to be recognized; inputting each image to be recognized into the OCR model to obtain the text recognition result of each image to be recognized output by the OCR model; performing matching filtering on each text recognition result based on regular expressions to obtain target strings that conform to the time format in each image to be recognized; and determining the earliest time and latest time of the video to be audited based on each target string. The OCR model is a model built based on the ChineseOCR algorithm.

[0115] Please see Figure 4 , Figure 4This is a flowchart illustrating the audit duration calculation method provided in the embodiments of this application.

[0116] Specifically, before performing optical character recognition (OCR), appropriate image preprocessing is performed on each image to be audited to obtain multiple images to be recognized, thereby improving the accuracy and effectiveness of subsequent recognition.

[0117] Furthermore, such as Figure 4 As shown, each image to be recognized is input into the optical character recognition model to obtain the text recognition result of each image to be recognized output by the optical character recognition model.

[0118] Among them, the optical character recognition model is a model built based on the ChineseOCR algorithm, namely the ChineseOCR model.

[0119] Specifically, when using the ChineseOCR model for text recognition, the image to be recognized needs to be loaded first, and the ChineseOCR model initialized. Then, the ChineseOCR model performs text detection and recognition on the image to determine the position of text characters and obtain the text recognition result. The text recognition result includes the detected text box positions and the recognized text content. During the recognition process, the ChineseOCR model performs text detection on the image to find the positions of text regions, and then recognizes these regions to obtain the text content. Finally, the recognition result is visualized to check the recognition effect.

[0120] Specifically, the ChineseOCR model's recognition process includes two stages: text detection and text recognition. The final output is the detected text box location and its corresponding text content. By filtering and recognizing characters in the text content, all target strings conforming to the time format can be extracted. After completing text recognition, the recognized characters are combined to form the final text output, which may include single characters, words, or complete text paragraphs. Through specific filtering logic, all target strings conforming to the time format can be extracted from the recognition results.

[0121] Furthermore, based on regular expressions, each text recognition result is matched and filtered to obtain the target string that conforms to the time format in each image to be recognized; based on each target string, the earliest time and the latest time of the video to be audited are determined.

[0122] Specifically, since the text recognition results may contain other characters, character anomalies, or missing characters, after obtaining the text recognition results of the video to be audited, for each image to be recognized, the target string conforming to the time format can be matched and filtered using regular expressions. After obtaining the target strings conforming to the time format in all images to be recognized, the time format can be unified, and the earliest time (i.e., the start time of the video) and the latest time (i.e., the end time of the video) of the video to be audited can be extracted from the series of target strings. The audit duration can then be calculated using the duration calculation formula, which is as follows: Duration = EndTime - StartTime; Where Duration represents the audit duration; EndTime is the latest time (i.e., video end time) of all videos to be audited in an audit work order; StartTime is the earliest time (i.e., video start time) of all videos to be audited in an audit work order.

[0123] In some embodiments, image preprocessing is performed on each image to be audited to obtain multiple images to be identified, including: performing grayscale processing on each image to be audited to obtain multiple grayscale images; performing binarization processing on each grayscale image to obtain multiple black and white images; performing denoising processing on each black and white image to obtain multiple denoised images; performing tilt correction processing on each denoised image to obtain multiple corrected images; performing size normalization processing on each corrected image to obtain multiple standard images; and performing contrast enhancement processing on each standard image to obtain multiple images to be identified.

[0124] Understandably, before performing optical character recognition (OCR), appropriate image preprocessing is performed on each image to be audited in order to improve the accuracy and effectiveness of subsequent recognition.

[0125] Specifically, the images to be audited are generally color images. In this case, each image to be audited can be converted to grayscale to convert the color image into multiple grayscale images, thereby reducing the amount of data and complexity.

[0126] Furthermore, by using global thresholding (e.g., the Otsu method), each grayscale image is binarized to convert it into multiple black and white images, i.e., containing only two colors.

[0127] Furthermore, median filtering, mean filtering, Gaussian filtering and other techniques are used to denoise each black and white image to remove noise from the image and obtain multiple denoised images.

[0128] Furthermore, using methods such as Hough transform, each denoised image is subjected to skew correction to ensure that the text lines are parallel to the horizontal lines, resulting in multiple corrected images.

[0129] Furthermore, each corrected image is resized and rescaled to adjust its size to meet the input requirements of the OCR engine in the ChineseOCR model, resulting in multiple standard images.

[0130] Furthermore, using techniques such as histogram equalization, each standard image is subjected to contrast enhancement to improve image contrast and make the text clearer, thus obtaining multiple images to be recognized.

[0131] The human-time auditing method provided in this application can solve the problems of slow efficiency and low accuracy caused by relying on manual auditing of work order attachment videos at this stage. According to the accuracy and timeliness requirements of the business, the method adopts data annotation, model training and model inference to identify the number of people and time in the video, which can perform audit judgment in real time and with high accuracy, thus improving the efficiency of video recognition.

[0132] This application also provides a personnel time auditing device. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the structure of the personnel time audit device provided in this application embodiment. In this application embodiment, the personnel time audit device includes an acquisition module 510, a target detection module 520, a personnel quantity determination module 530, a text character recognition module 540, and an audit duration determination module 550.

[0133] Module 510 is used to acquire the video to be audited.

[0134] The video to be audited is taken by the auditors and includes multiple images to be audited.

[0135] The target detection module 520 is used to input each image to be audited into the target detection model to obtain the safety helmet detection result and the personnel detection result of each image to be audited.

[0136] The personnel quantity determination module 530 is used to determine the number of auditors based on the inspection results of each safety helmet and each person.

[0137] The text character recognition module 540 is used to determine the earliest time and the latest time of the video to be audited based on each image to be audited using an optical character recognition model.

[0138] The audit duration determination module 550 is used to determine the audit duration based on the earliest and latest times.

[0139] In some embodiments, the target detection module 520 is used to acquire multiple images to be labeled; the images to be labeled are obtained by taking pictures of people at the work site; personnel bounding boxes and personnel category labels are labeled for the personnel in each image to be labeled, and safety helmet bounding boxes and safety helmet category labels are labeled for the safety helmets in each image to be labeled, thereby obtaining multiple sample images, sample personnel detection results for each sample image, and sample safety helmet detection results for each sample image; based on the multiple sample images, the sample personnel detection results for each sample image, and the sample safety helmet detection results for each sample image, an initial model built based on the improved Yolov10 algorithm is trained to obtain a target detection model.

[0140] In some embodiments, the target detection module 520 is used to acquire a sample video sequence; the sample video sequence is obtained by filming personnel at a work site, and the sample video sequence includes multiple image frames; based on the multiple image frames, multiple inter-frame similarities are determined; each inter-frame similarity is determined based on the distance between any two image frames in the sample video sequence; based on the multiple inter-frame similarities, the multiple image frames are clustered to obtain a first clustering result; the first clustering result includes multiple first clusters, and each first cluster includes at least one image frame; the first cluster center of each first cluster is determined, and each first cluster center is iteratively optimized based on a genetic algorithm to obtain multiple optimized cluster centers; the distance between each image frame and each optimized cluster center is calculated respectively, and the multiple image frames are clustered based on the minimum distance criterion to obtain a second clustering result; the second clustering result includes multiple second clusters, and each second cluster includes at least one image frame; the target image frame of each second cluster is determined, and the multiple target image frames are used as multiple images to be labeled; each target image frame is the image frame in each second cluster that is closest to the second cluster center of the second cluster.

[0141] In some embodiments, the helmet detection result includes the helmet bounding box coordinates, and the personnel detection result includes the personnel bounding box coordinates.

[0142] The personnel quantity determination module 530 is used to determine the intersection-union ratio of safety helmets and personnel in each image to be audited, and the relative positions between safety helmets and personnel in each image to be audited, based on the safety helmet bounding box coordinates and the personnel bounding box coordinates of each image to be audited; to determine the audit personnel in each image to be audited based on the intersection-union ratio of safety helmets and personnel in each image to be audited, and the relative positions between safety helmets and personnel in each image to be audited; and to determine the number of audit personnel based on the audit personnel in each image to be audited.

[0143] In some embodiments, the text character recognition module 540 is used to perform image preprocessing on each image to be audited to obtain multiple images to be recognized; input each image to be recognized into the optical character recognition model to obtain the text recognition result of each image to be recognized output by the optical character recognition model; perform matching filtering on each text recognition result based on regular expressions to obtain the target string that conforms to the time format in each image to be recognized; and determine the earliest time and the latest time of the video to be audited based on each target string; wherein, the optical character recognition model is a model built based on the ChineseOCR algorithm.

[0144] In some embodiments, the text character recognition module 540 is configured to perform grayscale processing on each image to be audited to obtain multiple grayscale images; perform binarization processing on each grayscale image to obtain multiple black and white images; perform denoising processing on each black and white image to obtain multiple denoised images; perform tilt correction processing on each denoised image to obtain multiple corrected images; perform size standardization processing on each corrected image to obtain multiple standard images; and perform contrast enhancement processing on each standard image to obtain multiple images to be recognized.

[0145] This application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute real-time auditing methods.

[0146] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the human time auditing methods provided by the above methods.

[0148] This application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the human auditing methods provided by the above methods.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for auditing personnel time, characterized in that, include: Obtain the video to be audited; The video to be audited was taken by the auditors and includes multiple images to be audited. Each of the images to be audited is input into the target detection model to obtain the safety helmet detection result and the personnel detection result of each of the images to be audited, which are output by the target detection model. Based on the test results of each safety helmet and the test results of each person, the number of auditors is determined; Based on each of the images to be audited, the earliest time and the latest time of the video to be audited are determined by an optical character recognition model. The audit duration is determined based on the earliest time and the latest time.

2. The personnel-time auditing method according to claim 1, characterized in that, Before inputting each of the images to be audited into the target detection model and obtaining the helmet detection result and personnel detection result of each image to be audited output by the target detection model, the method further includes: Multiple images to be labeled are acquired; these images are photographs taken of personnel at the work site. Personnel bounding boxes and personnel category labels are annotated for each person in each image to be labeled, and safety helmets are annotated with safety helmet bounding boxes and safety helmet category labels for each safety helmet in each image to be labeled, thereby obtaining multiple sample images, sample personnel detection results for each sample image, and sample safety helmet detection results for each sample image. Based on multiple sample images, the detection results of the sample personnel corresponding to each sample image, and the detection results of the sample safety helmet corresponding to each sample image, the initial model constructed based on the improved Yolov10 algorithm is trained to obtain the target detection model.

3. The personnel-time auditing method according to claim 2, characterized in that, The process of acquiring multiple images to be labeled includes: Obtain a sample video sequence; the sample video sequence is obtained by filming personnel at the work site, and the sample video sequence includes multiple image frames; Based on multiple image frames, multiple inter-frame similarities are determined; each inter-frame similarity is determined based on the distance between any two image frames in the sample video sequence. Based on the inter-frame similarity, the multiple image frames are clustered to obtain a first clustering result; the first clustering result includes multiple first clusters, and each first cluster includes at least one of the image frames; Determine the first cluster center for each of the first clusters, and iteratively optimize each of the first cluster centers based on a genetic algorithm to obtain multiple optimized cluster centers; The distance between each image frame and each optimized cluster center is calculated, and the multiple image frames are clustered based on the minimum distance criterion to obtain a second clustering result; the second clustering result includes multiple second clusters, and each second cluster includes at least one image frame; Determine the target image frame for each of the second clusters, and use the multiple target image frames as multiple images to be labeled; each target image frame is the image frame in each of the second clusters that is closest to the center of the second cluster.

4. The personnel-time auditing method according to claim 1, characterized in that, The helmet detection result includes the helmet bounding box coordinates, and the personnel detection result includes the personnel bounding box coordinates. The determination of the number of auditors based on the test results of each safety helmet and each personnel includes: Based on the bounding box coordinates of the safety helmet and the bounding box coordinates of the personnel in each image to be audited, the intersection-union ratio of the safety helmet and the personnel in each image to be audited, and the relative positions between the safety helmet and the personnel in each image to be audited are determined. Based on the intersection-over-union ratio of the safety helmet and the person in each of the images to be audited, and the relative positions between the safety helmet and the person in each of the images to be audited, the audit personnel in each of the images to be audited are determined; The number of auditors is determined based on the auditors in each of the images to be audited.

5. The personnel-time auditing method according to claim 1, characterized in that, The step of determining the earliest time and the latest time of each video to be audited, based on each image to be audited, using an optical character recognition model, includes: Each of the images to be audited is preprocessed to obtain multiple images to be identified. Each of the images to be recognized is input into the optical character recognition model to obtain the text recognition result of each image to be recognized output by the optical character recognition model; Based on regular expressions, each text recognition result is matched and filtered to obtain the target string that conforms to the time format in each image to be recognized. Based on each of the target strings, determine the earliest time and the latest time of the video to be audited; The optical character recognition model is a model built based on the ChineseOCR algorithm.

6. The personnel-time auditing method according to claim 5, characterized in that, The step of performing image preprocessing on each of the images to be audited to obtain multiple images to be identified includes: Each of the images to be audited is converted to grayscale to obtain multiple grayscale images; Each grayscale image is binarized to obtain multiple black and white images; Each of the black and white images is denoised to obtain multiple denoised images. Each of the denoised images is subjected to tilt correction processing to obtain multiple corrected images; Each of the corrected images is subjected to size normalization processing to obtain multiple standard images; Each of the standard images is subjected to contrast enhancement processing to obtain multiple images to be identified.

7. A personnel time auditing device, characterized in that, include: The acquisition module is used to acquire the video to be audited; The video to be audited was taken by the auditors and includes multiple images to be audited. The target detection module is used to input each of the images to be audited into the target detection model, and obtain the safety helmet detection result and the personnel detection result of each of the images to be audited output by the target detection model. The personnel quantity determination module is used to determine the number of auditors based on the detection results of each safety helmet and the detection results of each person. The text character recognition module is used to determine the earliest time and the latest time of the video to be audited based on each of the images to be audited, using an optical character recognition model. The audit duration determination module is used to determine the audit duration based on the earliest time and the latest time.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the human auditing method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the human time auditing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the human time auditing method as described in any one of claims 1 to 6.

Images