An automatic recognition system for on-site supervision behavior in railway operations

By using deep learning models and multi-dimensional data fusion to judge, the system automatically identifies the identity and behavior of supervisors, solving the shortcomings of existing technologies in supervising on-site supervision and achieving efficient and real-time supervision management of railway engineering sites.

CN123072079APending Publication Date: 2026-11-10SHUOHUANG RAILWAY DEV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611027717.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-10
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In the existing technology, the supervision of on-site supervision mainly relies on manual sign-in, paper records or simple video playback verification. It cannot automatically identify whether the supervisors are actually on duty or in the designated on-site supervision area, and it is difficult to determine whether their behavior complies with the on-site supervision regulations.

Method used

A deep learning-based supervisor identification model is adopted, which combines a video acquisition module, a spatial positioning analysis module, and a process status perception module. The model identifies the supervisor's identity, obtains their spatial coordinates and construction status characteristics through video stream data, and integrates these to determine whether their behavior complies with the on-site supervision specifications.

Benefits of technology

It enables automated and real-time monitoring of the identity and behavior of supervisors, improving the authenticity and effectiveness of supervision work and ensuring that supervisors perform on-site supervision in the correct location and at the correct stage of the process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123072079A_ABST
    Figure CN123072079A_ABST
Patent Text Reader

Abstract

This application relates to an automatic identification system for on-site supervision behavior in railway operations. The system includes a video acquisition module, a personnel identification module, a spatial positioning analysis module, a process status perception module, and a multi-dimensional data fusion judgment module. The video acquisition module acquires video stream data at the railway operation site. The personnel identification module uses a deep learning-based personnel identification model to identify personnel in the video stream data, obtaining the identification result. The spatial positioning analysis module outputs the three-dimensional spatial coordinates of the identified personnel. The process status perception module generates a process status judgment result. The multi-dimensional data fusion judgment module outputs a multi-dimensional data fusion judgment result. This system can determine whether on-site supervision personnel comply with on-site supervision standards.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of railway intelligent supervision technology, and in particular to an automatic identification system for on-site supervision behavior in railway operations. Background Technology

[0002] With the continuous expansion of railway construction scale and the continuous improvement of construction intelligence level, the requirements for engineering quality and safety supervision at railway operation sites are getting higher and higher. In the process of railway engineering construction, on-site supervision, as an important means of quality control of key processes, has become a core link to ensure construction compliance and safety.

[0003] Current technologies for supervising on-site supervision mainly rely on manual sign-in, paper records, or simple video playback verification. However, manual sign-in, paper records, or simple video playback verification cannot determine whether the supervisor's behavior complies with the on-site supervision regulations. Summary of the Invention

[0004] Therefore, it is necessary to provide an automatic identification system for on-site supervision behavior in railway operations that can determine whether the behavior of supervisors conforms to the on-site supervision specifications, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides an automatic identification system for on-site supervision behavior in railway operations. This system includes a video acquisition module, a personnel identification module, a spatial positioning analysis module, a process status perception module, and a multi-dimensional data fusion and judgment module.

[0006] The video acquisition module is used to acquire video stream data at the railway operation site;

[0007] The personnel identification module is used to identify the personnel in the video stream data using a deep learning-based supervisory personnel identification model, and obtain the supervisory personnel identification result.

[0008] The spatial positioning analysis module is used to obtain the two-dimensional spatial coordinates of the identified supervisor in the video stream data if the supervisor identification result includes the identified supervisor; perform coordinate mapping estimation on the two-dimensional spatial coordinates, and output the three-dimensional spatial coordinates of the identified supervisor;

[0009] The process status perception module is used to acquire the construction status characteristics of the railway operation site in the video stream data; and generate a process status judgment result based on the construction status characteristics and the supervision requirements of the current construction process at the railway operation site.

[0010] The multi-dimensional data fusion judgment module is used to determine the effectiveness of the supervisor's on-site supervision behavior under compliance judgment conditions based on the supervisor's identity recognition result, the three-dimensional spatial coordinates, and the process status judgment result, and output the multi-dimensional data fusion judgment result.

[0011] In one embodiment, the video acquisition module includes a panoramic coverage submodule, a region focusing submodule, and an adaptive illumination adjustment submodule; wherein,

[0012] The panoramic coverage submodule is deployed at a high-level deployment point on the perimeter of the railway operation site to collect wide-angle, blind-spot-free video stream data of the railway operation site in order to record the overall situation of supervision activities and construction operations at the railway operation site.

[0013] The area focusing submodule is used to collect local video streams from the supervisory on-site locations at the railway operation site, so as to record at least one of the facial features and behavioral details of the supervisory personnel at the supervisory on-site locations;

[0014] The illumination adaptive adjustment submodule is used to acquire at least one of the ambient illumination parameters and the meteorological sensing parameters; and dynamically adjust the imaging optical control strategy based on the ambient illumination parameters and the meteorological sensing parameters to maintain the video imaging quality of the video stream data.

[0015] In one embodiment, the personnel identification module includes a keyframe extraction submodule, a supervisor's facial feature enhancement submodule, and a bystander qualification verification submodule; wherein:

[0016] The keyframe extraction submodule is used to extract keyframe images from the video stream data based on at least one of time series and perspective diversity.

[0017] The supervisor's facial feature enhancement submodule is used to robustly enhance the facial region in the keyframe image based on the deep learning-based supervisor identification algorithm to generate supervisor facial features.

[0018] The on-site qualification verification submodule is used to compare the facial features of the supervisor with the registration feature templates pre-stored in the supervisor qualification database; if the similarity between the facial features of the supervisor and the registration feature templates is greater than a preset similarity threshold, the authorization information of the current construction procedure is obtained; based on the authorization information of the current construction procedure, the supervisor's facial features are used to verify the authorization qualification of the supervisor for the current construction procedure.

[0019] In one embodiment, the spatial positioning analysis module includes an electronic fence configuration submodule, a target detection and tracking submodule, and a spatial mapping submodule; wherein:

[0020] The electronic fence configuration submodule is used to set the logical fence boundary corresponding to the on-site area in the spatial coordinate reference system of the railway operation site based on the on-site area requirements of the current construction process, and generate dynamic electronic fence data bound to the current construction process.

[0021] The target detection and tracking submodule is used to perform real-time cross-frame target tracking of the identified supervisors in the video frames corresponding to the video stream data, and output two-dimensional trajectory data in each video frame.

[0022] The spatial mapping submodule is used to obtain the calibration parameters of the camera, which is used to collect the video stream data; based on the calibration parameters of the camera and the spatial coordinate reference system of the railway operation site, the two-dimensional trajectory data is estimated by coordinate mapping, and the three-dimensional spatial coordinates are output; the three-dimensional spatial coordinates are compared with the dynamic electronic fence data to determine the positional relationship between the identified supervisor and the on-site monitoring area required by the current construction procedure.

[0023] In one embodiment, the process status perception module includes a construction process parsing submodule, a feature extraction submodule, and a semantic logic comparison submodule; wherein:

[0024] The construction process parsing submodule is used to obtain the rule set of on-site supervision related processes; and to generate a semantic rule library of construction processes that require on-site supervision based on the rule set of on-site supervision.

[0025] The feature extraction submodule is used to extract features from the video stream data based on a pre-trained construction state feature perception network to obtain the construction state features. The construction state feature perception network is driven by deep learning.

[0026] The semantic logic comparison submodule is used to compare the construction status features with the construction procedure semantic rule library to determine the supervision requirements of the current construction procedure at the railway operation site and generate the procedure status determination result.

[0027] In one embodiment, the multidimensional data fusion judgment module includes a four-dimensional element alignment submodule, a consistency verification submodule, and a side-site behavior recording submodule; wherein:

[0028] The four-dimensional element alignment submodule is used to obtain the system timestamp of the automatic identification system for on-site supervision behavior of railway operations; and to synchronize the time and unify the spatial coordinates of the identification results of the supervisors, the three-dimensional spatial coordinates, the system timestamp, and the process status judgment results to form a spatiotemporally aligned four-dimensional element fusion dataset.

[0029] The consistency verification submodule is used to construct a four-dimensional consistency verification mechanism based on the business logic of on-site supervision; and to perform consistency verification on the four-dimensional element fusion dataset based on the four-dimensional consistency verification mechanism to determine the effectiveness of the on-site supervision behavior performed by the supervisor under the compliance judgment conditions, and output the multi-dimensional data fusion judgment result.

[0030] The side-site behavior recording submodule is used to encapsulate the multi-dimensional data fusion judgment result, the four-dimensional element fusion dataset, and the system timestamp into a side-site behavior record to connect with the abnormal behavior early warning function.

[0031] Secondly, this application also provides a method for automatically identifying on-site supervision behavior at railway work sites, including:

[0032] Acquire video stream data at railway operation sites;

[0033] The supervisors in the video stream data are identified using a deep learning-based supervisor identification model, and the supervisor identification results are obtained.

[0034] If the identification result of the supervisor includes an identified supervisor, then the two-dimensional spatial coordinates of the identified supervisor in the video stream data are obtained;

[0035] The coordinate mapping estimation is performed on the two-dimensional spatial coordinates, and the three-dimensional spatial coordinates of the identified supervisors are output.

[0036] Obtain the construction status characteristics of the railway operation site from the video stream data;

[0037] Based on the construction status characteristics and the supervision requirements of the current construction procedure at the railway work site, a procedure status determination result is generated;

[0038] Based on the identification results of the supervisors, the three-dimensional spatial coordinates, and the process status determination results, the effectiveness of the supervisors' on-site supervision behavior under compliance determination conditions is determined, and a multi-dimensional data fusion determination result is output.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0040] Acquire video stream data at railway operation sites;

[0041] The supervisors in the video stream data are identified using a deep learning-based supervisor identification model, and the supervisor identification results are obtained.

[0042] If the identification result of the supervisor includes an identified supervisor, then the two-dimensional spatial coordinates of the identified supervisor in the video stream data are obtained;

[0043] The coordinate mapping estimation is performed on the two-dimensional spatial coordinates, and the three-dimensional spatial coordinates of the identified supervisors are output.

[0044] Obtain the construction status characteristics of the railway operation site from the video stream data;

[0045] Based on the construction status characteristics and the supervision requirements of the current construction procedure at the railway work site, a procedure status determination result is generated;

[0046] Based on the identification results of the supervisors, the three-dimensional spatial coordinates, and the process status determination results, the effectiveness of the supervisors' on-site supervision behavior under compliance determination conditions is determined, and a multi-dimensional data fusion determination result is output.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0048] Acquire video stream data at railway operation sites;

[0049] The supervisors in the video stream data are identified using a deep learning-based supervisor identification model, and the supervisor identification results are obtained.

[0050] If the identification result of the supervisor includes an identified supervisor, then the two-dimensional spatial coordinates of the identified supervisor in the video stream data are obtained;

[0051] The coordinate mapping estimation is performed on the two-dimensional spatial coordinates, and the three-dimensional spatial coordinates of the identified supervisors are output.

[0052] Obtain the construction status characteristics of the railway operation site from the video stream data;

[0053] Based on the construction status characteristics and the supervision requirements of the current construction procedure at the railway work site, a procedure status determination result is generated;

[0054] Based on the identification results of the supervisors, the three-dimensional spatial coordinates, and the process status determination results, the effectiveness of the supervisors' on-site supervision behavior under compliance determination conditions is determined, and a multi-dimensional data fusion determination result is output.

[0055] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0056] Acquire video stream data at railway operation sites;

[0057] The supervisors in the video stream data are identified using a deep learning-based supervisor identification model, and the supervisor identification results are obtained.

[0058] If the identification result of the supervisor includes an identified supervisor, then the two-dimensional spatial coordinates of the identified supervisor in the video stream data are obtained;

[0059] The coordinate mapping estimation is performed on the two-dimensional spatial coordinates, and the three-dimensional spatial coordinates of the identified supervisors are output.

[0060] Obtain the construction status characteristics of the railway operation site from the video stream data;

[0061] Based on the construction status characteristics and the supervision requirements of the current construction procedure at the railway work site, a procedure status determination result is generated;

[0062] Based on the identification results of the supervisors, the three-dimensional spatial coordinates, and the process status determination results, the effectiveness of the supervisors' on-site supervision behavior under compliance determination conditions is determined, and a multi-dimensional data fusion determination result is output.

[0063] The aforementioned automatic identification system for on-site supervision behavior at railway work sites utilizes a deep learning-based supervisor identification model to identify personnel in the video stream data, obtaining identification results to verify whether the supervisors meet the identity requirements. It acquires the two-dimensional spatial coordinates of the identified supervisors in the video stream data; performs coordinate mapping estimation on these coordinates to output the three-dimensional spatial coordinates of the identified supervisors; and compares these three-dimensional spatial coordinates with the supervision area of ​​the railway work site to verify whether the supervisors are in the correct on-site supervision area. It also acquires the construction status characteristics of the current construction process at the railway work site in the video stream data, determines whether the railway work site is currently in a process stage requiring on-site supervision based on these characteristics, and generates a process status judgment result to verify whether the supervisors correspond to the correct required process. Finally, based on the supervisor identification results, the three-dimensional spatial coordinates, and the process status judgment result, it can determine whether the supervisors comply with on-site supervision standards, thereby improving the authenticity, real-time performance, and effectiveness of railway supervision work. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a diagram illustrating the application environment of an automatic identification system for on-site supervision behavior in railway operations, as shown in one embodiment.

[0066] Figure 2 This is a schematic diagram of the automatic identification system for on-site supervision behavior of railway operations in one embodiment.

[0067] Figure 3 This is a schematic diagram of the video acquisition module in one embodiment;

[0068] Figure 4 This is a schematic diagram of the personnel identification module in one embodiment;

[0069] Figure 5 This is a schematic diagram of the spatial positioning analysis module in one embodiment;

[0070] Figure 6 This is a schematic diagram of the process status sensing module in one embodiment;

[0071] Figure 7This is a schematic diagram of the structure of a multi-dimensional data fusion judgment module in one embodiment;

[0072] Figure 8 This is a flowchart illustrating an automatic identification method for on-site supervision behavior at railway work sites in one embodiment.

[0073] Figure 9 This is an internal structural diagram of a computer device in one embodiment;

[0074] Figure 10 This is a schematic diagram of the structure of a computer program product in one embodiment. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0076] The terms “comprising” and “having” as used in this application, and any variations thereof, are intended to cover non-exclusive inclusion.

[0077] With the continuous expansion of railway construction scale and the continuous improvement of construction intelligence level, the requirements for engineering quality and safety supervision at railway operation sites are getting higher and higher. In the process of railway engineering construction, on-site supervision, as an important means of quality control of key processes, has become a core link to ensure construction compliance and safety.

[0078] However, faced with increasingly complex construction environments, dispersed work areas, and high-intensity supervision requirements, the traditional supervision and management model that relies on manual recording and on-site spot checks can no longer meet the needs of modern railway engineering for full-process, full-coverage, and high-efficiency supervision. It is urgent to introduce automated and intelligent technologies to improve the traceability and authenticity of supervision activities.

[0079] Currently, the existing technology for supervising on-site supervision mainly relies on manual sign-in, paper records, or simple video playback verification, which has many shortcomings. Manual records are easily forged or omitted, and lack objectivity and real-time performance. Even if a video monitoring system is deployed, it is mostly used for post-event review and cannot automatically identify whether the supervisors are actually on duty or in the designated on-site supervision area, let alone determine whether their behavior complies with the on-site supervision regulations.

[0080] To address this issue, this application provides a solution. In this embodiment, a deep learning-based supervisor identification model is used to identify personnel in video stream data, obtaining the supervisor identification results. This verifies whether the supervisor meets the identity requirements. The solution involves acquiring the two-dimensional spatial coordinates of the identified supervisor in the video stream data; performing coordinate mapping estimation on these coordinates to output the three-dimensional spatial coordinates of the identified supervisor; and comparing these three-dimensional spatial coordinates with the supervision area of ​​the railway work site to verify whether the supervisor is in the correct on-site supervision area. Furthermore, the solution involves acquiring the construction status characteristics of the current construction process at the railway work site in the video stream data; determining whether the railway work site is currently in a process requiring on-site supervision based on these characteristics; and generating a process status determination result to verify whether the supervisor corresponds to the correct process requiring on-site supervision. Finally, based on the supervisor identification results, three-dimensional spatial coordinates, and process status determination results, it can be determined whether the supervisor meets the on-site supervision specifications, thereby improving the authenticity, real-time performance, and effectiveness of railway supervision work.

[0081] The automatic identification system for on-site supervision behavior in railway operations provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other network servers. The video acquisition module, personnel identification module, spatial positioning analysis module, and process status perception module of the railway operation site supervision and on-site behavior automatic recognition system can be located in terminal 102. The spatial positioning analysis module and process status perception module can also be located in server 104, and the multi-dimensional data fusion judgment module can be located in server 104. Terminal 102 can be, but is not limited to, various cameras, personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0082] In one exemplary embodiment, such as Figure 2 As shown, an automatic identification system for on-site supervision behavior in railway operations is provided. The system includes a video acquisition module 101, a personnel identification module 102, a spatial positioning analysis module 103, a process status perception module 104, and a multi-dimensional data fusion judgment module 105; wherein,

[0083] Video acquisition module 101 is used to acquire video stream data at railway operation sites;

[0084] This can be achieved by deploying camera-equipped data acquisition terminals at railway work sites to obtain video stream data. For example, video can be acquired by deploying cameras, personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices.

[0085] The personnel identification module 102 is used to identify the personnel in the video stream data using a deep learning-based personnel identification model, and obtain the personnel identification results.

[0086] As one implementation method, a pre-built and trained deep learning-based supervisor identification model can be used to perform end-to-end supervisor identification of personnel in video stream data, thereby obtaining supervisor identification results.

[0087] The deep learning-based supervisor identification model can be constructed into four parts: an input layer, a feature extraction backbone network, an identity classification module, and an output layer. The input layer acquires single or consecutive frames from the video stream data and performs image preprocessing operations on these frames, including size scaling, pixel normalization, color gamut conversion, and adaptation to the model's input specifications.

[0088] Among them, deep learning visual networks, such as ResNet, MobileNet, and YOLO-backbone, can be selected as the feature extraction backbone network. The feature extraction backbone network can perform lightweight or precision optimization operations for complex lighting, background, and personnel movement scenarios at railway sites, and automatically mine multi-dimensional deep visual features such as human appearance, clothing, name tags, and body shape.

[0089] Among them, the identity classification module can be obtained by building a fully connected layer + activation function + classification layer, which maps the deep visual features extracted by the feature extraction backbone network to personnel identity categories, and the identity categories correspond one-to-one with the roster of supervisors.

[0090] The system outputs classification confidence scores and the corresponding supervisor's ID and identity results through the output layer, thus completing end-to-end identity recognition.

[0091] Furthermore, after constructing the deep learning-based supervisor identification model, it can be trained. For example, by collecting video stream data from railway operation sites, the video stream data is divided into video frames, and supervisor identification tags are labeled for the personnel in the video frames. The labeled video frames are then divided into training, validation, and test sets. Data augmentation techniques such as flipping, cropping, brightness perturbation, and noise can be applied to the training, validation, and test sets to simulate complex on-site conditions and improve the model's generalization ability. Pre-trained weights are used to initialize the feature extraction backbone network of the deep learning-based supervisor identification model. Cross-entropy loss is selected as the classification loss, paired with the Adam / SGD optimizer. The learning rate, batch size, and number of iterations are set. The constructed deep learning-based supervisor identification model is iteratively trained, validated, and tested using the training, validation, and test sets. The network structure and hyperparameters of the deep learning-based supervisor identification model are adjusted based on the recognition accuracy and false recognition rate to obtain the final model.

[0092] As another implementation method, a deep learning-based supervisor identification model can be used to perform interference robust processing on faces in video stream data, thereby improving the recognizability of faces under complex working conditions and generating stable and highly distinguishable structured supervisor facial features. The structured supervisor facial features are then compared with a preset supervisor qualification database to complete the identification of supervisors.

[0093] The spatial positioning analysis module 103 is used to obtain the two-dimensional spatial coordinates of the identified supervisor in the video stream data if the supervisor identification result includes the identified supervisor; perform coordinate mapping estimation on the two-dimensional spatial coordinates, and output the three-dimensional spatial coordinates of the identified supervisor;

[0094] If the identification results of the supervisors include identified supervisors, it is necessary to obtain the two-dimensional spatial coordinates of the identified supervisors and convert them into three-dimensional spatial coordinates.

[0095] Specifically, as one implementation method, the video stream data can be divided into several video frames, and each frame can be identified independently. For example, target detection and identity recognition algorithms based on models such as YOLO, RetinaNet, and Faster R-CNN can be used to locate and identify supervisors in each frame, and output the two-dimensional spatial coordinates of the identified supervisors in each frame.

[0096] As another implementation method, the video stream data can be divided into several video frames, and faces or bodies in each frame can be independently identified. For example, a general human detection algorithm can be used to traverse the video frames to locate the supervisor, output the two-dimensional coordinates of the human body's outer bounding box, and use these two-dimensional coordinates as the two-dimensional spatial coordinates of the identified supervisor. Alternatively, face detection models such as MTCNN and RetinaFace can be used to extract the two-dimensional pixel coordinates of the face region in each frame, and use these two-dimensional pixel coordinates as the two-dimensional spatial coordinates of the identified supervisor.

[0097] As another implementation method, the video stream data can be divided into several video frames. Based on the independent identification of each frame through the above implementation method, the same person can be tracked and associated using tracking algorithms such as ByteTrack, DeepSORT, and JDE, thereby extracting the continuous two-dimensional spatial coordinate sequence of the identified supervisor.

[0098] This can be achieved by using deep learning networks to estimate the coordinates of two-dimensional spatial coordinates and output the three-dimensional spatial coordinates of the identified supervisors. For example, a monocular depth estimation network (such as MiDaS, DPT, RA-Depth, etc.) can be introduced to perform pixel-by-pixel prediction for each frame of the image, obtaining a pixel-level depth map for each frame. Based on the pixel-level depth map, the corresponding depth values ​​are matched to the two-dimensional spatial coordinates of the identified supervisors in each frame. Combined with simplified camera parameters, the sum of the two-dimensional spatial coordinates of the identified supervisors and the matched depth values ​​is converted into three-dimensional spatial coordinates.

[0099] In addition, two-dimensional spatial coordinates can be estimated by binocular or multi-view stereo vision mapping to output the three-dimensional spatial coordinates of the identified supervisors. For example, two or more cameras with overlapping fields of view can be used to simultaneously capture images, perform left and right view feature matching on the same person, calculate the disparity map, and obtain the depth from the disparity based on the camera baseline and optical parameters. The two-dimensional spatial coordinates of the identified supervisors can then be combined with the depth to obtain the three-dimensional spatial coordinates.

[0100] In addition, coordinate mapping estimation of two-dimensional spatial coordinates can be performed using laser ranging to output the three-dimensional spatial coordinates of the identified supervisors. For example, a single-point laser rangefinder or lidar can be integrated into the camera to measure the straight-line distance from the camera lens to the personnel. Using the pinhole camera image set, the straight-line distance is used as a fixed constraint to first convert the two-dimensional spatial coordinates of the identified supervisors into normalized coordinates of the imaging plane. Then, the depth value of the target relative to the camera is solved by combining spatial geometric formulas, and the three-dimensional spatial coordinates in the camera coordinate system are further calculated.

[0101] The process status perception module 104 is used to acquire the construction status characteristics of the railway operation site in the video stream data; and generate process status judgment results based on the construction status characteristics and the supervision requirements of the current construction process at the railway operation site.

[0102] Among them, feature extraction algorithms can be used to extract the construction status features of railway operation sites in video stream data. Based on the construction status features and the supervision requirements of the current construction process at the railway operation site, a process status judgment result is generated, that is, based on the construction status features and the supervision requirements of the current construction process at the railway operation site, it is determined whether the current construction process requires supervision personnel.

[0103] As one implementation method, background modeling can be performed on idle video streams during continuous time periods at the railway operation site. A Gaussian mixture background modeling algorithm is used to remove dynamic noise such as wind-blown weeds and light fluctuations, generating a static background template for the operation area. Then, a difference operation is performed between each frame of the real-time video stream and the static background template of the operation area to filter out foreground pixel clusters whose pixel grayscale difference exceeds a preset grayscale difference threshold. Through K-means pixel clustering, the foreground pixel clusters are classified into types, and the construction status features are output.

[0104] As another implementation method, a deep learning network can be constructed and trained to extract construction status features from the video stream data of the railway operation site and output the construction status features. These construction status features can include the operational actions of construction personnel, the operating status of large machinery (such as track laying machines and tamping machines), and the technological features exhibited by the tracks or structures.

[0105] After obtaining the construction status characteristics, the current construction procedure, i.e., the current procedure stage, can be determined based on these characteristics. Simultaneously, based on the supervision requirements of the current procedure stage, it can be determined whether on-site supervision is required, generating a procedure status determination result. For example, the current construction status characteristics include: multiple tamping machines are operating continuously, construction personnel are following the tamping machines, and supervision personnel are stationed at fixed points around the work area. Therefore, based on the construction status characteristics, it is determined that the current stage is the track tamping construction procedure. According to the supervision requirements for the track tamping construction procedure, this is a critical procedure, requiring on-site supervision by supervisors throughout the entire process. Since the current construction status characteristics include the presence of supervisors at fixed points around the work area, a procedure status determination result indicating that on-site supervision is in place and the procedure is compliant can be obtained.

[0106] One implementation method is to determine the current construction process based on the construction status characteristics: sort out the construction characteristics corresponding to each construction process in advance, form a rule set of construction characteristics, compare the construction status characteristics with the rule set of construction characteristics, and if a certain process condition is met, then it is determined to be the current corresponding construction process.

[0107] As another implementation method, the current construction process can be determined through traditional machine learning classification. For example, the construction status features can be used as feature vectors, and "process name and process stage" can be used as classification labels. Traditional machine learning models can be used to train and classify the feature vectors and corresponding classification labels to achieve automatic classification and identification of the process.

[0108] As another implementation method, a feature template vector library can be established based on each process stage. The similarity between the construction status features and the feature template vector library can be calculated, and the template with the highest similarity in the feature template vector library is the current corresponding construction process.

[0109] The multi-dimensional data fusion judgment module 105 is used to determine the effectiveness of the supervisor's on-site supervision behavior under compliance judgment conditions based on the supervisor's identity recognition results, three-dimensional spatial coordinates, and process status judgment results, and outputs the multi-dimensional data fusion judgment results.

[0110] The compliance determination criteria include compliance constraints on at least one of the following: the identity of the supervisor, the location of the supervisor, and the current status of the work process. For example, the current work process is a critical process that requires on-site supervision and is in normal operation; the target personnel have been verified as on-duty supervisors with the corresponding on-site supervision qualifications; the three-dimensional spatial coordinates of the supervisor fall within the preset on-site supervision compliance area and have not entered the high-risk prohibited area.

[0111] The effectiveness of on-site supervision by construction personnel under compliance conditions can be understood as the degree of compliance of their on-site supervision. By quantifying at least one of the following factors—the supervisor's identity, location, and current work process status—a corresponding effectiveness score can be generated. This score reflects whether the supervisor is on duty throughout the process, whether their position is appropriate, and whether their performance matches the on-site construction procedures. For example, scores can be assigned to the supervisor's on-site time, location, and the degree of matching between their performance and the on-site construction procedures. Different weights can be assigned to each of these factors, and the effectiveness score can be calculated by weighted summation of the scores.

[0112] The aforementioned automatic identification system for on-site supervision behavior at railway work sites utilizes a deep learning-based model to identify personnel in video stream data. This identification verifies whether the supervisors meet the required identity criteria. The system acquires the two-dimensional spatial coordinates of the identified supervisors from the video stream data, performs coordinate mapping estimation, and outputs their three-dimensional spatial coordinates. These coordinates are then compared with the supervision area of ​​the railway work site to verify if the supervisors are in the correct on-site supervision area. Furthermore, the system acquires the construction status characteristics of the current construction process at the railway work site from the video stream data. Based on these characteristics, it determines whether the railway work site is currently in a process requiring on-site supervision, generating a process status judgment result. This verifies whether the supervisors are corresponding to the correct required process. Finally, based on the supervisor identification results, three-dimensional spatial coordinates, and process status judgment results, it determines whether the supervisors comply with on-site supervision standards, thereby improving the authenticity, real-time performance, and effectiveness of railway supervision work.

[0113] In one exemplary embodiment, such as Figure 3 As shown, the video acquisition module includes a panoramic coverage submodule 201, a region focusing submodule 202, and an adaptive illumination adjustment submodule 203; among which,

[0114] The panoramic coverage submodule 201 is used to deploy at high-level deployment points around the perimeter of the railway operation site to collect wide-angle video stream data of the railway operation site without blind spots, so as to record the overall situation of supervision activities and construction operations at the railway operation site.

[0115] The video acquisition module consists of three collaborative sub-modules to adapt to the complex and ever-changing environment of railway operation sites and the need for high-precision recognition.

[0116] Among them, the panoramic coverage sub-module is deployed at a high point on the perimeter of the construction area. It uses a wide-angle lens to conduct video monitoring of the entire work surface without blind spots and over a wide area, comprehensively capturing the overall situation of the activities of the supervisors and the construction work, and providing basic video data for subsequent global behavior analysis.

[0117] The area focusing submodule 202 is used to collect local video streams from the supervisor's on-site location at the railway operation site, so as to record at least one of the facial features and behavioral details of the supervisor at the supervisor's on-site location;

[0118] Among them, the regional focusing submodule is deployed near the key process operation points and equipped with high-definition camera equipment with optical zoom capability. It performs close-up, high-resolution local video stream acquisition of the construction site, ensuring that even under interference conditions such as personnel wearing safety helmets or in a side-facing posture, their facial features and subtle behaviors can still be clearly recorded, meeting the image quality requirements for identity recognition and behavior discrimination.

[0119] The illumination adaptive adjustment submodule 203 is used to acquire at least one of the ambient illumination parameters and meteorological sensing parameters; and dynamically adjust the imaging optical control strategy based on at least one of the ambient illumination parameters and meteorological sensing parameters to maintain the video imaging quality of the video stream data.

[0120] The adaptive lighting adjustment submodule is embedded in the aforementioned high-definition camera device. It can sense changes in ambient light intensity and weather conditions in real time, and dynamically adjust imaging parameters such as exposure time, gain, white balance, and infrared illumination accordingly. This forms an adaptive imaging optical control strategy, thereby maintaining the clarity and usability of the video stream in complex lighting scenarios such as day-night cycles, strong backlight, rain, and fog. It ensures that multi-angle video data always meets the video imaging quality assurance indicators required for subsequent intelligent analysis by the system.

[0121] In this embodiment, the three sub-modules—panoramic coverage, region focusing, and adaptive illumination adjustment—work together to construct a comprehensive, detailed, and environmentally robust video perception foundation.

[0122] In one exemplary embodiment, such as Figure 4 As shown, the personnel identification module includes a keyframe extraction submodule 301, a supervisor's facial feature enhancement submodule 302, and a bystander qualification verification submodule 303; wherein:

[0123] Keyframe extraction submodule 301 is used to extract keyframe images from video stream data based on at least one of time series and viewpoint diversity.

[0124] The personnel identification module consists of three closely connected sub-modules, forming a complete processing link from the original video to the determination of a trusted identity.

[0125] The keyframe extraction submodule first extracts high-quality keyframe images containing supervisors from the multi-angle video stream output by the video acquisition module, based on the principles of time series and perspective diversity. This ensures that the selected samples cover different angles, poses, and lighting conditions, providing rich and representative visual input for subsequent recognition.

[0126] It should be noted that the time series principle refers to extracting keyframes at fixed intervals and in an orderly manner along the time dimension, ensuring that the time sequence between frames is coherent, without time gaps, jumps, or missing segments, and fully reproducing the entire process of changes in the supervisor's and on-site construction status over time. For example, a uniform time step can be set for sampling, which can be 2 seconds.

[0127] The principle of perspective diversity refers to simultaneously extracting key frames from cameras in different directions at the same or similar time points, using different shooting perspectives to supplement each other's image information, and making up for the problems of occlusion, blind spots, and image limitations that exist in a single perspective.

[0128] The supervisor's face feature enhancement submodule 302 is used to robustly enhance the face region in the key frame image using a deep learning-based supervisor identification algorithm to generate the supervisor's face features.

[0129] Among them, the supervisor's face feature enhancement submodule is based on a deep learning-based supervisor identification algorithm. It performs targeted robust enhancement processing on the face region in key frames, including super-resolution reconstruction, occlusion compensation and pose normalization for image degradation caused by typical interference factors in railway sites such as wearing a safety helmet, side profile, occlusion or low light. This significantly improves the recognizability of faces under complex working conditions and generates stable and highly discriminative structured supervisor face features.

[0130] Furthermore, the deep learning-based supervisor identification model can include a face preprocessing deep learning network and a face feature extraction deep learning network. The face preprocessing deep learning network is used to complete super-resolution reconstruction, occlusion compensation, pose normalization, etc. of blurred or occluded faces, while the face feature extraction deep learning network is used to generate highly discriminative structured face features from the face information obtained by the face processing deep learning network.

[0131] Among them, the deep learning network for face preprocessing addresses issues such as blurred faces, occlusion by safety helmets / masks, and pitch / turn of personnel at railway sites. It can be constructed by connecting three functional sub-modules: a super-resolution reconstruction network, an occlusion compensation network, and a pose normalization network.

[0132] (1) Super-resolution reconstruction network: Select classic image super-resolution networks (such as SRCNN, Real-ESRGAN) to restore high-definition face details and improve basic clarity by using low-resolution and blurry faces as input.

[0133] (2) Occlusion compensation network: Image restoration deep learning models (such as GAN and Transformer restoration network) are used to perform pixel completion and feature restoration for areas occluded by masks, helmets and debris, and restore the effective area of ​​the face.

[0134] (3) Pose normalization network: Based on the face key point detection network (such as MTCNN, RetinaFace) to locate the key points of the facial features, combined with the face pose correction model, the non-frontal face with deviation and pitch is converted into a standard frontal face pose, and the face perspective is unified.

[0135] A deep learning network for face preprocessing is constructed by connecting three functional sub-modules: a super-resolution reconstruction network, an occlusion compensation network, and a pose normalization network.

[0136] The deep learning network for face feature extraction can use any one of ArcFace, CosFace, or FaceNet as its core. The standard high-definition face image processed by the face preprocessing deep learning network is input into the deep learning network for face feature extraction to extract deep, fine-grained features of the face. These features are then embedded into a metric learning layer to transform the face into a fixed-dimensional structured feature vector, thereby obtaining highly discriminative structured face features.

[0137] Subsequently, the highly discriminative structured facial features output by the deep learning network for facial feature extraction can be compared with a pre-built database of supervisor qualifications to obtain the results of supervisor identification.

[0138] The on-site qualification verification submodule 303 is used to compare the facial features of the supervisor with the registration feature templates pre-stored in the supervisor qualification database. If the similarity between the supervisor's facial features and the registration feature template is greater than a preset similarity threshold, the authorization information for the current construction procedure is obtained. Based on the authorization information for the current construction procedure, the supervisor's facial features are used to verify the supervisor's authorization qualification for the current construction procedure.

[0139] The on-site supervision qualification verification submodule performs a high-precision similarity comparison between the facial features and the registration feature templates pre-stored in the supervisor's qualification database. It further integrates the information of the list of on-site supervisors authorized for the current construction task for dual permission verification. Only when the biometric features match the registration feature templates in the supervisor's qualification database and the person has the authorization to supervise the current construction task is the person determined to be a valid on-site supervisor. This enables accurate, reliable, and business-compliant automatic determination of the identity of personnel in the video stream data.

[0140] In this embodiment, the keyframe extraction submodule extracts high-quality keyframe images containing supervisors, ensuring that the selected samples cover different angles, poses, and lighting conditions, providing rich and representative visual input for subsequent recognition. The supervisor face feature enhancement submodule performs super-resolution reconstruction, occlusion compensation, and pose normalization for blurred or occluded faces, significantly improving facial recognizability under complex conditions and generating stable, highly discriminative structured supervisor face features. The on-site qualification verification submodule performs a high-precision similarity comparison between these face features and pre-stored registration feature templates in the supervisor qualification database, further integrating the authorized on-site personnel list information for the current construction task for dual authorization verification, thereby achieving accurate, reliable, and business-compliant automatic determination of personnel identity in the video stream data.

[0141] In one exemplary embodiment, such as Figure 5 As shown, the spatial positioning analysis module includes an electronic fence configuration submodule 401, a target detection and tracking submodule 402, and a spatial mapping submodule 403; wherein:

[0142] The electronic fence configuration submodule 401 is used to set the logical fence boundary corresponding to the on-site station area in the spatial coordinate reference system of the railway operation site based on the on-site station area requirements of the current construction process, and generate dynamic electronic fence data bound to the current construction process.

[0143] The spatial positioning analysis module consists of three progressively functioning sub-modules that work together to accurately determine the spatial location of supervisors at the railway work site.

[0144] The electronic fence configuration submodule first dynamically presets the logical fence boundary corresponding to the station area in the unified spatial coordinate reference system of the railway operation site according to the specific requirements of the current construction process for the station area, and generates dynamic electronic fence data with spatiotemporal context correlation, clearly defining the physical space range of the effective station under this process.

[0145] The specific requirements for the current construction procedure can be obtained from the cloud server or other systems. It should be noted that these specific requirements include management regulations and constraints such as on-site safety control, the scope of on-site supervision, and restricted areas for different railway construction procedures.

[0146] Among them, the unified spatial coordinate reference system for railway operation sites is a global standard three-dimensional coordinate system covering the entire railway operation site and all monitoring points. All equipment, personnel, and areas share the same coordinate origin, coordinate axes, and scale units.

[0147] Among them, the dynamically preset logical fence boundary can be changed, enabled, or disabled in real time according to process switching, operation start and stop, and area adjustment, rather than a fixed static fence boundary.

[0148] It should be noted that the logical fence boundary bound to the construction process is a corresponding logical fence boundary for each construction process in the system, based on the specific requirements of that construction process for the on-site monitoring area, and this logical fence boundary is bound to the construction process. When the current construction process is identified, the logical fence boundary corresponding to the current construction process is automatically retrieved; when the current construction process changes, the logical fence boundary is changed synchronously.

[0149] Among them, the dynamic electronic fence data with spatiotemporal context correlation is a logical fence boundary that simultaneously contains spatial location information and time-based validity information.

[0150] Spatial location information may include fence vertex coordinates, area shape, and size.

[0151] The time-based effective information may include the effective period of the fence, the expiration period, and the time rules for starting and stopping the fence according to the process.

[0152] The target detection and tracking submodule 402 is used to perform real-time cross-frame target tracking of the identified supervisors in the video frames corresponding to the video stream data, and output two-dimensional trajectory data in each video frame.

[0153] The target detection and tracking submodule, based on the identification of the supervisory personnel in the personnel identification module, performs real-time and continuous cross-frame tracking of the target in the video stream data, stably outputs the two-dimensional pixel coordinates of the personnel in each video frame, and forms coherent two-dimensional trajectory data, providing a highly temporally consistent position input for subsequent spatial mapping.

[0154] The spatial mapping submodule 403 is used to obtain the calibration parameters of the camera, which is used to collect video stream data. Based on the calibration parameters of the camera and the spatial coordinate reference system of the railway operation site, the two-dimensional trajectory data is estimated by coordinate mapping and output three-dimensional spatial coordinates. The three-dimensional spatial coordinates are compared with the dynamic electronic fence data to determine the positional relationship between the identified supervisor and the on-site monitoring area required by the current construction procedure.

[0155] The spatial mapping submodule combines pre-calibrated camera internal and external parameters with a constructed spatial coordinate benchmark system for railway operation sites (such as a track BIM model or geographic coordinate system) to convert two-dimensional trajectory data into three-dimensional spatial coordinates in the real world through coordinate mapping estimation technology. It then compares these three-dimensional spatial coordinates with the aforementioned dynamic electronic fence data to accurately determine whether the identified supervisors are actually within the on-site monitoring area specified in the current construction procedure, thus achieving precise positioning and compliance verification from image pixels to the engineering space.

[0156] Specifically, the camera's intrinsic and extrinsic parameters are first calibrated, and the camera's pose is bound to the spatial coordinate reference system of the railway operation site. Then, the two-dimensional trajectory data of the supervisor is processed for distortion removal, noise reduction, and smoothing. Based on the pinhole camera imaging model, the processed two-dimensional trajectory data is converted into the camera's local three-dimensional coordinates. Through the rotation matrix and translation vector of the camera's extrinsic parameters, the conversion of the camera's local three-dimensional coordinates to the spatial coordinate reference system is completed, resulting in three-dimensional spatial coordinates. The complete three-dimensional spatial trajectory is generated point by point and then spatially contained with the boundary of the dynamic electronic fence to verify whether the personnel are in the compliant on-site monitoring area.

[0157] In this embodiment, the electronic fence configuration submodule can clearly define the physical space range of the effective on-site supervision during the process. The target detection and tracking submodule can generate continuous two-dimensional trajectory data of the supervisors, providing a highly time-consistent position input for subsequent spatial mapping. The spatial mapping submodule converts the continuous two-dimensional trajectory data of the supervisors into a three-dimensional spatial trajectory, and then performs spatial inclusion judgment with the dynamic electronic fence boundary to verify whether the personnel are in the compliant on-site supervision area, realizing accurate positioning and compliance verification from image pixels to the engineering space.

[0158] In one exemplary embodiment, such as Figure 6 As shown, the process status perception module includes a construction process parsing submodule 501, a feature extraction submodule 502, and a semantic logic comparison submodule 503; wherein:

[0159] The construction procedure parsing submodule 501 is used to obtain the rule set of the on-site supervision associated procedures; and to generate a semantic rule library of construction procedures that require on-site supervision based on the rule set of the on-site supervision associated procedures.

[0160] Among them, the process status perception module, through three organically connected sub-modules, can automatically and accurately determine whether the construction site is in a critical process stage that requires on-site supervision.

[0161] Among them, the construction process analysis submodule can obtain the rule set of on-site supervision related processes from other systems or the cloud. For example, it can connect to the railway engineering digital construction planning system to parse out the list of processes strongly related to on-site supervision and their triggering conditions, forming a structured rule set of on-site supervision related processes.

[0162] It should be noted that the rule set for on-site supervision related processes stores all original related information such as process name, work type, start conditions, work time period, work area, risk level, and basic control requirements that are strongly related to on-site supervision.

[0163] Furthermore, a semantic rule base for construction procedures that can be understood by machines can be built based on the rule set of on-site related procedures, clarifying the business logic of "which procedures need to be monitored on-site under what conditions".

[0164] It should be noted that the construction process semantic rule base is a machine-parseable intelligent rule base formed by secondary processing, semantic modeling, logical association, and rule reasoning encapsulation based on the rule set of the adjacent related processes.

[0165] The feature extraction submodule 502 is used to extract features from video stream data based on a pre-trained construction state feature perception network to obtain construction state features. The construction state feature perception network is driven by deep learning.

[0166] The feature extraction submodule, based on the video stream data provided by the video acquisition module, uses a deep learning-driven construction status feature perception network to perform real-time analysis of the scene, simultaneously identifying the operation actions of construction personnel, the operating status of large machinery (such as track laying machines and tamping machines), and the technological features presented by the track or structure, thereby extracting construction status features covering multiple dimensions of people, machines, and materials.

[0167] As one implementation method, the construction status feature perception network includes an input layer, a shared backbone feature extraction layer, three functional branch layers, a feature fusion layer, and an output layer. The input layer receives a sequence of consecutive image frames decoded from the video stream and performs unified preprocessing.

[0168] The shared backbone feature extraction layer uses a lightweight improved convolutional network (such as ConvNeXt-T or MobileNetV4) as the global backbone. Its function is to extract basic visual features of a single frame, such as texture, contour, color, and local shape. The lightweight improved convolutional network includes shallow convolutions, deep convolutions, and a scene attention module.

[0169] (1) Shallow convolution: extract shallow detail features such as safety helmets, reflective vests, rail textures, and machine appearance;

[0170] (2) Deep convolution: extracting high-level semantic features such as machine type, personnel outline, and material stacking pattern;

[0171] (3) Scene attention module: Add railway scene mask to suppress invalid backgrounds such as sky, weeds, and blank track bed, and strengthen the feature weight of the working area.

[0172] The shared feature maps output by the shared backbone feature extraction layer are fed into the three functional branch layers respectively.

[0173] The three functional branches correspond to three parallel branches for human, machine, and object recognition, respectively. These include the construction worker perception branch, the large machinery perception branch, and the track / structure process feature branch. The construction worker perception branch can detect personnel and recognize their actions. The large machinery perception branch can classify machinery and determine its operating status. The track / structure process feature branch can extract static process features for fixed facilities.

[0174] By performing dimensional alignment and redundant information filtering on the three major branches of features through feature fusion and output layer, the features are integrated into a unified construction status feature that covers multiple dimensions of people, machines, and objects and then output.

[0175] The semantic logic comparison submodule 503 is used to compare the construction status features with the construction procedure semantic rule library to determine the supervision requirements of the current construction procedure at the railway operation site and generate the procedure status judgment result.

[0176] The semantic logic comparison submodule performs semantic logic fusion comparison between the construction status features and the aforementioned construction procedure semantic rule library. By matching the semantic consistency between the construction status features and the construction procedure semantic rule library, it comprehensively judges whether the construction site has truly entered the key procedure stage that requires on-site supervision, and outputs structured procedure status judgment results. This provides accurate, dynamic, and verifiable procedure context for the subsequent multi-dimensional data fusion judgment module, ensuring that the on-site supervision behavior verification is always anchored to the actual construction process.

[0177] The structured process status determination results can include information such as the currently identified process, process stage, on-site monitoring requirements, matching confidence level, determination conclusion, and anomaly markers.

[0178] In this embodiment, the construction procedure parsing submodule generates a semantic rule library of construction procedures requiring on-site supervision. The feature extraction submodule uses a deep learning-driven construction status feature perception network to extract construction status features covering multiple dimensions of people, machines, and objects. The semantic logic comparison submodule performs semantic logic fusion and comparison between the construction status features and the aforementioned construction procedure semantic rule library to comprehensively determine whether the construction site has truly entered the key procedure stage requiring on-site supervision, and outputs a structured procedure status judgment result to support the compliance verification of on-site supervision behavior. This provides accurate, dynamic, and verifiable procedure context for the subsequent multi-dimensional data fusion judgment module, ensuring that the on-site supervision behavior verification is always anchored to the actual construction process.

[0179] Existing technologies also have the following shortcomings: they generally lack the ability to intelligently analyze the four elements of people, location, time, and process, which makes on-site supervision a mere formality and makes it difficult to achieve closed-loop management throughout the entire process, thus seriously restricting the effectiveness and authority of railway engineering supervision.

[0180] Therefore, the problem to be solved by this invention is: how to automatically, in real time and accurately identify whether railway operation site supervisors are performing effective on-site supervision at the correct time, in the correct location and for the correct procedures, so as to achieve intelligent supervision and management that is traceable and anti-counterfeiting throughout the entire process.

[0181] Based on the aforementioned deficiencies and problems, an exemplary embodiment is proposed. In this embodiment, as follows: Figure 7 As shown, the multidimensional data fusion judgment module includes a four-dimensional element alignment submodule 601, a consistency verification submodule 602, and a side-view behavior recording submodule 603; wherein:

[0182] The four-dimensional element alignment sub-module 601 is used to obtain the system timestamp of the automatic identification system for on-site supervision behavior of railway operations; it synchronizes the time and unifies the spatial coordinates of the identification results of supervisors, three-dimensional spatial coordinates, system timestamps, and process status judgment results to form a spatiotemporally aligned four-dimensional element fusion dataset.

[0183] The multi-dimensional data fusion judgment module consists of three logically connected sub-modules, which realizes automated, structured and traceable judgment of the supervision and on-site behavior.

[0184] The four-dimensional element alignment submodule first gathers key information from various front-end modules, including the supervisor's identification output from the personnel identification module, the three-dimensional spatial coordinates provided by the spatial positioning analysis module, the high-precision timestamp built into the system, and the process status judgment results generated by the process status perception module. It then performs time synchronization and spatial coordinate system unification on these four types of heterogeneous data, eliminating the time offset or coordinate deviation caused by multi-source acquisition, and forming a strictly spatiotemporally aligned four-dimensional element fusion dataset of "person-location-time-process".

[0185] The consistency verification submodule 602 is used to construct a four-dimensional consistency verification mechanism based on the business logic of on-site supervision; based on the four-dimensional consistency verification mechanism, the consistency verification of the four-dimensional element fusion dataset is performed to determine the validity of the on-site supervision behavior performed by the supervisor under the compliance judgment condition, and the multi-dimensional data fusion judgment result is output.

[0186] Based on the business logic defined in the railway supervision specifications, the consistency verification submodule constructs a four-dimensional consistency verification mechanism. It performs multi-condition joint reasoning on the four-dimensional element fusion dataset to strictly verify whether the supervisors simultaneously meet the four core conditions: "correct identity, in the correct on-site supervision area, appearing in the correct time window and corresponding to the correct on-site supervision process." These are the compliance judgment conditions, thereby accurately determining whether their on-site supervision behavior is valid.

[0187] Furthermore, constructing a four-dimensional consistency verification mechanism involves constructing a four-dimensional consistency verification function, the calculation formula for which the four-dimensional consistency verification function is constructed is as follows:

[0188]

[0189] Where P represents the identification of the identified supervisor, L represents the three-dimensional spatial coordinates, T represents the current system timestamp, S represents the construction process stage, A(S) represents the set of supervisors authorized for process S, R(S) represents the set of on-site supervision areas defined by process S, and W(S) represents the set of on-site supervision time windows allowed by process S. This indicates an indicator function.

[0190] The side-by-side behavior recording submodule 603 is used to encapsulate the multi-dimensional data fusion judgment result, the four-dimensional element fusion dataset, and the system timestamp into a side-by-side behavior record to connect with the abnormal behavior early warning function.

[0191] The on-site behavior recording submodule encapsulates the multi-dimensional data fusion judgment results along with the complete four-dimensional element fusion data and system timestamps in a structured manner to generate standardized on-site behavior records. It not only supports real-time connection to the abnormal behavior early warning system, but can also be used for post-event auditing, responsibility tracing and data reporting to the supervision platform. Ultimately, it realizes a closed-loop, automated and verifiable intelligent supervision of the entire process of railway operation site supervision on-site behavior from perception and judgment to archiving.

[0192] This embodiment receives the supervisor's identity information from the personnel identification module, the three-dimensional spatial coordinates output by the spatial positioning analysis module, the system timestamp, and the process status judgment result generated by the process status perception module through the four-dimensional element alignment submodule. It synchronizes the four-dimensional element information of the on-site supervision in time and unifies the spatial coordinates to form a spatiotemporally aligned four-dimensional element fusion dataset. The consistency verification submodule constructs a four-dimensional consistency verification mechanism for personnel location, time, and process, and performs logical reasoning and compliance verification on the four-dimensional element fusion dataset to determine whether the supervisor has performed valid on-site supervision behavior under the three-positive compliance judgment conditions. The on-site supervision behavior recording submodule encapsulates the multi-dimensional data fusion judgment result, the four-dimensional element fusion dataset, and the system timestamp into an on-site supervision behavior record to connect with the abnormal behavior early warning function, thereby completing the closed-loop data archiving of the entire process of on-site supervision behavior at railway operation sites.

[0193] In one exemplary embodiment, such as Figure 8 As shown, the automatic identification method for on-site supervision behavior at railway work sites includes steps S701 to S707, wherein:

[0194] Step S701: Acquire video stream data at the railway operation site;

[0195] This can be achieved by deploying camera-equipped data acquisition terminals at railway work sites to obtain video stream data. For example, video can be acquired by deploying cameras, personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices.

[0196] Step S702: Using a deep learning-based supervisor identification model, the supervisors in the video stream data are identified to obtain the supervisor identification results.

[0197] Among them, a deep learning-based supervisor identification model can be used to determine the identity of personnel in video stream data, identify whether they are qualified supervisors to stand by, and obtain the supervisor identification results.

[0198] Step S703: If the identification result of the supervisor includes the identified supervisor, then obtain the two-dimensional spatial coordinates of the identified supervisor in the video stream data;

[0199] Among them, it can perform real-time cross-frame target tracking of the identified supervisors in the video frames corresponding to the video stream data, and output the two-dimensional spatial coordinates in each video frame.

[0200] Step S704: Perform coordinate mapping estimation on the two-dimensional spatial coordinates and output the three-dimensional spatial coordinates of the identified supervisors;

[0201] Among them, the calibration parameters of the camera can be obtained, and the camera is used to collect video stream data; based on the calibration parameters of the camera and the spatial coordinate reference system of the railway operation site, the two-dimensional spatial coordinates are estimated by coordinate mapping and output three-dimensional spatial coordinates.

[0202] Step S705: Obtain the construction status characteristics of the railway operation site from the video stream data;

[0203] Among them, the construction status feature perception network can be pre-trained to extract features from video stream data to obtain construction status features. The construction status feature perception network is driven by deep learning.

[0204] Step S706: Generate process status determination results based on construction status characteristics and the supervision requirements of the current construction process at the railway operation site;

[0205] This includes obtaining the rule set of on-site supervision associated processes; generating a semantic rule library of construction processes requiring on-site supervision based on the rule set of on-site supervision associated processes; comparing the construction status characteristics with the semantic rule library of construction processes to determine the supervision requirements of the current construction process at the railway operation site and generating the process status determination result.

[0206] Step S707: Based on the identification results of the supervisors, the three-dimensional spatial coordinates, and the process status judgment results, determine the effectiveness of the supervisors' on-site supervision behavior under the compliance judgment conditions, and output the multi-dimensional data fusion judgment result.

[0207] One implementation method is to construct a consistency verification mechanism based on the identification results of supervisors, three-dimensional spatial coordinates, and process status judgment results. The consistency verification mechanism is used to determine the effectiveness of supervisors' on-site supervision behavior under compliance judgment conditions and output multi-dimensional data fusion judgment results.

[0208] As another implementation method, the system timestamp of the automatic identification system for on-site supervision behavior of railway operation can be obtained. Based on the system timestamp, the identification result of the supervisor, the three-dimensional spatial coordinates, and the process status judgment result, a four-dimensional consistency verification mechanism can be constructed. The effectiveness of the supervisor's on-site supervision behavior under the compliance judgment condition can be determined through the four-dimensional consistency verification mechanism, and the multi-dimensional data fusion judgment result can be output.

[0209] This invention, through the construction of an automatic identification system integrating a video acquisition module, a personnel identification module, a spatial positioning analysis module, a process status perception module, and a multi-dimensional data fusion judgment module, achieves intelligent full-process supervision of on-site supervision behavior at railway work sites. The system can automatically determine whether the supervisor's identity is compliant, locate in real time whether they are in the designated on-site supervision area for the current process, dynamically identify whether the construction is in a critical stage requiring on-site supervision, and accurately determine the validity of on-site supervision behavior based on a four-dimensional consistency verification mechanism of "person-location-time-process," completely abandoning the inefficient traditional model relying on manual sign-in, paper records, or post-event video review. Therefore, it not only effectively eliminates regulatory loopholes such as absenteeism, impersonation, and perfunctory performance of duties, but also significantly improves the objectivity, real-time nature, and traceability of supervision behavior, providing reliable technical assurance for the quality and safety of railway engineering construction, while supporting the digital, closed-loop, and automated management of supervision work.

[0210] Furthermore, this embodiment of the invention constructs a real-time monitoring system for the safety posture of railway workers that integrates video access edge analysis, intelligent recognition capability set, multi-dimensional business fusion linkage, and intelligent early warning closed-loop management. This system achieves full-chain technology integration from video stream access, lightweight edge inference, scenario-based model management, context fusion discrimination to hierarchical early warning and closed-loop handling.

[0211] On the one hand, low-latency and high-efficiency localized analysis of posture and behavior is completed at the edge, making effective use of existing video resources; on the other hand, relying on the railway operation compliance context quadruple and posture-rule mapping mechanism, the accuracy of the judgment of typical unsafe behaviors such as bending over, running, and abnormal standing posture is significantly improved, and the false alarm rate is greatly reduced.

[0212] Meanwhile, through no-code training interaction and cloud-edge collaborative strategy distribution, on-site operation and maintenance personnel can independently complete model iteration and strategy updates, get rid of dependence on vendor closed architecture, and truly realize the adaptive evolution of intelligent analysis capabilities and closed-loop management of the entire process for complex working conditions of heavy-haul railways.

[0213] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0214] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an automatic identification method for on-site supervision behavior in railway operations.

[0215] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0216] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0217] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0218] In one embodiment, such as Figure 10 As shown, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method.

[0219] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0220] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0221] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0222] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An automatic identification system for on-site supervision behavior in railway operations, characterized in that, The automatic identification system for on-site supervision behavior at railway work sites includes a video acquisition module, a personnel identification module, a spatial positioning analysis module, a process status perception module, and a multi-dimensional data fusion and judgment module; among which... The video acquisition module is used to acquire video stream data at the railway operation site; The personnel identification module is used to identify the personnel in the video stream data using a deep learning-based supervisory personnel identification model, and obtain the supervisory personnel identification result. The spatial positioning analysis module is used to obtain the two-dimensional spatial coordinates of the identified supervisor in the video stream data if the supervisor identification result includes the identified supervisor; perform coordinate mapping estimation on the two-dimensional spatial coordinates, and output the three-dimensional spatial coordinates of the identified supervisor; The process status perception module is used to acquire the construction status characteristics of the railway operation site in the video stream data; and generate a process status judgment result based on the construction status characteristics and the supervision requirements of the current construction process at the railway operation site. The multi-dimensional data fusion judgment module is used to determine the effectiveness of the supervisor's on-site supervision behavior under compliance judgment conditions based on the supervisor's identity recognition result, the three-dimensional spatial coordinates, and the process status judgment result, and outputs the multi-dimensional data fusion judgment result.

2. The system according to claim 1, characterized in that, The video acquisition module includes a panoramic coverage submodule, a region focusing submodule, and an adaptive illumination adjustment submodule; wherein... The panoramic coverage submodule is deployed at a high-level perimeter of the railway work site to collect wide-angle, blind-spot-free video stream data of the railway work site, so as to record the overall situation of supervision activities and construction operations at the railway work site. The area focusing submodule is used to collect local video streams from the supervisory on-site locations at the railway operation site, so as to record at least one of the facial features and behavioral details of the supervisory personnel at the supervisory on-site locations; The adaptive illumination adjustment submodule is used to acquire at least one of the ambient illumination parameters and the meteorological sensing parameters; and dynamically adjust the imaging optical control strategy based on the ambient illumination parameters and the meteorological sensing parameters to maintain the video imaging quality of the video stream data.

3. The system according to claim 1, characterized in that, The personnel identification module includes a keyframe extraction submodule, a supervisor's facial feature enhancement submodule, and a site visit qualification verification submodule; wherein: The keyframe extraction submodule is used to extract keyframe images from the video stream data based on at least one of time series and perspective diversity. The supervisor's facial feature enhancement submodule is used to robustly enhance the facial region in the keyframe image based on the deep learning-based supervisor identification algorithm to generate supervisor facial features. The on-site qualification verification submodule is used to compare the facial features of the supervisor with the registration feature templates pre-stored in the supervisor qualification database; if the similarity between the facial features of the supervisor and the registration feature templates is greater than a preset similarity threshold, the authorization information of the current construction procedure is obtained; based on the authorization information of the current construction procedure, the supervisor's facial features are used to verify the authorization qualification of the supervisor for the current construction procedure.

4. The system as described in claim 1, characterized in that, The spatial positioning analysis module includes an electronic fence configuration submodule, a target detection and tracking submodule, and a spatial mapping submodule; wherein: The electronic fence configuration submodule is used to set the logical fence boundary corresponding to the on-site area in the spatial coordinate reference system of the railway operation site based on the on-site area requirements of the current construction process, and generate dynamic electronic fence data bound to the current construction process. The target detection and tracking submodule is used to perform real-time cross-frame target tracking of the identified supervisors in the video frames corresponding to the video stream data, and output two-dimensional trajectory data in each video frame. The spatial mapping submodule is used to obtain the calibration parameters of the camera, which is used to collect the video stream data; based on the calibration parameters of the camera and the spatial coordinate reference system of the railway operation site, the two-dimensional trajectory data is estimated by coordinate mapping, and the three-dimensional spatial coordinates are output; the three-dimensional spatial coordinates are compared with the dynamic electronic fence data to determine the positional relationship between the identified supervisor and the on-site monitoring area required by the current construction procedure.

5. The system as described in claim 1, characterized in that, The process status perception module includes a construction process parsing submodule, a feature extraction submodule, and a semantic logic comparison submodule; wherein: The construction process parsing submodule is used to obtain the rule set of on-site supervision related processes; and to generate a semantic rule library of construction processes that require on-site supervision based on the rule set of on-site supervision. The feature extraction submodule is used to extract features from the video stream data based on a pre-trained construction state feature perception network to obtain the construction state features. The construction state feature perception network is driven by deep learning. The semantic logic comparison submodule is used to compare the construction status features with the construction procedure semantic rule library to determine the supervision requirements of the current construction procedure at the railway operation site and generate the procedure status determination result.

6. The system as described in claim 1, characterized in that, The multi-dimensional data fusion judgment module includes a four-dimensional element alignment submodule, a consistency verification submodule, and a side-site behavior recording submodule; wherein: The four-dimensional element alignment submodule is used to obtain the system timestamp of the automatic identification system for on-site supervision behavior of railway operations; and to synchronize the time and unify the spatial coordinates of the identification results of the supervisors, the three-dimensional spatial coordinates, the system timestamp, and the process status judgment results to form a spatiotemporally aligned four-dimensional element fusion dataset. The consistency verification submodule is used to construct a four-dimensional consistency verification mechanism based on the business logic of on-site supervision; and to perform consistency verification on the four-dimensional element fusion dataset based on the four-dimensional consistency verification mechanism to determine the effectiveness of the on-site supervision behavior performed by the supervisor under the compliance judgment conditions, and output the multi-dimensional data fusion judgment result. The side-site behavior recording submodule is used to encapsulate the multi-dimensional data fusion judgment result, the four-dimensional element fusion dataset, and the system timestamp into a side-site behavior record to connect with the abnormal behavior early warning function.

7. A method for automatically identifying on-site supervision behavior at railway work sites, characterized in that, The method includes the following steps: Acquire video stream data at railway operation sites; The supervisors in the video stream data are identified using a deep learning-based supervisor identification model, and the supervisor identification results are obtained. If the identification result of the supervisor includes an identified supervisor, then the two-dimensional spatial coordinates of the identified supervisor in the video stream data are obtained; The coordinate mapping estimation is performed on the two-dimensional spatial coordinates, and the three-dimensional spatial coordinates of the identified supervisors are output. Obtain the construction status characteristics of the railway operation site from the video stream data; Based on the construction status characteristics and the supervision requirements of the current construction procedure at the railway work site, a procedure status determination result is generated; Based on the identification results of the supervisors, the three-dimensional spatial coordinates, and the process status determination results, the effectiveness of the supervisors' on-site supervision behavior under compliance determination conditions is determined, and a multi-dimensional data fusion determination result is output.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 7.