Human cabin monitoring system based on visual neural network

Abstract

The invention relates to the technical field of shield tunneling machine man cabin monitoring, in particular to a man cabin monitoring system based on an optic neural network, which is characterized in that light scattering prior is constructed based on dust particle size, dust concentration and relative positions of a camera and light by collecting original images in a shield tunneling machine man cabin and data of dust particle size, concentration, temperature and smoke concentration; the transmissivity is inversed in combination with a monocular depth estimation network, and the image is further corrected in a cross-scale manner through a residual pyramid compensation network, so that a dust removal image sequence suitable for a complex light scattering environment is obtained; and then accurate identification of abnormal staying of personnel and tool missing is realized based on a three-dimensional space-time diagram convolutional network and a causal attention mechanism, and a Transform model is utilized to fuse vision, temperature and smoke concentration data to realize rapid and accurate prediction of fire and in-cabin leakage risks.

Description

Technical Field

[0001] The present invention relates to the technical field of shield machine cabin monitoring, and in particular to a cabin monitoring system based on a visual neural network. Background Art

[0002] As a critical piece of equipment in underground tunnel construction, the shield machine's cabin area carries out crucial tasks such as operational control and on-site inspections, directly impacting construction safety and personnel risks. However, the shield machine's enclosed operating environment and limited ventilation easily generate large amounts of dust, heat, and smoke during operation. These can severely interfere with the camera system's image clarity, impacting operator visibility and remote monitoring, and even masking potential abnormal human behavior and safety risks.

[0003] Under these complex working conditions, traditional camera surveillance systems deployed inside the manhole have the following shortcomings. First, high-concentration dust particles produce severe forward scattering and multi-angle reflection under strong light, resulting in an overall gray image, reduced contrast, and blurred edges, affecting the camera's ability to clearly image workers, tools, and equipment inside the manhole. In particular, in tasks such as key component identification and personnel behavior analysis, reduced image clarity directly leads to reduced system availability and recognition accuracy. Furthermore, when emergencies such as fires and leaks occur, existing systems often use single-channel alarms via temperature sensors or smoke alarms. These systems lack the ability to jointly model image visual cues and sensor data, resulting in delayed responses and high false alarm rates within the shield machine's manhole. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a cabin monitoring system based on visual neural network.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A cabin monitoring system based on a visual neural network, comprising:

[0007] Data acquisition module, used to collect original image frames, dust particle size, dust concentration, temperature and smoke concentration data inside the shield machine cabin;

[0008] The image processing module is used to construct a light scattering prior based on dust particle size, dust concentration, and the relative position of the camera and light. It combines the monocular depth estimation network to perform transmittance inversion on the original image and descatters the original image through the residual pyramid compensation network to obtain a dust-free image sequence.

[0009] A three-dimensional synchronization module is used to extract the operator's skeletal key points, tool coordinates, and hatch status based on the dust removal image sequence, and to construct a cross-frame spatiotemporal graph structure using the skeletal key points and tool coordinates as nodes. The structure is input into a three-dimensional spatiotemporal graph convolutional network and outputs a continuous temporal behavior vector.

[0010] A human anomaly recognition module, based on a causal attention mechanism network, uses a time-series behavior vector as input and identifies and outputs human anomaly states, such as abnormal personnel stays and left-behind tools.

[0011] An environmental recognition module is used to construct environmental feature parameters from the dust-removed image sequence and the synchronously collected temperature and smoke concentration, and input them into a Transformer model to extract and generate time series features. The module then identifies abnormal environmental conditions based on the time series features, including fire and cabin leakage.

[0012] The alarm module is used to issue cabin alarms based on abnormal human and environmental conditions.

[0013] Furthermore, the light scattering prior is constructed based on the dust particle size, dust concentration, and the relative position of the camera and the light, including the following steps:

[0014] Obtain the particle size distribution and concentration parameters corresponding to the current image frame, and construct the scattering influence function of dust in the image by combining the relative angle and distance between the camera and the light source;

[0015] Based on the scattering influence function, the brightness attenuation and contrast loss of different pixel areas are spatially modeled to generate a light scattering prior map for constraining image transmittance inversion. The light scattering prior map includes a brightness attenuation map and a contrast loss distribution map.

[0016] Furthermore, the scattering influence function is as follows:

[0017] S(x)=exp(–k(d,θ)×C×L);

[0018] Where S(x) is the scattering attenuation of image brightness caused by dust at pixel position x; d is the average dust particle size; C is the dust mass concentration; θ is the scattering angle between the camera's visual axis and the light direction; L is the estimated optical path length from the camera to the target scene; and k(d,θ) is the scattering attenuation coefficient from the dust particle size and scattering angle lookup table.

[0019] Furthermore, the spatial modeling of brightness attenuation and contrast loss in different pixel areas based on the scattering influence function includes the following steps:

[0020] According to the camera angle of view, the target scene range in the current image frame is determined, and the target scene range is divided into several sub-areas;

[0021] Calculating the brightness attenuation value of pixels in each sub-area according to the scattering influence function, and generating a corresponding brightness attenuation map pixel by pixel;

[0022] According to the brightness attenuation map, the degree of local contrast reduction of each sub-region is calculated, and a spatial contrast loss distribution map is generated.

[0023] Furthermore, the transmittance inversion of the original image in combination with the monocular depth estimation network includes the following steps:

[0024] Calculating a weight coefficient for each pixel based on the brightness attenuation map and the spatial contrast loss distribution map, and applying the weight coefficient to the original image frame pixel by pixel to obtain a weighted input image frame;

[0025] According to the weighted input image frame, a monocular depth estimation network is used to predict the depth value pixel by pixel to obtain a depth map corresponding to each pixel of the original image frame;

[0026] Based on the pixel values of the depth map and the light scattering prior map, the transmittance value is calculated pixel by pixel to obtain a transmittance map.

[0027] Furthermore, the residual pyramid compensation network is trained by the following steps:

[0028] The original image frame and the corresponding transmittance map are combined to form the training input, and the corresponding clear image sequence in the same scene is used as the supervision label to construct the training sample set;

[0029] Inputting the training sample set into a residual pyramid compensation network, wherein the network comprises multiple scale residual channels and feature reconstruction branches for extracting image structure information and illumination degradation residual features at different spatial scales;

[0030] Residual mapping and image fusion operations are performed in each scale channel to generate a multi-scale residual compensation map, and image reconstruction is performed under the skip connection mechanism to output a dust-free image sequence for comparative supervision;

[0031] Constructing a joint loss function that includes pixel reconstruction error, structural similarity loss, and edge preservation loss, and performing error feedback on the dust-free image sequence and label image output by the network;

[0032] Gradient descent and parameter update are performed based on the joint loss function.

[0033] Furthermore, the three-dimensional synchronization module is used to perform the following steps:

[0034] Based on the dust removal image sequence, the skeleton key points of the operator are extracted frame by frame through the skeleton posture estimation network, and the three-dimensional coordinates of the tool and the hatch opening and closing status are obtained through instance segmentation and target detection algorithms;

[0035] The skeleton key points and tool coordinates are paired within a frame and tracked between frames in a time series to construct a node set, and a graph structure is established based on the spatial adjacency relationship and temporal correspondence between the nodes to form a cross-frame spatiotemporal graph containing spatial edges and temporal edges;

[0036] The spatiotemporal graph is input into a three-dimensional spatiotemporal graph convolutional network, and the behavior change characteristics and spatial structure information across time steps are extracted through graph convolution operations, and a continuous time series behavior vector is output.

[0037] Furthermore, the causal attention mechanism network is constructed by the following steps:

[0038] Input the continuous time series behavior vector into the position encoding module, embed the time step using the sine and cosine functions, and generate a time-dependent behavior representation vector;

[0039] Based on the behavior representation vector, a query vector, a key vector, and a value vector are constructed, and a causal mask is used in the multi-head self-attention structure to limit the current time step to only focus on historical information, thereby generating a causal attention matrix;

[0040] Applying the causal attention matrix to the value vector to complete context weighting processing and output a sequence of abnormal behavior candidate vectors containing temporal dependency features;

[0041] The abnormal behavior candidate vector sequence is input into the fully connected classification layer, and feature alignment and probability matching are performed with the preset abnormal behavior template to identify the target abnormal state, which includes abnormal stay of personnel and left-behind tools.

[0042] Furthermore, the environment recognition module is used to perform the following steps:

[0043] Extracting smoke distribution characteristics and brightness gradients within the image region based on the dust removal image sequence, and constructing a multimodal environment feature vector sequence in combination with temperature and smoke concentration data;

[0044] Performing time alignment processing on the environmental feature vector sequence and inputting it into a Transformer model including a multi-head self-attention mechanism and a feedforward network structure for cross-modal time series modeling;

[0045] In the encoder of the Transformer model, the self-attention layer is used to obtain the dependency of environmental state changes between different time steps and generate a global feature expression with time weights;

[0046] The global feature expression is input into the multi-layer perceptron classification, and combined with the preset fire and cabin leakage event labels for feature matching and probability prediction, the abnormal state of the environment and its confidence level are output.

[0047] Furthermore, the cabin alarm includes triggering a voice broadcast in the cabin and a highlighted border prompt on the monitoring screen.

[0048] The beneficial effects of the present invention are as follows: by collecting raw image frames, dust particle size, dust concentration, temperature, and smoke concentration data from the cabin, the present invention constructs a more targeted light scattering prior model based on the real-time collected dust particle size, concentration, and the relative position of the camera and light. This model clearly quantifies the image degradation mechanism of strong forward scattering and multi-angle reflection actually existing in the cabin, provides real-world constraints for subsequent transmittance inversion, and avoids the modeling distortion problem caused by the use of general empirical formulas in the prior art. Based on the above-mentioned light scattering prior model, a monocular depth estimation network is further used, with the raw image frames weightedly processed as input, to predict image depth pixel by pixel and invert transmittance, thereby establishing a transmittance map with clear spatial structural consistency. This design overcomes the large transmittance estimation biases of traditional blind estimation methods or purely data-driven models, making it more reliable and robust in the dynamic dust environment of a shield machine. Descattering is then performed using a residual pyramid compensation network to generate a dust-free image sequence. This network employs a multi-scale architecture and introduces cross-layer skip connections to achieve cross-scale residual mapping from shallow textures to deep semantics and image detail compensation. This design is particularly well-suited for the complex and heterogeneous visual scenes found in the shield machine manhole, such as the numerous metal surface reflections, fine tool boundaries, and local details such as human posture. Furthermore, during training, this network uses a loss function combining pixel reconstruction error, structural similarity, and edge-preserving loss to accurately guide the network in learning structural restoration strategies for images at all scales, unlike existing techniques that suffer from oversmoothing or severe detail loss. Through a progressive approach of "physical scattering modeling - deep transmittance inversion - multi-scale residual compensation," this method achieves a reliable image enhancement method in the shield machine manhole environment, achieving refined image quality and structural restoration. Based on the dust-free image sequence, the cross-frame spatiotemporal graph with a time series structure is constructed by dynamically mapping the key points of the personnel skeleton and the three-dimensional coordinates of the tools. The collaborative temporal modeling of multiple entities and multiple actions is realized with the help of the three-dimensional spatiotemporal graph convolutional network. The dynamic anomaly recognition is then combined with the causal attention mechanism to effectively improve the recognition accuracy and response time of complex behaviors such as "abnormal stay of personnel" and "left behind tools". In terms of environmental risk identification, the present invention proposes an anomaly recognition mechanism based on multimodal feature fusion, combining the smoke distribution and brightness change characteristics extracted from the dust-free image with the temperature, smoke concentration and other time series parameters obtained by the sensor, and using the Transformer model to model the evolution of the environmental state. It can identify the risks of fire and cabin leakage at an early stage, realize the deep linkage and mutual verification of image visual information and environmental perception information, and significantly reduce the delay and missed reporting rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 2 is a diagram of a cabin monitoring system based on a visual neural network in the present invention.

[0050] Figure 2 It is a flow chart of the training steps of the residual pyramid compensation network. DETAILED DESCRIPTION

[0051] See also Figure 1-2 As shown, the present invention relates to a cabin monitoring system based on a visual neural network, comprising:

[0052] Data acquisition module, used to collect original image frames, dust particle size, dust concentration, temperature and smoke concentration data inside the shield machine cabin;

[0053] The image processing module is used to construct a light scattering prior based on dust particle size, dust concentration, and the relative position of the camera and light. It combines the monocular depth estimation network to perform transmittance inversion on the original image and descatters the original image through the residual pyramid compensation network to obtain a dust-free image sequence.

[0054] A three-dimensional synchronization module is used to extract the operator's skeletal key points, tool coordinates, and hatch status based on the dust removal image sequence, and to construct a cross-frame spatiotemporal graph structure using the skeletal key points and tool coordinates as nodes. The structure is input into a three-dimensional spatiotemporal graph convolutional network and outputs a continuous temporal behavior vector.

[0055] A human anomaly recognition module, based on a causal attention mechanism network, uses a time-series behavior vector as input and identifies and outputs human anomaly states, such as abnormal personnel stays and left-behind tools.

[0056] An environmental recognition module is used to construct environmental feature parameters from the dust-removed image sequence and the synchronously collected temperature and smoke concentration, and input them into a Transformer model to extract and generate time series features. The module then identifies abnormal environmental conditions based on the time series features, including fire and cabin leakage.

[0057] The alarm module is used to issue cabin alarms based on abnormal human and environmental conditions.

[0058] In some embodiments, a data acquisition module is deployed at key locations within the shield machine's cabin, including environmentally adaptable industrial cameras, laser particle size analyzers, dust concentration sensors, temperature sensors, and smoke concentration sensors. This module captures real-time image frames, dust particle size distribution, dust mass concentration, ambient temperature, and smoke concentration data during operations within the cabin. The image processing module utilizes a unique light scattering prior modeling mechanism. This innovatively constructs a light scattering prior model corresponding to the actual characteristics of the shield machine environment based on the current dust particle size and concentration parameters and the relative spatial position between the camera and lighting equipment. This differs from the fixed scattering coefficient assumption or empirical data estimation model used in traditional de-scattering methods, thereby more accurately describing the actual image degradation characteristics under high dust concentration conditions within the cabin. A monocular depth estimation network is then used, using weighted image frames guided by the scattering prior model as input, to accurately predict the depth information of each pixel position and invert the spatial transmittance distribution map pixel by pixel, thereby obtaining a transmittance feature with a clearer spatial structure and higher detail fidelity. This combined strategy of physical scattering modeling and deep learning significantly improves transmittance estimation accuracy and effectively addresses the error amplification problem caused by the lack of environmental constraints in traditional blind estimation methods. Furthermore, the image processing module uses a residual pyramid compensation network to jointly compensate for the transmittance map and the original degraded image. The residual pyramid network structure comprises residual convolution channels at multiple scales, with a cross-layer skip connection mechanism between channels. This effectively fuses low-level texture details with high-level semantic structure information, ultimately outputting a dust-free image sequence with high-quality detail fidelity and structural integrity. This is particularly suitable for the heterogeneous environments of shield machines, which are subject to strong reflective lighting and complex equipment edge structures. Subsequently, the 3D synchronization module, based on this high-quality dust-free image sequence, uses a skeletal pose estimation network to extract 17 key skeletal points of the cabin worker's body frame by frame. Instance segmentation and object detection algorithms are then used to accurately obtain the 3D coordinates of the cabin's working tools and the real-time opening and closing status of the cabin door. Using the extracted skeletal key points and tool coordinates as graph nodes, a cross-frame spatiotemporal graph structure is established based on their spatial proximity and cross-frame temporal continuity. This structure is then fed into a 3D spatiotemporal graph convolutional network for deep feature extraction, generating a continuous temporal behavior vector that reflects the cabin personnel's movements, tool movement, and changes in cabin door status. This overcomes the limitation of traditional single-frame recognition techniques that cannot effectively model complex action sequences. The human anomaly recognition module, based on these temporal behavior vectors, introduces a causal attention mechanism network. This network utilizes a self-attention mechanism combining position encoding and causal masks to emphasize the dependency of the current frame on historical states, addressing the misidentification problem often associated with traditional temporal recognition methods due to the lack of temporal dependency analysis. This module accurately identifies human anomalies, including abnormal personnel stops and dropped tools, significantly improving the accuracy and real-time performance of abnormal behavior recognition.Furthermore, the environmental recognition module of this embodiment further constructs a multimodal environmental feature vector sequence that fuses image visual information with sensor data. Specifically, it extracts image smoke distribution features and local brightness gradient features from the dust-removed image sequence, combines these features with real-time, synchronously collected ambient temperature and smoke concentration data, and uses them as input to a Transformer network. This Transformer network, with its built-in multi-head self-attention and feedforward neural network architecture, effectively mines and models the temporal evolution of the cabin's environmental state. It promptly and accurately identifies environmental anomalies, such as fires and leaks, and outputs confidence scores for these events, significantly reducing the risk of false alarms or missed alarms associated with traditional single-channel sensor alarm strategies. Finally, the alarm module, based on the anomaly information and confidence scores output by the human anomaly recognition and environmental recognition modules, develops a hierarchical response linkage strategy. This strategy simultaneously triggers the cabin's voice announcement system, highlights alerts on the visual monitoring interface, and pushes alarm data to a remote security monitoring platform in real time, completing a complete closed-loop monitoring system from front-end data collection, processing, and analysis to risk response and disposal. Compared with the existing technical solutions that only use a single-channel or single-modal alarm method, the multi-channel, multi-modal linkage strategy of this embodiment has obvious technical advantages and engineering application value in the safety monitoring and accident warning scenarios under the complex operating environment of the shield machine.

[0059] Furthermore, the light scattering prior is constructed based on the dust particle size, dust concentration, and the relative position of the camera and the light, including the following steps:

[0060] Obtain the particle size distribution and concentration parameters corresponding to the current image frame, and construct the scattering influence function of dust in the image by combining the relative angle and distance between the camera and the light source;

[0061] Based on the scattering influence function, the brightness attenuation and contrast loss of different pixel areas are spatially modeled to generate a light scattering prior map for constraining image transmittance inversion. The light scattering prior map includes a brightness attenuation map and a contrast loss distribution map.

[0062] In this embodiment, a targeted light scattering prior model construction method was designed to accurately characterize the image degradation caused by dust scattering, a phenomenon unique to the shield machine's cabin environment. First, a laser particle size analyzer and dust concentration sensor installed in the shield machine's cabin acquire dust particle size distribution parameters and mass concentration data corresponding to each image frame in real time. Simultaneously, based on the calibrated spatial layout between the camera and the light source, the relative angle between the camera's viewing axis and the light projection direction, as well as the spatial distance between them, are clearly determined. Next, using this real-time particle size, concentration, and spatial layout information as input, a dust particle scattering influence function adapted to the actual cabin environment is constructed. This function quantitatively describes the effect of dust particles of varying sizes and concentrations on the brightness attenuation of each image region under specific illumination angles. Based on this scattering influence function, this embodiment further performs pixel-by-pixel spatial modeling and precise estimation of the brightness attenuation and local contrast loss across different pixel regions within the image frame to generate a light scattering prior map used as a constraint for subsequent transmittance inversion. The specific method is as follows: the image scene is divided into multiple sub-regions based on the camera's field of view, the brightness attenuation of pixels within each sub-region is calculated using a scattering influence function, and an accurate brightness attenuation map is generated pixel by pixel. At the same time, based on the brightness differences of local neighborhood pixels, the degree of local contrast reduction corresponding to each sub-region is evaluated and calculated, generating a contrast loss distribution map that can accurately depict spatial differences. This spatial subdivision and refined modeling approach differs from the spatial inconsistency defects caused by the use of a single global scattering estimate in existing technologies. This allows this embodiment to more accurately constrain the subsequent transmittance inversion network, significantly improving the accuracy and robustness of image enhancement processing under the complex optical conditions inside the shield machine cabin.

[0063] Furthermore, the scattering influence function is as follows:

[0064] S(x)=exp(–k(d,θ)×C×L);

[0065] Where S(x) is the scattering attenuation of image brightness caused by dust at pixel position x; d is the average dust particle size; C is the dust mass concentration; θ is the scattering angle between the camera's visual axis and the light direction; L is the estimated optical path length from the camera to the target scene; and k(d,θ) is the scattering attenuation coefficient from the dust particle size and scattering angle lookup table.

[0066] It should be noted that, in the formula, S(x) is the degree of scattering attenuation of image brightness caused by dust at pixel position x, and its value range is between 0 and 1. The closer the value is to 0, the more significant the image brightness attenuation caused by scattering; the closer the value is to 1, the smaller the effect of scattering on the pixel brightness at that position. Among them, d is the average dust particle size measured in real time by a laser particle size analyzer, which is used to accurately characterize the physical properties of dust particles of different sizes under specific scattering conditions; C is the dust mass concentration at the current moment, which is used to reflect the contribution of the amount of dust in the cabin environment to the intensity of the scattering effect; θ is the scattering angle formed between the camera's visual axis direction and the projection direction of the lighting, which is used to express the impact of the directional difference of light on the scattering effect under different shooting angles; L is the estimated optical path length from the camera position to the target shooting scene, which is used to reflect the cumulative effect of the propagation path of the light beam through the dusty environment on the overall brightness attenuation; k(d,θ) is the scattering attenuation coefficient corresponding to the relationship between dust particle size and scattering angle, which is calculated in advance according to Mie scattering theory and corrected by actual measurement. It reflects the scattering ability of dust particles of a specific particle size to light under given scattering angle conditions. It can be obtained by lookup table to determine the parameter value efficiently and in real time during actual operation.

[0067] Furthermore, the spatial modeling of brightness attenuation and contrast loss in different pixel areas based on the scattering influence function includes the following steps:

[0068] According to the camera angle of view, the target scene range in the current image frame is determined, and the target scene range is divided into several sub-areas;

[0069] Calculating the brightness attenuation value of pixels in each sub-area according to the scattering influence function, and generating a corresponding brightness attenuation map pixel by pixel;

[0070] According to the brightness attenuation map, the degree of local contrast reduction of each sub-region is calculated, and a spatial contrast loss distribution map is generated.

[0071] Specifically, before image modeling, the spatial imaging range corresponding to the current image frame is first determined using the field of view projection function based on the camera's current viewing angle parameters and its intrinsic parameter matrix. Considering that the internal structure of the shield machine's cabin has a variety of different surface materials (such as metal, oil stains, tool edges, etc.) and non-homogeneous lighting conditions, this embodiment uses a spatially uniform partitioning strategy in the image coordinate system to divide the entire target scene into several static subregions. Each subregion is typically a fixed-size window (e.g., 32×32 pixels). This facilitates independent modeling and differentiated processing of the dust's impact on the image at the regional level. Subsequently, within each subregion, the brightness attenuation is calculated pixel by pixel for each pixel within the region based on the previously calculated scattering influence function. This operation constructs a brightness attenuation map, consistent with the original image size, that reflects the attenuation distribution trend of the entire image under the current physical conditions of dust, lighting, and viewing angle. Low-value areas in the brightness attenuation map often correspond to backlit areas, long-path dust accumulation areas, or structural surfaces with large light scattering angles, and are key target areas for subsequent image enhancement. On the basis of obtaining the brightness attenuation map, in order to further quantify the degree of degradation of the image texture level, this embodiment constructs a contrast loss estimation model based on local brightness variance and grayscale gradient entropy. Specifically, a sliding window convolution operation is used to extract the local statistical features of the brightness attenuation map in each sub-region, such as the mean brightness attenuation amplitude and variance distribution, and by comparing the entropy difference of the grayscale gradient in the original image and the brightness attenuation map, the degradation intensity index of the contrast of each sub-region is obtained. This index is then mapped to a regularized contrast loss value and combined to generate a spatial contrast loss distribution map. This map can reveal which image areas have lost key texture information (such as edges, broken lines, circular holes and other structures) under the current dust conditions, and provide regional attention guidance for the subsequent transmittance inversion module during spatial weighting. Compared to existing methods that use mean fuzzy estimation, fixed contrast templates, or analysis based solely on the overall image grayscale range, this embodiment combines actual dust distribution, illumination structure, and pixel spatial statistical characteristics to introduce real-world-driven pixel-level brightness attenuation estimation and regional contrast loss modeling at the algorithmic level. This approach offers greater physical consistency and environmental adaptability, making it particularly suitable for image perception modeling tasks in extreme working conditions such as non-uniform lighting and drastic dust dynamics in shield machine cabins. The light scattering prior map constructed using this modeling approach can significantly improve the structural constraints and detail recovery accuracy of subsequent image enhancement processing.

[0072] Furthermore, the transmittance inversion of the original image in combination with the monocular depth estimation network includes the following steps:

[0073] Calculating a weight coefficient for each pixel based on the brightness attenuation map and the spatial contrast loss distribution map, and applying the weight coefficient to the original image frame pixel by pixel to obtain a weighted input image frame;

[0074] According to the weighted input image frame, a monocular depth estimation network is used to predict the depth value pixel by pixel to obtain a depth map corresponding to each pixel of the original image frame;

[0075] Based on the pixel values of the depth map and the light scattering prior map, the transmittance value is calculated pixel by pixel to obtain a transmittance map.

[0076] It should be noted that, first of all, before the transmittance inversion, the system has obtained the brightness attenuation map and the contrast loss distribution map constructed in the previous stage, which are used to characterize the spatial distribution of the degree of pixel brightness degradation and the clarity of the texture structure, respectively. In order to enhance the depth estimation network's perception of key image areas, this embodiment constructs a pixel-level weighted coefficient map based on these two maps, where the weight function is designed as a nonlinear combination, so that higher attention weights are assigned to areas with severe brightness attenuation or significant contrast loss. The weight map is then multiplied pixel by pixel with the original image to generate a weighted input image frame, which is used to guide the subsequent deep network to pay more attention to structural degradation areas, thereby improving the robustness of depth estimation in areas with uneven image quality. Next, a well-trained monocular depth estimation network is used to perform pixel-by-pixel depth prediction on the weighted image frame. The network adopts a multi-scale convolution structure, combined with residual connections and up and down sampling mechanisms, to gradually fuse local texture and global layout information in the image, and finally output a depth map with the same size as the original image. This depth map provides a spatial depth estimate for each pixel relative to the camera, has good geometric consistency, and maintains stable structure edge detection capabilities in dust interference scenes. It is the core foundation for transmittance inversion. Finally, based on the obtained depth image and the light scattering prior map, the transmittance calculation is performed pixel by pixel. The transmittance at pixel position x is as follows:

[0077] t(x)=exp(-α·Z(x)·A(x)·(1+λ·D(c)));

[0078] Where Z(x) is the depth value of pixel x in the depth map; A(x) is the normalized attenuation coefficient of pixel x in the brightness attenuation map (range: 0-1); D(x) is the normalized contrast loss value of pixel x in the contrast loss map (range: 0-1); α is the calibration value of the total scattering coefficient, reflecting the degree to which the dust density in the overall environment controls transmittance; and λ is the contrast adjustment coefficient, which controls the contribution of D(x) to the overall transmittance. Specifically, the depth value of each pixel is weighted by its corresponding brightness attenuation value in the brightness attenuation map. The brightness attenuation map records the degree of energy attenuation caused by forward scattering, reflecting the energy loss characteristics of light during propagation. The system uses the brightness attenuation map as an energy compensation factor for transmittance calculations. For two pixels with the same depth value, the pixel with the larger brightness attenuation value has a greater actual light intensity loss, resulting in lower transmittance. To this end, the system has designed a mapping method that proportionally adjusts the brightness attenuation value's contribution to the depth value, achieving a more physically accurate calculation of local transmittance performance. On this basis, in order to further improve the sensitivity of transmittance estimation to image details, the system introduces a contrast loss distribution map. This map shows the degree of texture clarity loss in each area due to multi-angle scattering during the imaging process. Specifically, the system uses contrast loss as an adjustment item to perform a secondary correction on the result after brightness attenuation adjustment. If a pixel has a medium value in the brightness attenuation map, but its contrast loss value is high, it means that although the pixel has not produced a strong overall brightness attenuation, the local structural details have been severely degraded, and the transmittance should be further reduced. Therefore, the contrast loss distribution map plays a role in adjusting the sensitivity of regional texture degradation in this process, thereby avoiding over-estimation caused by focusing only on brightness and ignoring texture changes.

[0079] Furthermore, the residual pyramid compensation network is trained by the following steps:

[0080] The original image frame and the corresponding transmittance map are combined to form the training input, and the corresponding clear image sequence in the same scene is used as the supervision label to construct the training sample set;

[0081] Inputting the training sample set into a residual pyramid compensation network, wherein the network comprises multiple scale residual channels and feature reconstruction branches for extracting image structure information and illumination degradation residual features at different spatial scales;

[0082] Residual mapping and image fusion operations are performed in each scale channel to generate a multi-scale residual compensation map, and image reconstruction is performed under the skip connection mechanism to output a dust-free image sequence for comparative supervision;

[0083] Constructing a joint loss function that includes pixel reconstruction error, structural similarity loss, and edge preservation loss, and performing error feedback on the dust-free image sequence and label image output by the network;

[0084] Gradient descent and parameter update are performed based on the joint loss function.

[0085] In this embodiment, a pyramid compensation network based on a multi-scale residual mechanism is constructed, and network parameter training is performed using supervised learning to obtain model weights with strong image restoration capabilities. During training, the system first uses the actual raw image frames and the transmittance map calculated by the image processing module as joint inputs, which are then spliced into multi-channel training samples to enhance the model's perception of the physical mechanisms of image degradation. The corresponding supervised label images are derived from a sequence of high-quality, dust-free images acquired under clean conditions from the same shooting scene, serving as ideal visual targets. The residual pyramid compensation network is internally designed as a multi-scale pyramid structure. Its backbone network consists of several scale residual channels, each of which includes a shallow convolution module, a feature compression module, and a residual mapping module. This structure extracts local texture features and global structural edges in images at different spatial resolutions, while also generating residual enhancement responses to nonlinear brightness degradation and texture blur caused by dust scattering. During the feature fusion stage, the system introduces a cross-scale skip connection mechanism to map semantic features extracted at low resolution to high-resolution channels, where they are weighted and combined with shallow local features to generate a multi-scale residual compensation map. Ultimately, the decoding and reconstruction module restores a dust-free image sequence with the same resolution as the original image. To improve the convergence efficiency of the training process and the structural fidelity of the image restoration, the system designed a three-component joint loss function: pixel reconstruction error, structural similarity loss (SSIM), and edge preservation loss. The pixel reconstruction error minimizes the pixel-by-pixel difference between the dust-free image and the labeled image; the structural similarity loss constrains the consistency of the model output in the overall structural distribution, especially for key structures such as hatch edges and tool boundaries in the image; and the edge preservation loss extracts image edge contour features using a high-pass filter or Sobel operator, ensuring high edge sharpness and physical realism after image enhancement. The network is trained using a standard backpropagation algorithm, with gradient calculation and parameter updates based on the joint loss function. The learning rate is dynamically adjusted using the Adam optimizer. Through multiple rounds of iterations, the network continuously converges to the optimal parameter state that can adapt to image degradation characteristics under various dust concentrations, particle size distributions, and illumination interference conditions, achieving adaptive restoration of image structure and texture under complex working conditions. The training process in this embodiment is different from existing methods based only on image contrast enhancement or traditional denoising networks. By introducing transmission priors and multi-scale residual modeling under the control of real environmental parameters, it achieves refined enhancement of dust interference areas, and shows stronger adaptability and robustness in image reconstruction quality, edge structure restoration and local information preservation. It is particularly suitable for image enhancement tasks in extremely low-visibility working environments such as shield machine cabins.

[0086] Furthermore, the three-dimensional synchronization module is used to perform the following steps:

[0087] Based on the dust removal image sequence, the skeleton key points of the operator are extracted frame by frame through the skeleton posture estimation network, and the three-dimensional coordinates of the tool and the hatch opening and closing status are obtained through instance segmentation and target detection algorithms;

[0088] The skeleton key points and tool coordinates are paired within a frame and tracked between frames in a time series to construct a node set, and a graph structure is established based on the spatial adjacency relationship and temporal correspondence between the nodes to form a cross-frame spatiotemporal graph containing spatial edges and temporal edges;

[0089] The spatiotemporal graph is input into a three-dimensional spatiotemporal graph convolutional network, and the behavior change characteristics and spatial structure information across time steps are extracted through graph convolution operations, and a continuous time series behavior vector is output.

[0090] In this embodiment, the 3D synchronization module is used to achieve unified spatiotemporal modeling of multi-source object states based on the dust-removed image sequence, supporting the downstream behavior recognition module in accurately analyzing human movements and changes in the work environment. Specifically, the system first invokes a skeletal pose estimation network to extract human keypoints for each frame in the sequence. This network utilizes a pretrained human pose detection model to extract the human skeleton structure, including 17 key components, through convolutional features and heatmap regression. It then represents the spatial pose of the worker in the current frame as 3D coordinates. Simultaneously, the system concurrently invokes an instance segmentation algorithm and a lightweight object detection network to extract the contour mask and bounding box coordinates of the tool in the image. This is then combined with the transmittance map and depth estimation results to reconstruct the 3D spatial coordinates. The open / closed state of the hatch is classified based on object contour stability, angle estimation, and edge symmetry, outputting its current displacement and closed state indicator. To ensure target consistency across time, the system matches the extracted skeletal keypoints of the person with the 3D coordinates of the tool in each frame based on intra-frame spatial structure and inter-frame state continuity. Using a trajectory re-identification algorithm and a temporal nested matching strategy, the system constructs a node set across the entire image sequence. A spatiotemporal graph is constructed based on the spatial adjacency between nodes (e.g., the relative position of a limb and a handheld tool) and temporal coherence (e.g., the motion trajectory of a node across multiple frames). In this graph, nodes represent key points or tool coordinates, and edges express dynamic connections within and between frames, resulting in a cross-frame graph containing both spatial and temporal edges. This spatiotemporal graph is then fed into a constructed 3D spatiotemporal graph convolutional network. Based on a graph neural network framework, this network integrates a spatial adjacency matrix with a temporal skip connection mechanism to perform convolution operations and feature aggregation based on motion patterns and structural topological changes between nodes. Within each convolutional layer, the system aggregates not only the human motion features within the current time step but also incorporates information from previous and subsequent frames to identify behavioral state transitions, such as nonlinear trajectory changes from grasping a tool to dropping it or prolonged periods of inactivity. Ultimately, this module outputs a continuous set of temporal behavior vectors, each representing the combined state of the skeleton, tool, and hatch in the current frame, which serve as input to the downstream behavior recognition module.

[0091] Furthermore, the causal attention mechanism network is constructed by the following steps:

[0092] Input the continuous time series behavior vector into the position encoding module, embed the time step using the sine and cosine functions, and generate a time-dependent behavior representation vector;

[0093] Based on the behavior representation vector, a query vector, a key vector, and a value vector are constructed, and a causal mask is used in the multi-head self-attention structure to limit the current time step to only focus on historical information, thereby generating a causal attention matrix;

[0094] Applying the causal attention matrix to the value vector to complete context weighting processing and output a sequence of abnormal behavior candidate vectors containing temporal dependency features;

[0095] The abnormal behavior candidate vector sequence is input into the fully connected classification layer, and feature alignment and probability matching are performed with the preset abnormal behavior template to identify the target abnormal state, which includes abnormal stay of personnel and left-behind tools.

[0096] Specifically, the causal attention network first receives a continuous sequence of behavior representations output by a 3D spatiotemporal graph convolutional network. This sequence, encoded by time step, contains a high-dimensional fusion vector representing the worker's skeletal posture, tool space state, and hatch opening / closing status in each frame. To preserve temporal position information, the system first inputs this vector sequence into a position encoding module. By superimposing a position embedding vector constructed using sine and cosine functions at each time step, this explicitly encodes the temporal position information in the behavior sequence, ensuring that the subsequent self-attention mechanism is able to perceive the order of precedence during modeling. Subsequently, the system constructs the query, key, and value vectors of the self-attention structure based on the embedded behavior vectors. A multi-head parallel mechanism is employed to enhance the ability to perceive different behavior channels and local patterns. During the attention computation process, to prevent recognition bias caused by future information leakage, the system introduces a causal masking mechanism, which masks the attention weights of all positions after the current time step. This ensures that each time step can only access its own historical behavior state and its own, thereby constructing an attention matrix that strictly adheres to temporal unidirectionality. The causal attention matrix then acts on the value vector to achieve weighted aggregation of contextual features and output a set of time-sensitive candidate abnormal behavior vector sequences. To achieve final state recognition, the system inputs the candidate vector sequence into the subsequent fully connected classification layer, and performs alignment and similarity matching with the preset abnormal behavior template in the vector feature space. The specific matching logic combines Euclidean distance, cosine similarity and confidence gating strategy to compare the behavior fragments corresponding to each time step with the label template one by one, and finally outputs the corresponding abnormal behavior category and judgment probability. The abnormal category includes but is not limited to abnormal stay of personnel (for example, a skeleton that remains stationary for multiple consecutive frames) and tool loss (such as interruption of tool trajectory, departure from critical areas, etc.). The system uses the probability threshold to determine whether to enter the abnormal alarm state.

[0097] Furthermore, the environment recognition module is used to perform the following steps:

[0098] Extracting smoke distribution characteristics and brightness gradients within the image region based on the dust removal image sequence, and constructing a multimodal environment feature vector sequence in combination with temperature and smoke concentration data;

[0099] Performing time alignment processing on the environmental feature vector sequence and inputting it into a Transformer model including a multi-head self-attention mechanism and a feedforward network structure for cross-modal time series modeling;

[0100] In the encoder of the Transformer model, the self-attention layer is used to obtain the dependency of environmental state changes between different time steps and generate a global feature expression with time weights;

[0101] The global feature expression is input into the multi-layer perceptron classification, and combined with the preset fire and cabin leakage event labels for feature matching and probability prediction, the abnormal state of the environment and its confidence level are output.

[0102] In some embodiments, the system first uses the dust-free image sequence output by the image processing module as input. Visual layer analysis is performed using the enhanced high-quality images. A feature extraction network extracts representative environmental risk indicators, such as smoke distribution patterns, abnormal brightness gradients, and localized saturation patches, from each frame. This generates a spatially distributed feature vector representing the degree of optical interference. Simultaneously, the system synchronously accesses acquired time-series temperature and smoke concentration data, aligns them with the image frame timestamps, and fuses them into a complete multimodal environmental feature vector sequence along the time axis. To fully capture the temporal variations of environmental variables and their potential cross-correlations, the system inputs this multimodal vector sequence into a constructed Transformer temporal modeling framework. This Transformer architecture incorporates a multi-head self-attention mechanism and a positional encoding structure, enabling it to simultaneously learn the coupling relationships between channels and over time while maintaining sequence order. In the encoder, the system uses a self-attention layer to calculate the state correlation matrix between each time step, adaptively focusing on the correspondence between recent temperature fluctuations and image brightness changes. This creates a temporally weighted global feature representation vector that captures the combined signs of localized temperature rise and image smoke drift during the initial stages of a fire. Subsequently, the system inputs the global feature representation into the multi-layer perceptron classifier, and by aligning and matching the feature space with the preset fire and cabin leakage event label samples, combining the classification results with the confidence gating mechanism, it makes a judgment on whether the current time point is in an abnormal state. The prediction output not only includes the abnormal category identification, but also the corresponding probability value, which is used to drive the alarm module to execute response strategies such as cabin voice prompts and monitoring interface highlights. The environmental recognition module constructed in this embodiment breaks through the limitations of single-channel rule methods such as traditional temperature threshold judgment or image histogram detection. Through the advantages of multimodal joint modeling and Transformer structure in long-term dependency modeling, it realizes robust recognition of complex evolving environmental signals. It is particularly suitable for abnormal situation monitoring under working conditions such as high dust, high temperature, and low visibility in shield operation areas, and significantly improves the warning sensitivity and response reliability of the environmental perception system.

[0103] Furthermore, the cabin alarm includes triggering a voice broadcast in the cabin and a highlighted border prompt on the monitoring screen.

[0104] It should be noted that the in-cabin alarm module is designed to implement a multimodal feedback mechanism based on risk level and confidence level when identifying abnormal human behavior or environmental conditions, providing immediate warnings to operators and remote visual notifications to the command center. Specifically, once the abnormal status category output by the human anomaly recognition module or environmental recognition module is confirmed by the system, the alarm module first determines whether the trigger conditions are met based on pre-set alarm rules. If the trigger threshold is reached, the in-cabin voice announcement system immediately generates a scenario-specific warning statement, such as "Tools dropped, please be careful" or "Signs of fire detected, please evacuate immediately," through the in-cabin voice announcement system. This voice announcement is generated in real time using semantic templates and a text-to-speech (TTS) speech synthesis module and broadcast through the cabin's internal speakers, ensuring that operators receive clear and unambiguous warning instructions immediately, even in high-noise and low-visibility environments. The system also simultaneously transmits the alarm status to the on-site monitoring terminal, highlighting the corresponding camera view on the monitoring screen, using a red pulse stroke or flashing animation for enhanced visual perception, allowing on-duty personnel to quickly locate the alarm source. In addition, for abnormal conditions that persist and have not been recovered, the system will periodically refresh the alarm prompts and can link with the construction control system to generate digital alarm logs, realizing the full process recording and traceability analysis of abnormal events in the cabin.

[0105] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A cabin monitoring system based on visual neural network, characterized in that: include: Data acquisition module, used to collect original image frames, dust particle size, dust concentration, temperature and smoke concentration data inside the shield machine cabin; The image processing module is used to construct a light scattering prior based on dust particle size, dust concentration, and the relative position of the camera and light. It combines the monocular depth estimation network to perform transmittance inversion on the original image and descatters the original image through the residual pyramid compensation network to obtain a dust-free image sequence. A three-dimensional synchronization module is used to extract the operator's skeletal key points, tool coordinates, and hatch status based on the dust removal image sequence, and to construct a cross-frame spatiotemporal graph structure using the skeletal key points and tool coordinates as nodes. The structure is input into a three-dimensional spatiotemporal graph convolutional network and outputs a continuous temporal behavior vector. A human anomaly recognition module, based on a causal attention mechanism network, uses a time-series behavior vector as input and identifies and outputs human anomaly states, such as abnormal personnel stays and left-behind tools. An environmental recognition module is used to construct environmental feature parameters from the dust-removed image sequence and the synchronously collected temperature and smoke concentration, and input them into a Transformer model to extract and generate time series features. The module then identifies abnormal environmental conditions based on the time series features, including fire and cabin leakage. The alarm module is used to issue cabin alarms based on abnormal human and environmental conditions.

2. The cabin monitoring system based on visual neural network according to claim 1 is characterized in that: The method of constructing a light scattering prior based on dust particle size, dust concentration, and the relative position of the camera and the light includes the following steps: Obtain the particle size distribution and concentration parameters corresponding to the current image frame, and construct the scattering influence function of dust in the image by combining the relative angle and distance between the camera and the light source; Based on the scattering influence function, the brightness attenuation and contrast loss of different pixel areas are spatially modeled to generate a light scattering prior map for constraining image transmittance inversion. The light scattering prior map includes a brightness attenuation map and a contrast loss distribution map.

3. The cabin monitoring system based on visual neural network according to claim 2 is characterized in that: The scattering influence function is as follows: S(x)=exp(–k(d,θ)×C×L); Where S(x) is the scattering attenuation of image brightness caused by dust at pixel position x; d is the average dust particle size; C is the dust mass concentration; θ is the scattering angle between the camera's visual axis and the light direction; L is the estimated optical path length from the camera to the target scene; and k(d,θ) is the scattering attenuation coefficient from the dust particle size and scattering angle lookup table.

4. The cabin monitoring system based on visual neural network according to claim 3 is characterized in that: The spatial modeling of brightness attenuation and contrast loss in different pixel areas based on the scattering influence function comprises the following steps: According to the camera angle of view, the target scene range in the current image frame is determined, and the target scene range is divided into several sub-areas; Calculating the brightness attenuation value of pixels in each sub-area according to the scattering influence function, and generating a corresponding brightness attenuation map pixel by pixel; According to the brightness attenuation map, the degree of local contrast reduction of each sub-region is calculated, and a spatial contrast loss distribution map is generated.

5. The cabin monitoring system based on visual neural network according to claim 4 is characterized in that: The method of performing transmittance inversion on the original image in combination with the monocular depth estimation network comprises the following steps: Calculating a weight coefficient for each pixel based on the brightness attenuation map and the spatial contrast loss distribution map, and applying the weight coefficient to the original image frame pixel by pixel to obtain a weighted input image frame; According to the weighted input image frame, a monocular depth estimation network is used to predict the depth value pixel by pixel to obtain a depth map corresponding to each pixel of the original image frame; Based on the pixel values of the depth map and the light scattering prior map, the transmittance value is calculated pixel by pixel to obtain a transmittance map.

6. The cabin monitoring system based on visual neural network according to claim 5, characterized in that: The residual pyramid compensation network is trained through the following steps: The original image frame and the corresponding transmittance map are combined to form the training input, and the corresponding clear image sequence in the same scene is used as the supervision label to construct the training sample set; Inputting the training sample set into a residual pyramid compensation network, wherein the network comprises multiple scale residual channels and feature reconstruction branches for extracting image structure information and illumination degradation residual features at different spatial scales; Residual mapping and image fusion operations are performed in each scale channel to generate a multi-scale residual compensation map, and image reconstruction is performed under the skip connection mechanism to output a dust-free image sequence for comparative supervision; Constructing a joint loss function that includes pixel reconstruction error, structural similarity loss, and edge preservation loss, and performing error feedback on the dust-free image sequence and label image output by the network; Gradient descent and parameter update are performed based on the joint loss function.

7. The cabin monitoring system based on visual neural network according to claim 1 is characterized in that: The three-dimensional synchronization module is used to perform the following steps: Based on the dust removal image sequence, the skeleton key points of the operator are extracted frame by frame through the skeleton posture estimation network, and the three-dimensional coordinates of the tool and the hatch opening and closing status are obtained through instance segmentation and target detection algorithms; The skeleton key points and tool coordinates are paired within a frame and tracked between frames in a time series to construct a node set, and a graph structure is established based on the spatial adjacency relationship and temporal correspondence between the nodes to form a cross-frame spatiotemporal graph containing spatial edges and temporal edges; The spatiotemporal graph is input into a three-dimensional spatiotemporal graph convolutional network, and the behavior change characteristics and spatial structure information across time steps are extracted through graph convolution operations, and a continuous time series behavior vector is output.

8. The cabin monitoring system based on visual neural network according to claim 1 is characterized in that: The causal attention mechanism network is constructed through the following steps: Input the continuous time series behavior vector into the position encoding module, embed the time step using the sine and cosine functions, and generate a time-dependent behavior representation vector; Based on the behavior representation vector, a query vector, a key vector, and a value vector are constructed, and a causal mask is used in the multi-head self-attention structure to limit the current time step to only focus on historical information, thereby generating a causal attention matrix; Applying the causal attention matrix to the value vector to complete context weighting processing and output a sequence of abnormal behavior candidate vectors containing temporal dependency features; The abnormal behavior candidate vector sequence is input into the fully connected classification layer, and feature alignment and probability matching are performed with the preset abnormal behavior template to identify the target abnormal state, which includes abnormal stay of personnel and left-behind tools.

9. The cabin monitoring system based on visual neural network according to claim 6, characterized in that: The environment recognition module is used to perform the following steps: Extracting smoke distribution characteristics and brightness gradients within the image region based on the dust removal image sequence, and constructing a multimodal environmental feature vector sequence in combination with temperature and smoke concentration data; Performing time alignment processing on the environmental feature vector sequence and inputting it into a Transformer model including a multi-head self-attention mechanism and a feedforward network structure for cross-modal time series modeling; In the encoder of the Transformer model, the self-attention layer is used to obtain the dependency of environmental state changes between different time steps and generate a global feature expression with time weights; The global feature expression is input into the multi-layer perceptron classification, and combined with the preset fire and cabin leakage event labels for feature matching and probability prediction, the abnormal state of the environment and its confidence level are output.

10. The cabin monitoring system based on visual neural network according to claim 9, characterized in that: The cabin alarm includes triggering a voice broadcast in the cabin and a highlighted border prompt on the monitoring screen.

Images