Method for recognizing actions of construction workers based on voxels and sparse convolutional neural networks

Abstract

This application relates to the field of image recognition technology and provides a method for recognizing construction worker actions based on voxels and sparse convolutional neural networks. The method includes: acquiring construction monitoring video and processing the video using a VIBE model to obtain a 3D human body mesh for each frame of the video; converting the 3D human body mesh into a voxel mesh and constructing a sparse coordinate matrix and a voxel feature matrix based on the converted voxel mesh; constructing a sparse tensor based on the sparse coordinate matrix and the voxel feature matrix; and inputting the sparse tensor into a construction worker action recognition model based on a sparse convolutional neural network for processing to obtain the action recognition result of the construction worker in the construction monitoring video. This application can improve the accuracy of construction worker action recognition.

Description

Technical Field

[0001] This application belongs to the field of image recognition technology, and in particular relates to a method for recognizing the actions of construction workers based on voxels and sparse convolutional neural networks. Background Technology

[0002] In civil engineering construction sites, worker motion recognition and behavior monitoring are crucial for ensuring construction safety management and are also primary means of analyzing worker activities and productivity. As project scale expands and construction procedures become increasingly complex, worker errors, fatigue, and dangerous actions can lead to safety risks and unsatisfactory construction progress. Therefore, timely and accurate worker posture identification is of paramount importance. However, existing worker behavior monitoring methods still have several shortcomings and cannot meet the higher requirements for real-time performance, robustness, and automation in real-world engineering scenarios.

[0003] Currently, monitoring construction worker movements still primarily relies on manual inspections and traditional video playback. Manual inspections are not only labor-intensive but also susceptible to subjective judgment. Furthermore, manual video playback is inherently time-consuming and can only be used for post-event analysis. In addition, construction sites are typically complex and variable environments with uneven lighting, frequent obstructions, and dust interference, making manual monitoring even more difficult. With the development of sensor technology, researchers have proposed using contact-based devices (such as GPS, IMU, and RFID) to collect information about construction workers during their work, thereby enabling analysis of their activities. However, this non-visual recognition method requires workers to wear sensor devices continuously, which is inconvenient and expensive. In contrast, with the development of computer vision technology, visual recognition, a non-contact method, has shown significant advantages. Two-dimensional pose recognition methods based on RGB video are increasingly being applied to construction site monitoring scenarios. These methods automatically identify worker movements by extracting and classifying key points or motion features from images. However, two-dimensional images are significantly affected by lighting, background clutter, and changes in viewing angle, making it difficult to accurately represent the three-dimensional spatial posture structure of workers. This can easily lead to unstable recognition results, especially with a significant decrease in accuracy under complex postures and movements. In recent years, three-dimensional human posture reconstruction technology has been continuously developing, enabling the estimation of the three-dimensional structure and posture of workers' bodies through monocular video. Most existing three-dimensional recognition methods use two-dimensional convolutional neural networks, which have limited processing capabilities for temporal data and cannot fully exploit the joint features of workers' movements in the spatial and temporal dimensions, resulting in insufficient accuracy in recognizing construction worker movements. Summary of the Invention

[0004] This application provides a method for recognizing construction worker actions based on voxels and sparse convolutional neural networks, which can solve the problem of insufficient accuracy in recognizing construction worker actions.

[0005] This application provides a method for recognizing construction worker actions based on voxels and sparse convolutional neural networks, including: Acquire construction monitoring videos and process them using the VIBE model to obtain a 3D human body mesh for each frame of the construction monitoring video. The 3D human body mesh is converted into a voxel mesh, and a sparse coordinate matrix and voxel feature matrix are constructed based on the converted voxel mesh. Sparse tensors are constructed based on sparse coordinate matrices and voxel feature matrices; The sparse tensor is input into the construction worker action recognition model based on sparse convolutional neural network for processing, and the action recognition results of construction workers in construction monitoring videos are obtained.

[0006] Optionally, the 3D human body mesh can be converted into a voxel mesh, including: The coordinates of each vertex in the 3D human body mesh are aligned and normalized to obtain the normalized coordinates of each vertex in the 3D human body mesh. Based on the pre-defined voxel grid side length, the normalized coordinates of each grid vertex in the 3D human body grid are converted into the center coordinates of the voxel grid. The center coordinates of all voxel meshes obtained by the transformation are deduplicated, and the spatial position of all voxel meshes is determined based on the center coordinates of all voxel meshes after deduplication.

[0007] Optionally, based on a pre-defined voxel mesh edge length, the normalized coordinates of each vertex in the 3D human body mesh are converted into the center coordinates of the voxel, including: The normalized coordinates of each vertex in the 3D human body mesh are converted to the center coordinates of the voxel mesh using the following formula. : ; ; ; in, Indicates the first The third frame of the 3D human body mesh Normalized coordinates of each grid vertex This indicates the pre-defined voxel mesh side length. This indicates the floor function.

[0008] Optionally, each vertex in the 3D human body mesh has a body part label. The construction worker action recognition method also includes: For each voxel grid, the part labels corresponding to all grid vertices contained in the voxel grid are counted, and the part label with the highest number of occurrences is used as the semantic label of the voxel grid.

[0009] Optional, sparse coordinate matrix for: ; in, This represents the first voxel in the grid corresponding to all voxels of the construction monitoring video. The coordinates of the individual pixel grid, This indicates the total number of voxel grids corresponding to the construction monitoring video.

[0010] Optionally, the voxel feature matrix includes the first voxel feature matrix. Second voxel feature matrix : , ; , ; in, Indicates the first Semantic labels for individual elements.

[0011] Optionally, the sparse tensor includes a first sparse tensor and a second sparse tensor; based on the sparse coordinate matrix and the voxel feature matrix, the sparse tensor is constructed, including: Based on sparse coordinate matrix and first voxel feature matrix Construct the first sparse tensor; Based on sparse coordinate matrix and second voxel feature matrix Construct a second sparse tensor; the dimension of the second sparse tensor is larger than that of the first sparse tensor.

[0012] Optionally, the construction worker action recognition model includes: a first sparse convolutional layer, a second sparse convolutional layer, a sparse attention module, and a pyramid feature fusion module, a residual structure backbone network, a global pooling layer, a fully connected layer, and a classification layer connected in sequence. The input data of the first sparse convolutional layer is the first sparse tensor, the input data of the second sparse convolutional layer is the second sparse tensor, the output of the second sparse convolutional layer is connected to the input of the sparse attention module, and the output of the sparse attention module and the output of the first sparse convolutional layer are both connected to the input of the pyramid feature fusion module.

[0013] Optionally, the residual structure backbone network includes a third, fourth, fifth, and sixth sparse convolutional layer connected in sequence. The output of the pyramid feature fusion module is directly connected to the input of the third sparse convolutional layer. After a projection shortcut connection, the output of the pyramid feature fusion module is fused with the output of the fourth sparse convolutional layer to form a first fused output. The first fused output is directly connected to the input of the fifth sparse convolutional layer. After a projection shortcut connection, the first fused output is fused with the output of the sixth sparse convolutional layer to form a second fused output. The second fused output is connected to the input of the global pooling layer.

[0014] The above-mentioned solution in this application has the following beneficial effects: In the embodiments of this application, the VIBE model is used to process construction monitoring videos to obtain a 3D human body mesh for each frame of the video. This 3D human body mesh is then converted into a voxel mesh. A sparse coordinate matrix and a voxel feature matrix are constructed based on this voxel mesh, and a sparse tensor is constructed based on these matrices. Finally, the sparse tensor is used as input data for the construction worker action recognition model to achieve action recognition. Because a sparse tensor is used as the model input, spatiotemporal features can be convolved within a unified sparse tensor structure, avoiding information loss caused by spatial and temporal feature segmentation in traditional methods, thereby effectively improving the accuracy of construction worker action recognition.

[0015] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a construction worker action recognition method based on voxels and sparse convolutional neural networks provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a construction worker motion recognition model provided in an embodiment of this application. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0019] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0020] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0021] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0022] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0024] To address the issue of insufficient accuracy in current construction worker action recognition methods, this application provides a method based on voxels and sparse convolutional neural networks. This method processes construction monitoring videos using a VIBE model to obtain a 3D human body mesh for each frame, converts the 3D human body mesh into a voxel mesh, and then constructs a sparse coordinate matrix and a voxel feature matrix based on this voxel mesh. Finally, a sparse tensor is constructed based on this sparse coordinate matrix and voxel feature matrix, and the sparse tensor is used as the input data for the construction worker action recognition model to achieve action recognition. Because a sparse tensor is used as the model input, spatiotemporal features can be convolved within a unified sparse tensor structure, avoiding information loss caused by spatial and temporal feature segmentation modeling in traditional methods, thereby effectively improving the accuracy of construction worker action recognition.

[0025] The following describes the construction worker action recognition method based on voxels and sparse convolutional neural networks provided in this application by way of specific embodiments.

[0026] like Figure 1 As shown in the embodiments of this application, the construction worker action recognition method based on voxels and sparse convolutional neural networks includes the following steps: Step 11: Acquire construction monitoring video and process the construction monitoring video using the VIBE model to obtain a three-dimensional human body mesh for each frame of the construction monitoring video.

[0027] The aforementioned surveillance videos can be captured by video acquisition equipment at the construction site (such as fixed cameras or mobile monitoring terminals). To ensure real-time monitoring, the video acquisition equipment will send the construction surveillance video for each monitoring cycle to the terminal device used to execute the construction worker action recognition method provided in this application at the end of each monitoring cycle. This allows the terminal device to analyze the construction surveillance video for that cycle in a timely manner and output the action recognition results of the construction workers, thereby achieving real-time recognition and monitoring of the construction workers' postures. This enables timely alerts to workers exhibiting dangerous actions, preventing safety accidents.

[0028] Understandably, to facilitate data processing, the construction monitoring video needs to be preprocessed after it is obtained before proceeding with subsequent steps. This preprocessing mainly includes: removing irrelevant, blurry, and low-quality footage unusable for attitude reconstruction; and operations such as frame rate unification, resolution unification, color standardization, lens stability correction, and video cropping.

[0029] In related technologies, the aforementioned VIBE model is constructed based on the VIBE method and has been optimized (i.e., trained, the training process of which will be explained later). Therefore, by inputting the construction monitoring video into the VIBE model for processing, a three-dimensional human body mesh for each frame of the construction monitoring video can be obtained.

[0030] Step 12: Convert the 3D human body mesh into a voxel mesh, and construct a sparse coordinate matrix and a voxel feature matrix based on the converted voxel mesh.

[0031] In some embodiments of this application, it is necessary to convert each three-dimensional human body mesh into a voxel mesh, and then construct a sparse coordinate matrix and a voxel feature matrix based on all voxel meshes.

[0032] In some embodiments of this application, the 3D human body mesh is converted into a voxel mesh to transform the 3D human body mesh into a 3D voxel representation suitable for sparse convolutional neural network processing. By discretizing the continuous 3D vertex coordinates into voxel units on a regular mesh, sparse voxel encoding of the construction worker's pose is achieved, providing a foundation for the subsequent construction of a temporal sparse voxel flow.

[0033] Specifically, the 3D human body mesh can be converted into a voxel mesh through the following steps 12.1 to 12.3: Step 12.1: Align and normalize the coordinates of each vertex in the 3D human body mesh to obtain the normalized coordinates of each vertex in the 3D human body mesh.

[0034] In some embodiments of this application, the coordinates of each grid vertex in the 3D human body mesh are aligned and normalized in order to eliminate the influence of differences in the height of different construction workers, camera positions and shooting angles on subsequent voxelization and sparse convolution modeling, so that the 3D pose distribution is unified in a normalized coordinate space.

[0035] For ease of description, let the first... The set of vertices of a 3D human body mesh in a frame image for: ; ; In the above formula, This represents the number of vertices in the 3D human body mesh. Indicates the first The third frame of the 3D human body mesh Each grid vertex For the first The coordinates of each grid vertex Indicates the first The grid vertices in The coordinate values ​​of the axis. Indicates the first Each grid vertex is in The coordinate values ​​of the axis. Indicates the first Each grid vertex is in The coordinate values ​​of the axis. axis, shaft and The axes are coordinate axes in the world coordinate system. It should be noted that the coordinate representation in this application... Related (as mentioned above) ) all indicate in The coordinate values ​​of the axis. Related (as mentioned above) ) all indicate in The coordinate values ​​of the axis. Related (as mentioned above) ) all indicate in The coordinate values ​​of the axis.

[0036] Record No. The coordinates of the pelvic joint points corresponding to the frame image are First, the entire 3D human body mesh is translated and aligned using the pelvis as a reference point, moving the pelvis to the origin of the coordinate system to achieve pose centering. ; Obtain the vertex set after translation and alignment Based on this, to eliminate scale inconsistencies caused by differences in individual body size and shooting distance, the aligned mesh was scaled. The maximum Euclidean distance from all vertices to the origin was selected as the scale factor. ,Right now ,in The L2 norm is used. Then, all vertices are scaled according to this scale factor to obtain the normalized coordinates of the mesh vertices. : ; This yields the normalized 3D human body mesh set. ,in .

[0037] Through the above translation alignment and scale normalization processes, it can be ensured that the 3D human body meshes of different video clips, different workers, and different shooting conditions are all unified into a standardized coordinate system with the pelvis as the origin and a unit scale. This makes the 3D voxel models generated in the subsequent voxelization process comparable in spatial position and scale, which is beneficial for sparse convolutional neural networks to learn the essential features of construction worker postures in a unified 3D space, thereby improving the stability and generalization ability of the action recognition model.

[0038] Step 12.2: Based on the pre-set voxel grid side length, convert the normalized coordinates of each grid vertex in the 3D human body grid into the center coordinates of the voxel grid.

[0039] In some embodiments of this application, the normalized coordinates of each vertex in the 3D human body mesh can be converted into the center coordinates of the voxel mesh using the following formula. : ; ; ; in, Indicates the first The third frame of the 3D human body mesh Normalized coordinates of each grid vertex This indicates the preset voxel mesh side length; the specific value can be set according to the actual situation. This indicates the floor function.

[0040] The above quantization process assigns points in continuous space to integer raster coordinates on a voxel grid.

[0041] Step 12.3: Remove duplicate center coordinates from all voxel meshes obtained by conversion, and determine the spatial position of all voxel meshes based on the center coordinates of all voxel meshes after deduplication.

[0042] In some embodiments of this application, for multiple mesh vertices falling into the same voxel mesh, a set deduplication operation is used to retain only the center coordinates corresponding to a unique mesh vertex. (That is, when multiple grid vertices correspond to the same center coordinates, these identical center coordinates are deduplicated, and only one is kept), thus obtaining the first... The set of sparse voxel coordinates corresponding to a frame image ,in For the first The number of voxels in the voxel grid corresponding to each frame of the image. This number is typically much smaller than the total number of meshes defined in the entire voxel space (the resolution of the entire voxel space can be set according to actual needs), reflecting the sparsity of human pose in the entire voxel space.

[0043] It is understandable that, given the center coordinates and side length of a voxel, the corresponding voxel can be determined based on these two factors. Therefore, after determining the center coordinates, this application can determine the corresponding voxel by combining it with the pre-set voxel side length, thereby obtaining a voxel mesh.

[0044] In related technologies, when obtaining a 3D human body mesh using the VIBE model, each vertex in the 3D human body mesh has a body part label (such as arm, head, etc.; for example, 0 represents the arm and 1 represents the head). Based on this, to facilitate subsequent data processing, the above-mentioned construction worker action recognition method also includes the following step of obtaining semantic labels for voxels: For each voxel mesh, the part labels corresponding to all mesh vertices within the voxel mesh are counted, and the part label with the highest percentage of occurrences is used as the semantic label for that voxel mesh. It's understandable that, based on the deduplication process described above, a voxel may contain (correspond to) multiple 3D human mesh vertices, and thus multiple corresponding part labels. In this case, the part label with the highest percentage of occurrences among all part labels can be used as the semantic label for that voxel. Conversely, if a voxel contains only one mesh vertex, the part label of that vertex is directly used as the semantic label for that voxel.

[0045] Step 13: Construct a sparse tensor based on the sparse coordinate matrix and the voxel feature matrix.

[0046] In some embodiments of this application, in order to model the temporal sequence structure of actions, it is necessary to extend the three-dimensional voxel set to a four-dimensional voxel flow, that is, to append a time index to the coordinates of each voxel. , , This indicates the number of image frames contained in the construction monitoring video.

[0047] The aforementioned sparse coordinate matrix is ​​used to record the coordinates of all voxel grids corresponding to the construction monitoring video, representing the positional information of each voxel grid in the time dimension and three-dimensional spatial dimension. (That is, the sparse coordinate matrix obtained after removing duplicates from the center coordinates of the voxel mesh) is: ; in, This represents the first voxel in the grid corresponding to all voxels of the construction monitoring video. The coordinates of the individual pixel grid, This indicates the total number of voxel grids corresponding to the construction monitoring video.

[0048] The aforementioned voxel feature matrix is ​​used to describe the features of all voxel meshes corresponding to the construction monitoring video. The aforementioned voxel feature matrix includes a first voxel feature matrix. Second voxel feature matrix The first voxel feature matrix Includes only the occupancy markers, second voxel feature matrix This includes occupancy markers and semantic labels. Specifically, if there is a pixel at the corresponding position of a voxel grid, the occupancy marker of that voxel grid is 1; if there is a pixel at the corresponding position of a voxel grid, the occupancy marker of that voxel grid is 0.

[0049] Specifically, the aforementioned first voxel feature matrix Second voxel feature matrix The expression is as follows: , ; , ; in, Indicates the first Semantic labels for individual elements.

[0050] The aforementioned voxel feature matrix can embed prior knowledge of human kinematics into the initial features of a sparse tensor in the form of "semantic labels." As the convolutional kernel slides over active voxels, it not only learns spatial geometric features but also performs "feature decoupling" through location labels, effectively improving the accuracy of action recognition. For example, when a voxel flow labeled "hand" with significant displacement features is detected, the network automatically activates discriminative neurons for "carrying" or "throwing" actions. Furthermore, this architecture solves the problem in existing technologies where relying solely on geometry makes it difficult to distinguish between "bending over" and "squatting," because the latter's semantic features are concentrated on the dramatic folding of the "knee joint" voxel, while the former is concentrated on the rotation of the "waist" voxel.

[0051] The voxel feature matrix includes a first voxel feature matrix and a second voxel feature matrix, and the sparse tensor includes a first sparse tensor and a second sparse tensor.

[0052] Specifically, it can be based on the sparse coordinate matrix and the first voxel feature matrix. Construct the first sparse tensor; based on the sparse coordinate matrix and the second voxel feature matrix... Construct a second sparse tensor; the dimension of the second sparse tensor is larger than that of the first sparse tensor. Specifically, the second sparse tensor has 5 dimensions and the first sparse tensor has 4 dimensions.

[0053] In some embodiments of this application, the sparse tensor construction method provided by the MinkowskiNet sparse convolution framework can be used to construct the sparse coordinate matrix. With the first voxel feature matrix Input the sparse tensor definition function (which is a function inside the MinkowskiNet sparse convolution framework) to construct the first sparse tensor stored in COO format.

[0054] Similarly, the sparse tensor construction method provided by the MinkowskiNet sparse convolution framework can be used to construct sparse coordinate matrices. With the second voxel feature matrix Input the sparse tensor definition function (which is a function inside the MinkowskiNet sparse convolution framework) to construct a second sparse tensor stored in COO format.

[0055] That is, by encoding the sparse coordinate matrix and the voxel feature matrix (the first voxel feature matrix or the second voxel feature matrix) in the COO format, a sparse tensor that can be input into the subsequent neural network is obtained (the first voxel feature matrix corresponds to the first sparse tensor, and the second voxel feature matrix corresponds to the second sparse tensor).

[0056] It is worth mentioning that the first and second sparse tensors mentioned above are the direct inputs to the construction worker action recognition model. This tensor structure has a spatial dimension... The above represents the location occupied by worker voxels in the time dimension. The process of continuously encoding the changes in actions is used to provide complete spatiotemporal structural information for subsequent construction worker action recognition models.

[0057] Step 14: Input the sparse tensor into the construction worker action recognition model based on sparse convolutional neural network for processing to obtain the action recognition results of construction workers in the construction monitoring video.

[0058] The aforementioned construction worker action recognition model is built upon MS-STANet (Multi-Scale Spatio-Temporal Attention Sparse Network, a deep learning neural network focused on processing spatiotemporal data). Based on two constructed sparse tensors, a lightweight multi-scale spatiotemporal attention sparse convolutional neural network (MS-STANet, the aforementioned construction worker action recognition model) is used to encode and classify the temporal voxel data of construction workers, achieving automatic recognition of worker action categories and outputting corresponding pose labels. The network as a whole adopts a structure of "multi-scale input + parallel processing and fusion + sparse convolutional backbone network + global pooling + classification layer," with the feature map resolution decreasing and the number of channels increasing progressively during computation.

[0059] Specifically, such as Figure 2 As shown, the above construction worker action recognition model includes: a first sparse convolutional layer, a second sparse convolutional layer, a sparse attention module, and a pyramid feature fusion module, a residual structure backbone network, a global pooling layer, a fully connected layer, and a classification layer connected in sequence.

[0060] The input data of the first sparse convolutional layer is the first sparse tensor, the input data of the second sparse convolutional layer is the second sparse tensor, the output of the second sparse convolutional layer is connected to the input of the sparse attention module, and the output of the sparse attention module and the output of the first sparse convolutional layer are both connected to the input of the pyramid feature fusion module.

[0061] From the above structure, the construction worker action recognition model obtains the action recognition results of construction workers in construction monitoring videos through multi-scale asynchronous input, weighted attention enhancement, pyramid feature fusion, residual structure backbone network and high-order abstraction, and global representation and classification output.

[0062] The multi-scale asynchronous input refers to the network employing an asymmetric dual-path parallel structure (i.e., the first and second sparse convolutional layers) to achieve differentiated capture of motion features at different granularities. The high-resolution branch receives a sparse tensor with a voxel spatial resolution of (64×64×64) (i.e., the second sparse tensor), whose feature vector includes occupancy markers and semantic markers. This branch focuses on capturing fine-grained motion features. The low-resolution branch receives a sparse tensor with a voxel spatial resolution of (32×32×32) (i.e., the first sparse tensor), whose feature vector only contains occupancy markers. This branch focuses on capturing the global momentum and overall displacement trajectory of the worker on the construction site.

[0063] Weighted attention enhancement refers to the introduction of a sparse attention module in the high-resolution branch. This sparse attention module automatically assigns higher computational weights to "highly moving parts" (such as hands and legs) by calculating the distribution density of active voxels in the sparse tensor, while suppressing redundant information in static parts (such as the torso), and significantly enhancing the ability to separate highly similar movements (such as bending over and squatting).

[0064] Pyramid feature fusion refers to the process where the pyramid feature fusion module arithmetically concatenates the 32-channel feature maps output from two parallel branches along the channel dimension to construct a composite spatiotemporal feature tensor with 64 channels. This operation achieves deep coupling between "local fine semantics" and "global topological trajectory," forming a feature descriptor with collaborative representation capabilities.

[0065] The residual structure backbone network and high-order abstraction refer to the following: the features fused by the pyramid feature fusion module enter the residual structure backbone network, which performs deep feature abstraction through two sets of sparse convolutional layers with residual connections. Here, the residual connection specifically refers to projective shortcut connections rather than identity mapping connections; its function is to map the input features to the target dimension before adding them to the residual terms. This structure increases network depth while effectively avoiding the gradient vanishing problem by utilizing residual mapping, ensuring stable extraction of high-order semantics from complex construction actions. The residual structure backbone network includes a third, fourth, fifth, and sixth sparse convolutional layer connected in sequence. The output of the pyramid feature fusion module is directly connected to the input of the third sparse convolutional layer. After a projection shortcut connection, the output of the pyramid feature fusion module is fused with the output of the fourth sparse convolutional layer to form a first fused output. The first fused output is directly connected to the input of the fifth sparse convolutional layer. After a projection shortcut connection, the first fused output is fused with the output of the sixth sparse convolutional layer to form a second fused output. The second fused output is connected to the input of the global pooling layer.

[0066] Global representation and classification output refer to the following: the features output from the residual backbone network are compressed into feature vectors of a specific length by a global pooling layer, mapped to the action space by a fully connected layer (FC), and finally output as probability results for various action categories of construction workers by a classification layer (Softmax). That is, the action recognition results corresponding to the construction monitoring video include the probability of various action categories for each construction worker, including actions such as bending over, climbing, and crouching. In practical applications, the system will identify the action with the highest probability as the worker's action and output an alarm when that action is considered dangerous, reminding the worker to adjust their actions and pay attention to safety.

[0067] In some embodiments of this application, the first, second, third, fourth, fifth, and sixth sparse convolutional layers are all implemented using commonly used sparse convolutional layers. Specifically, a hybrid convolutional kernel structure suitable for temporal voxel stream data is used, employing a 3×3×3 neighborhood in the spatial dimension and additionally introducing ( ) in the temporal dimension. t -1, t , t+1) Three time offsets are used to form a (3×3×3+3) four-dimensional convolutional kernel neighborhood, which captures both the local spatial structural features of the worker's body and the continuous changes in movement over time. The first and second sparse convolutional layers each have 32 channels, the third and fourth sparse convolutional layers each have 128 channels, and the fifth and sixth sparse convolutional layers each have 256 channels. The residual connection between the input of the third sparse convolutional layer and the output of the fourth sparse convolutional layer is a projective shortcut connection, implemented using a convolutional layer with a (1×1×1×1) four-dimensional convolutional kernel and 128 channels. The residual connection between the input of the fifth sparse convolutional layer and the output of the sixth sparse convolutional layer is also a projective shortcut connection, implemented using a convolutional layer with a (1×1×1×1) four-dimensional convolutional kernel and 128 channels.

[0068] It should be noted that before using the VIBE model and the construction worker motion recognition model to process construction monitoring videos and output the construction worker motion recognition results, the VIBE model and the construction worker motion recognition model need to be optimized (i.e., trained). It is understood that both the VIBE model and the construction worker motion recognition model can be trained using common deep learning model training methods; here, only a brief explanation of their training processes will be provided as an example.

[0069] The data acquisition process for training involves collecting continuous video data using fixed cameras or mobile monitoring terminals deployed at civil engineering construction sites. This yields raw video clips containing typical actions of construction workers (such as walking, bending, carrying, squatting, and climbing). The collected videos undergo initial screening, removing irrelevant, blurry, and low-quality clips unsuitable for pose reconstruction. Video sequences containing at least one construction worker's continuous actions are selected as valid samples. Furthermore, these valid video clips (i.e., valid samples) require preprocessing, including but not limited to: frame rate unification, resolution unification, color standardization, lens stability correction, and video cropping, to ensure the input data meets the requirements for subsequent 3D human pose reconstruction and voxelization.

[0070] The VIBE method is a Generative Adversarial Network (GAN) framework for body pose and shape estimation. The discriminator in this framework utilizes the AMASS dataset (a large open-source human motion capture database) to distinguish between real human motion and human motion generated by a temporal pose and shape regression network. Each preprocessed video sequence... The pre-trained HMR network (HMR network is a deep learning architecture for multi-person pose estimation) is input frame by frame for feature extraction. t Frame Image Extracting 2D visual feature vectors ,in This represents the feature extraction function of the pre-trained convolutional neural network. Subsequently, the feature sequences of all frames are input into a bidirectional gated recurrent unit (GRU) temporal encoder. By fusing forward and backward information from the sequence, a temporal hidden representation containing temporal context information is obtained. After obtaining the temporally encoded features, VIBE regresses the SMPL parameters, including pose parameters, for each frame of the image. Shape parameters And global rotation and translation parameters. The mesh vertices of the SMPL body representation model are given by the following differentiable generating functions: ; In the above formula, For SMPL mesh generator.

[0071] To improve the consistency of human body shape across different frames, VIBE averages the shape parameters of all frames in the sequence to obtain the final shape parameters. , This refers to the number of frames. During the human body mesh generation process, this is achieved through the joint regression matrix. W Regress the SMPL mesh vertices (the mesh vertices of the 3D human body mesh) to the 3D skeletal joint positions. Furthermore, by utilizing a weak perspective camera model, three-dimensional joints can be projected onto a two-dimensional plane: ; In the above formula, For camera scaling parameters, R This is a global rotation matrix. m It is a translation vector. It is an orthogonal projection operator.

[0072] The optimization objective of the VIBE model is to constrain the generated human pose and shape through a joint loss function consisting of adversarial loss and regression loss. as follows: ; In the above formula, For 3D joint supervision items, ; This is the two-dimensional reprojection error. ; For SMPL parameter regularization terms, ; This is an action prior adversarial loss, used to improve the temporal consistency and naturalness of predicted poses. ,in For action discriminators, For the pose sequence, To generate a distribution.

[0073] Finally, through the above-mentioned supervised and adversarial training, the VIBE model is trained so that it can generate stable, continuous, and human dynamics-compliant three-dimensional human posture results (i.e., three-dimensional human mesh).

[0074] When training the construction worker action recognition model, each sample of the training data consists of: a construction monitoring video clip, a sparse tensor obtained by processing the video clip using steps 11, 12, and 13 above, and the corresponding worker action label (e.g., 0 for walking, 1 for climbing, etc.). It is understood that each sample corresponds to a different 3D human body mesh to ensure sample diversity. In specific training, supervised training can be used to train the construction worker action recognition model based on this training data. During the training process, the sparse coordinate matrix... In addition to including coordinates and time indexes In addition, it also includes the batch number. .

[0075] It should be noted that the construction worker motion recognition method in this application is based on several mature and practically validated key technologies, providing a solid theoretical support and reliable foundation for its effectiveness. The 3D human parametric model SMPL has been widely used in human motion capture and 3D reconstruction tasks, characterized by its clear structure and strong consistency in expression, accurately describing the 3D geometric features of human posture. Video-level 3D human reconstruction technologies such as VIBE have demonstrated stable reconstruction performance under various complex shooting conditions, effectively acquiring temporal 3D posture information of continuous movements. Sparse convolutional neural networks and their representative framework MinkowskiNet have been validated in fields such as 3D point cloud analysis and temporal data reconstruction; their efficient computation in sparse space is particularly suitable for learning the spatiotemporal features of human voxel models. Furthermore, preprocessing methods such as voxelization encoding, normalization, and 3D sparse tensor construction have formed mature processes in multi-source 3D vision tasks, helping to ensure the uniformity of input data format and feature consistency. Based on this, this application provides a stable, scalable, and theoretically sound technical foundation for construction worker posture recognition.

[0076] In summary, the construction worker action recognition method of this application has the following advantages compared with existing worker action recognition methods based on two-dimensional images or traditional convolutional neural networks: I. This application utilizes a 3D SMPL model to reconstruct human pose and represents it in a voxelized form within a normalized coordinate system. Normalization alignment eliminates scale inconsistencies caused by individual body size differences, enhancing the model's generalizability. The input data can simultaneously contain both the spatial structure and temporal variation features of the human body, providing a more complete expression of motion compared to purely 2D keypoints or optical flow features.

[0077] Second, this application uses sparse voxel flow as network input, enabling spatiotemporal features to be convolved within a unified sparse tensor structure, avoiding information loss caused by spatial and temporal feature segmentation modeling in traditional methods. Furthermore, the sparse convolution kernel is computed only at locations with valid voxels, eliminating computational redundancy in traditional 3D CNNs for dense background data from construction sites. This allows the network to maintain acceptable computational complexity at higher spatial resolutions, improving the model's generalization stability in complex environments.

[0078] Third, by transforming the prior knowledge of human body parts in the SMPL model into voxel semantic features and combining it with a sparse attention gate mechanism, the learning of the essential features of movement is enhanced from a physical kinematics perspective. This enables the model to maintain extremely high pose estimation stability even when faced with frequent local occlusion, dramatic changes in lighting, and perspective changes such as overhead / side view at construction sites.

[0079] Fourth, the MS-STANet network architecture achieves parallel representation of high-resolution fine-grained semantic representation of motion parts and low-resolution global spatiotemporal displacement. This asymmetric design enables the model to simultaneously take into account both minute deformations of local human body movements and macroscopic spatial positional shifts, significantly improving recognition accuracy in complex motion sequences.

[0080] In summary, this application combines structured 3D representation with sparse spatiotemporal convolution, using sparse tensors as model input. This allows spatiotemporal features to be convolved within a unified sparse tensor structure, avoiding information loss caused by spatial and temporal feature segmentation in traditional methods, thus effectively improving the accuracy of construction worker action recognition. Furthermore, this application achieves a good balance between information representation capability and computational efficiency, ensuring real-time monitoring performance on edge devices with limited computing power at construction sites.

[0081] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for recognizing the actions of construction workers based on voxels and sparse convolutional neural networks, characterized in that, include: Acquire construction monitoring videos and process the construction monitoring videos using the VIBE model to obtain a three-dimensional human body mesh for each frame of the construction monitoring video. The three-dimensional human body mesh is converted into a voxel mesh, and a sparse coordinate matrix and a voxel feature matrix are constructed based on the converted voxel mesh. Based on the sparse coordinate matrix and the voxel feature matrix, a sparse tensor is constructed; The sparse tensor is input into a construction worker action recognition model based on a sparse convolutional neural network for processing to obtain the action recognition results of construction workers in the construction monitoring video.

2. The construction worker motion recognition method according to claim 1, characterized in that, The process of converting the three-dimensional human body mesh into a voxel mesh includes: The coordinates of each vertex in the three-dimensional human body mesh are aligned and normalized to obtain the normalized coordinates of each vertex in the three-dimensional human body mesh. Based on the pre-set voxel grid side length, the normalized coordinates of each grid vertex in the three-dimensional human body grid are converted into the center coordinates of the voxel grid. The center coordinates of all voxel meshes obtained by the transformation are deduplicated, and the spatial position of all voxel meshes is determined based on the center coordinates of all voxel meshes after deduplication.

3. The construction worker motion recognition method according to claim 1, characterized in that, The process of converting the normalized coordinates of each vertex in the 3D human body mesh into the center coordinates of the voxel, based on a pre-defined voxel mesh side length, includes: The normalized coordinates of each vertex in the 3D human body mesh are converted into the center coordinates of the voxel mesh using the following formula. : ； ； ； in, Indicates the first The third frame of the 3D human body mesh Normalized coordinates of each grid vertex This indicates the pre-defined voxel mesh side length. This indicates the floor function.

4. The construction worker motion recognition method according to claim 3, characterized in that, Each vertex in the three-dimensional human body mesh has a body part label, and the construction worker action recognition method further includes: For each voxel grid, the part labels corresponding to all grid vertices contained in the voxel grid are counted, and the part label with the highest number of occurrences is used as the semantic label of the voxel grid.

5. The construction worker motion recognition method according to claim 4, characterized in that, Sparse coordinate matrix for: ； in, This indicates the voxel grid corresponding to the construction monitoring video. The coordinates of the individual pixel grid, This indicates the total number of voxel grids corresponding to the construction monitoring video.

6. The construction worker motion recognition method according to claim 5, characterized in that, The voxel feature matrix includes the first voxel feature matrix. Second voxel feature matrix : ， ； ， ； in, Indicates the first Semantic labels for individual elements.

7. The construction worker motion recognition method according to claim 6, characterized in that, The sparse tensor includes a first sparse tensor and a second sparse tensor; The construction of a sparse tensor based on the sparse coordinate matrix and the voxel feature matrix includes: Based on the sparse coordinate matrix and the first voxel feature matrix Construct the first sparse tensor; Based on the sparse coordinate matrix and the second voxel feature matrix Construct a second sparse tensor; the dimension of the second sparse tensor is larger than the dimension of the first sparse tensor.

8. The construction worker motion recognition method according to claim 7, characterized in that, The construction worker action recognition model includes: a first sparse convolutional layer, a second sparse convolutional layer, a sparse attention module, and a pyramid feature fusion module, a residual structure backbone network, a global pooling layer, a fully connected layer, and a classification layer connected in sequence. The input data of the first sparse convolutional layer is a first sparse tensor, the input data of the second sparse convolutional layer is a second sparse tensor, the output of the second sparse convolutional layer is connected to the input of the sparse attention module, and the output of the sparse attention module and the output of the first sparse convolutional layer are both connected to the input of the pyramid feature fusion module.

9. The construction worker motion recognition method according to claim 8, characterized in that, The residual structure backbone network includes a third sparse convolutional layer, a fourth sparse convolutional layer, a fifth sparse convolutional layer, and a sixth sparse convolutional layer connected in sequence. The output of the pyramid feature fusion module is directly connected to the input of the third sparse convolutional layer. After a projection shortcut connection, the output of the pyramid feature fusion module is fused with the output of the fourth sparse convolutional layer to form a first fused output. The first fused output is directly connected to the input of the fifth sparse convolutional layer. After a projection shortcut connection, the first fused output is fused with the output of the sixth sparse convolutional layer to form a second fused output. The second fused output is connected to the input of the global pooling layer.

Images