Computer vision-based highway construction site safety intelligent monitoring method

Abstract

The application provides a computer vision-based highway construction site safety intelligent monitoring method, which identifies construction personnel through continuous video frame images of a highway construction site and extracts apparent feature vectors and skeleton feature vectors; each frame image in the continuous video frame images is subjected to construction scene semantic segmentation, a plurality of construction semantic region categories in the image are identified, and then a region semantic feature vector of a current position of the construction personnel is generated; the skeleton feature vector and the apparent feature vector are cross-modal fused, a category label to which a current behavior of the construction personnel belongs is generated based on the fused feature vector, and a final safety monitoring alarm signal of the current behavior is generated according to the category label and the region semantic feature vector. By adopting the scheme of the application, the computer vision technology can be used to accurately identify the behavior action of the construction personnel and intelligently judge the risk of the behavior and the space environment.

Description

Technical Field

[0001] This application relates to the field of intelligent safety monitoring technology, and more specifically, to a computer vision-based intelligent safety monitoring method for highway construction sites. Background Technology

[0002] Intelligent security monitoring is a new generation of security system that deeply integrates artificial intelligence, deep learning and big data analysis. It is no longer limited to traditional video recording and passive playback, but actively and intelligently analyzes massive real-time video streams through advanced algorithms. It can accurately identify specific behaviors, faces, vehicles and abnormal events, and provide real-time warnings and graded responses to potential risks.

[0003] Highway construction sites present complex and ever-changing environments, with frequent interactions between construction workers and various construction machinery, transport vehicles, and high-risk areas near edges, resulting in a complex interplay of safety risks. Traditional construction site safety monitoring primarily relies on on-site inspections by safety management personnel or manual monitoring using fixed cameras, which suffers from inherent shortcomings such as insufficient monitoring coverage, oversights due to personnel fatigue, and delayed response to hazardous events. In recent years, some construction sites have attempted to deploy video surveillance systems to assist in safety management; however, most existing systems only possess simple video recording and playback functions, or can only detect and identify single protective statuses such as whether construction workers are wearing safety helmets. They cannot provide a detailed understanding of the specific behaviors of construction workers, nor can they correlate personnel behavior with the spatial environmental characteristics of the surrounding construction scene. Consequently, many potential hazardous behaviors are difficult to detect and warn of in a timely manner. Therefore, how to utilize computer vision technology to accurately identify the behaviors of construction workers and intelligently assess the risks associated with these behaviors and the spatial environment has become a challenge for the industry. Summary of the Invention

[0004] This application provides a computer vision-based intelligent monitoring method for highway construction site safety, which can use computer vision technology to accurately identify the behavior of construction personnel and intelligently assess the risks associated with their behavior and the spatial environment.

[0005] In a first aspect, this application provides a computer vision-based intelligent monitoring method for safety at highway construction sites, comprising the following steps: Acquire continuous video frame images of the highway construction site; Identify construction workers in the continuous video frame images and extract appearance feature vectors representing the apparent texture of the body and hand areas, tool interaction status, and head protection status, as well as skeleton feature vectors representing the body posture and movement patterns of the construction workers. The construction scene semantic segmentation is performed on each frame of the continuous video frame image to identify multiple construction semantic region categories in the image, and then generate the regional semantic feature vector of the current location of the construction worker. The skeleton feature vector and the appearance feature vector are fused across modes. Based on the fused feature vector, the current behavior of the construction worker is classified and a category label is generated to determine the category label to which the current behavior belongs. The spatial hazard coefficient of the current behavior of the construction worker is determined according to the category label and the regional semantic feature vector, and then the final safety monitoring alarm signal of the current behavior is generated.

[0006] In some embodiments, the continuous video frame images include a multi-view video stream synchronously acquired by multiple panoramic cameras deployed at high points on the highway construction site and local cameras deployed on construction machinery. The multi-view video stream is timestamped and then framed at a fixed frame rate to obtain continuous video frame images.

[0007] In some embodiments, identifying construction workers in the continuous video frame images and extracting apparent feature vectors representing the surface texture of the body and hand regions, tool interaction status, and head protection status, as well as skeletal feature vectors representing the construction workers' body posture and movement patterns specifically includes: The continuous video frame images are input into a pre-trained target detection network to identify the detection bounding box of the construction workers in each frame image, and the image of the area where the construction workers are located is cropped according to the detection bounding box. The image of the area where the construction workers are located is input into the human pose estimation network to extract body key points and hand key points, and construct skeleton feature vectors based on the spatial connection relationship between body key points and hand key points. The image of the area where the construction workers are located is input into the appearance feature extraction network, and the appearance texture features, tool interaction status features and head protection status features of the body and hand areas are extracted respectively. The above features are concatenated into an appearance feature vector.

[0008] In some embodiments, the key body points include the head, shoulders, elbows, wrists, hips, knees, and ankles, and the key hand points include the center of the palm and the joints of each finger.

[0009] In some embodiments, performing construction scene semantic segmentation on each frame of the continuous video frame images, identifying multiple construction semantic region categories in the image, and then generating a region semantic feature vector of the current location of the construction worker specifically includes: Each frame of the continuous video frame images is input into a pre-trained semantic segmentation network, and a semantic category label is assigned to each pixel in the image; The set of pixels occupied by the construction workers is determined based on the position of the bounding box of the construction workers in the image, and the area ratio of each semantic category label in the pixel set is calculated. The area ratios of various semantic category labels are arranged into a vector in a preset order, and the ratio of the geometric center coordinates of the bounding box of the construction worker to the image size is appended to the end of the vector as the position normalization coordinates, thus generating the regional semantic feature vector of the construction worker's current position.

[0010] In some embodiments, the semantic category labels include driving lane area, shoulder area, construction operation area, material storage area, machinery operation area, and edge protection area.

[0011] In some embodiments, the skeleton feature vector and the appearance feature vector are fused across modalities, and the category label of the current behavior of the construction worker is identified based on the fused feature vector to generate the category label of the current behavior of the construction worker. Specifically, this includes: The skeleton feature vector is mapped to a skeleton semantic embedding vector through a first fully connected network, and the appearance feature vector is mapped to an appearance semantic embedding vector through a second fully connected network. A cross-modal attention mechanism is used to interactively weight and fuse the skeleton semantic embedding vector and the appearance semantic embedding vector. Specifically, the skeleton semantic embedding vector is used as the query vector and the appearance semantic embedding vector is used as the key vector to calculate the attention weights and obtain the fused feature vector. The fused feature vector is input into the behavior classifier, which outputs the probability that the current behavior of the construction worker belongs to each category in the preset behavior category set. The category with the highest probability is taken as the category label of the current behavior of the construction worker.

[0012] In some embodiments, the preset set of behavior categories includes normal walking, standing work, bending over work, climbing, carrying heavy objects, operating hand tools, driving machinery, running, waving to warn, and illegally climbing over guardrails.

[0013] In some embodiments, determining the spatial hazard coefficient of the construction worker's current behavior based on the category label and the regional semantic feature vector, and then generating a final safety monitoring alarm signal for the current behavior, specifically includes: A mapping table between behavior categories and risk levels is pre-established, and the basic risk coefficient of the current behavior is obtained by querying the mapping table according to the category label; The area category of the construction worker is determined based on the area ratio of various semantic category labels in the area semantic feature vector. If the construction worker is at the intersection of multiple area categories, the area category with the largest area ratio is taken as the current area category. The basic hazard coefficient is corrected based on the area category. When the current area category does not match the recommended work area corresponding to the category label, a penalty coefficient is added to the basic hazard coefficient to obtain the spatial hazard coefficient of the construction worker's current behavior. The final safety monitoring alarm signal for the current behavior is generated based on the spatial hazard coefficient.

[0014] In some embodiments, generating a final safety monitoring alarm signal for the current behavior based on the spatial hazard coefficient specifically includes: The shortest pixel distance between the current center of gravity of the construction worker and the boundary of the edge protection area in the semantic segmentation image is calculated based on the skeleton feature vector of the construction worker, and the shortest pixel distance is converted into the actual physical distance based on the camera calibration parameters. When the actual physical distance is less than the preset safe distance threshold and the category label belongs to the high-risk behavior category, the spatial hazard coefficient is amplified a second time according to the ratio of the actual physical distance to the preset safe distance threshold to obtain the final hazard coefficient; When the final hazard factor exceeds the preset alarm threshold, a final safety monitoring alarm signal is generated, which includes the location information of the construction personnel, behavior category labels, and hazard factor level.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The computer vision-based intelligent monitoring method for highway construction site safety provided in this application acquires continuous video frame images of the highway construction site, identifies construction workers, and simultaneously extracts appearance feature vectors containing torso and hand area textures, tool interaction status, and head protection status, as well as skeletal feature vectors representing body posture and movement patterns. This provides a comprehensive digital description of the construction worker's status from both static appearance and dynamic posture information dimensions, enabling synchronous perception of personnel's protective equipment wearing status, tool operation status, and limb movement patterns, avoiding misjudgments caused by insufficient description of single features. By performing semantic segmentation of the construction scene on each frame image and generating a regional semantic feature vector of the construction worker's current location, the spatial environment in which the construction worker is located is refined into multiple functional area categories with clear semantic meanings. This allows the system to accurately determine the spatial attributes of the construction worker's location and its relative relationship with each functional area. By fusing skeleton feature vectors and appearance feature vectors across modalities, the current behavior of construction workers is categorized and labeled based on the fused features. The spatial hazard coefficient is then determined jointly based on the category label and regional semantic feature vector, generating a final alarm signal. This organically combines the semantics of personnel behavior with the semantics of the spatial environment, achieving a comprehensive risk quantification assessment from both the inherent danger of the behavior itself and the additional environmental hazards of the location where the behavior occurs. This overcomes the shortcomings of traditional methods where behavior recognition and environmental analysis are separated, preventing spatial correlation risk assessment. It effectively reduces false alarm and false negative rates, thus achieving efficient and intelligent monitoring of personnel safety risks at highway construction sites. Using this solution, computer vision technology can be used to accurately identify the actions of construction workers and intelligently assess the risks associated with their behavior and the spatial environment. Attached Figure Description

[0016] Figure 1 This is an exemplary flowchart of a computer vision-based intelligent monitoring method for highway construction site safety, according to some embodiments of this application. Figure 2 This is an exemplary flowchart illustrating the determination of a region semantic feature vector according to some embodiments of this application; Figure 3 This is a schematic diagram illustrating the workflow of intelligent safety monitoring at highway construction sites according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a computer device for implementing a computer vision-based intelligent monitoring method for highway construction site safety, according to some embodiments of this application. Detailed Implementation

[0017] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] refer to Figure 1 The figure is an exemplary flowchart of a computer vision-based intelligent monitoring method for highway construction site safety, according to some embodiments of this application. This computer vision-based intelligent monitoring method for highway construction site safety mainly includes the following steps: In step 101, continuous video frame images of the highway construction site are acquired.

[0019] It should be noted that the continuous video frame images in this application are a set of two-dimensional pixel matrices used to characterize the visual information of a highway construction site at each acquisition moment in a continuous time series. The continuous video frame images include multi-view video streams synchronously acquired by multiple panoramic cameras deployed at high points on the highway construction site and local cameras deployed on construction machinery. These multi-view video streams are timestamped and then frame-sampled at a fixed frame rate to obtain the continuous video frame images. A panoramic camera is a fixed image acquisition device used to provide a wide-area visual coverage of the entire operation area of ​​the highway construction site from a high-point perspective. A local camera is a servo-type image acquisition device used to selectively acquire visual information about the machinery operation area and the interaction status of surrounding personnel from a near-end perspective of the construction machinery. A multi-view video stream is a data sequence carrying the original image signals with temporal continuity recorded by cameras in different spatial orientations. The continuous video frame images consist of an image sequence composed of all the original images obtained by multiple panoramic cameras deployed at high points on the highway construction site and multiple local cameras deployed on construction machinery, synchronously acquired under the same absolute time label and extracted at a fixed frame rate, arranged in ascending chronological order.

[0020] In step 102, the construction workers in the continuous video frame images are identified and the apparent feature vectors representing the surface texture of the body and hand areas, the tool interaction state, and the head protection state are extracted, as well as the skeleton feature vectors representing the body posture and movement pattern of the construction workers.

[0021] In some embodiments, identifying construction workers in the continuous video frame images and extracting apparent feature vectors representing the apparent texture of the body and hand regions, tool interaction status, and head protection status, as well as skeletal feature vectors representing the construction workers' body posture and movement patterns, can be achieved through the following steps: The continuous video frame images are input into a pre-trained target detection network to identify the detection bounding box of the construction workers in each frame image, and the image of the area where the construction workers are located is cropped according to the detection bounding box. The image of the area where the construction workers are located is input into the human pose estimation network to extract body key points and hand key points, and construct skeleton feature vectors based on the spatial connection relationship between body key points and hand key points. The image of the area where the construction workers are located is input into the appearance feature extraction network, and the appearance texture features, tool interaction status features and head protection status features of the body and hand areas are extracted respectively. The above features are concatenated into an appearance feature vector.

[0022] The key points of the torso include the head, shoulders, elbows, wrists, hips, knees, and ankles, while the key points of the hands include the center of the palm and the joints of each finger.

[0023] It should be noted that the object detection network in this application is a deep learning network model used to regress and locate the spatial position of construction workers in the input image and output the corresponding detection bounding box. The detection bounding box is a set of coordinate parameters used to mark the rectangular pixel region occupied by the construction worker in the image. The image of the construction worker's region is a local pixel matrix cropped from the original continuous video frame image based on the detection bounding box coordinates, containing only a single construction worker and its neighboring background. The human pose estimation network is a deep learning network model used to perform two-dimensional coordinate localization of the torso and hand joints of the construction worker in the input image. Torso keypoints are a set of two-dimensional coordinate points used to characterize the spatial position of the main joints of the torso and limbs of the construction worker. Hand keypoints are a set of two-dimensional coordinate points used to characterize the spatial position of the joints of the palm and fingers of the construction worker. The skeleton feature vector is a vectorized feature description used to characterize the spatial connection relationship and relative geometric configuration between the keypoints of the torso and the keypoints of the hands of the construction worker.

[0024] Additionally, it should be noted that the appearance feature extraction network in this application is a deep learning network model used to encode the visual appearance information of the torso and hand regions of construction workers in the input image. Appearance texture features are numerical feature vectors used to characterize the pixel grayscale distribution patterns and local structural patterns in the images of the torso and hand regions of construction workers. Tool interaction state features are status indicators used to characterize whether the construction worker's hand region is in contact with or holding a construction tool. Head protection state features are status indicators used to characterize whether the construction worker's head region is correctly wearing safety protective equipment. The appearance feature vector is a unified feature description vector formed by concatenating the appearance texture features of the torso region, the appearance texture features of the hand region, the tool interaction state features, and the head protection state features of the construction worker in a fixed dimensional order.

[0025] In the specific implementation of identifying construction workers in continuous video frames and extracting appearance and skeleton feature vectors, the continuous video frames are first input frame by frame into a pre-trained target detection network. This network takes the entire image of each frame as input, and after forward propagation, outputs detection bounding boxes corresponding to all construction workers in that frame. Each detection bounding box locates the specific pixel position of each construction worker in the image using the horizontal and vertical coordinates of the top-left corner of the rectangle and the width and height values ​​of the rectangle. For the same construction worker target, the target detection network outputs a unique detection bounding box with the highest confidence. Based on the coordinate parameters of each detection bounding box, a cropping operation is performed on the original image, extracting all pixels within the rectangular area defined by the detection bounding box. This generates multiple images of the construction worker areas, matching the number of construction workers. These cropped images of the construction worker areas are then used as input data for subsequent pose estimation and appearance feature extraction.

[0026] The image of the area where each construction worker is located is input into a pre-trained human pose estimation network. This network first performs deep convolutional feature encoding on the input image, and then uses a multi-stage heatmap regression method to sequentially locate the two-dimensional coordinates of torso and hand keypoints. The torso keypoints specifically include the head center point, left and right shoulder joints, left and right elbow joints, left and right wrist joints, left and right hip joints, left and right knee joints, and left and right ankle joints. The hand keypoints specifically include the palm center point and the metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints of each finger. The human pose estimation network outputs the x-coordinate and y-coordinate values ​​of each keypoint, with these coordinates in pixels representing the specific location of the keypoint in the image of the construction worker's area. After obtaining the coordinates of all body key points and hand key points, the spatial distance between each key point is calculated according to the predetermined human body topology connection relationship. This spatial distance is measured in Euclidean distance. At the same time, the tilt angle of the line connecting each adjacent key point with respect to the horizontal direction is calculated. The normalized coordinate values ​​of all key points, the spatial distance values ​​between all adjacent key points, and the directional angle values ​​of the line connecting all adjacent key points are concatenated into a one-dimensional vector in a fixed order. This one-dimensional vector is used as the skeleton feature vector of the current construction worker.

[0027] Simultaneously, the image of each construction worker's location is input into a pre-trained appearance feature extraction network, which contains multiple parallel feature extraction branches. The extraction process for the appearance texture features of the body and hand regions is as follows: the first branch of the appearance feature extraction network calculates the directional gradient histogram and performs local binary mode transformation on the pixels corresponding to the body and hand regions in the input image, and concatenates the feature vectors output by both to obtain the texture description vector. The extraction process for tool interaction state features is as follows: the second branch of the appearance feature extraction network detects and identifies tool targets in the input image. When the spatial overlap between the detection bounding box of the tool target detected within the construction worker's hand region and the key points of the hand exceeds a preset threshold, the tool holding state flag is output as active; otherwise, the tool holding state flag is output as inactive. The extraction process for head protection state features is as follows: the third branch of the appearance feature extraction network performs safety helmet color histogram statistics and classification recognition on the head region in the input image. When the recognition result indicates that the safety helmet is fully worn, the head protection state flag is output as effective protection; otherwise, the head protection state flag is output as ineffective protection. The appearance feature extraction network outputs the appearance texture features of the torso and hand regions as vectors, and outputs the tool interaction state features and head protection state features as scalar flags. The appearance texture feature vectors of the torso region, the appearance texture feature vectors of the hand region, the tool interaction state flags, and the head protection state flags are concatenated end to end according to a preset dimensional order to form a complete one-dimensional vector, and this one-dimensional vector is used as the appearance feature vector of the current construction worker.

[0028] The specific steps for determining the aforementioned object detection network, human pose estimation network, and appearance feature extraction network are as follows: First, a large number of labeled construction worker image samples are extracted from the historical highway construction site monitoring video database. The bounding box coordinates of the construction worker in each sample image are used as the ground truth. The object detection network is iteratively trained using this ground truth as the supervision signal. The object detection network calculates the predicted bounding box coordinates through forward propagation and calculates the regression loss based on the positional deviation between the predicted bounding box and the ground truth. At the same time, the classification loss is calculated based on the difference between the predicted class confidence and the true class label. The regression loss and the classification loss are weighted and summed. The summation is used as the total loss function. The weight parameters of the object detection network are updated layer by layer using the backpropagation algorithm. This training process is repeated until the total loss function value no longer decreases significantly, at which point training stops and the network parameters at this point are fixed as the final parameters of the object detection network. A large number of image samples containing complete human body and hand keypoint coordinate annotations are obtained from publicly available human pose datasets and a self-built hand pose dataset. The pixel coordinates of the torso and hand keypoints in each sample image are used as supervision signals. This image sample is input into the human pose estimation network, which outputs a predicted keypoint heatmap. The pixel-level differences between the predicted heatmap and the labeled ground truth heatmap are calculated. Cross-entropy loss is used to update network weights via backpropagation. Training is repeated until the loss converges, at which point the network parameters are fixed as the final parameters of the human pose estimation network. For the appearance feature extraction network, a multi-task learning strategy is employed for joint training. Training data is obtained from image samples labeled with body texture categories, hand texture categories, tool interaction state categories, and head protection state categories. Each sample image is input into the appearance feature extraction network, which outputs body texture feature vectors, hand texture feature vectors, tool interaction state classification results, and head protection state classification results, respectively. The body texture feature vector is then compared with the labeled texture feature vectors. The mean squared error between quantities is used as the texture regression loss. The cross-entropy between the tool interaction state classification result and the labeled state label is used as the tool state classification loss. The cross-entropy between the head protection state classification result and the labeled state label is used as the protection state classification loss. The texture regression loss, tool state classification loss and protection state classification loss are weighted and summed according to the preset weight coefficients to obtain the multi-task total loss. The weight parameters of each branch of the appearance feature extraction network are jointly optimized using the backpropagation algorithm. The above multi-task training process is repeated until the multi-task total loss converges. Then, the network parameters at this time are fixed as the final parameters of the appearance feature extraction network.

[0029] In step 103, construction scene semantic segmentation is performed on each frame of the continuous video frame images to identify multiple construction semantic region categories in the image, thereby generating a regional semantic feature vector of the current location of the construction worker.

[0030] In some embodiments, reference Figure 2 The figure is an exemplary flowchart for determining the semantic feature vector of a region in some embodiments of this application. In this embodiment, the construction scene semantic segmentation is performed on each frame of the continuous video frame image to identify multiple construction semantic region categories in the image, and then the region semantic feature vector of the current location of the construction worker is generated. This can be achieved by the following steps: In step 1031, each frame of the continuous video frame images is input into a pre-trained semantic segmentation network, and a semantic category label is assigned to each pixel in the image; In step 1032, the set of pixels occupied by the construction workers is determined based on the position of the detection bounding box of the construction workers in the image, and the area ratio of each semantic category label in the pixel set is calculated. In step 1033, the area ratios of various semantic category labels are arranged into a vector in a preset order, and the ratio of the geometric center coordinates of the bounding box of the construction worker detection to the image size is appended to the end of the vector as the position normalization coordinates, thereby generating the regional semantic feature vector of the construction worker's current position.

[0031] The semantic category labels include driving lane area, shoulder area, construction operation area, material storage area, machinery operation area, and edge protection area.

[0032] It should be noted that the semantic segmentation network in this application is a deep learning network model used to assign a semantic category label to each pixel location in the input image. The semantic category label is a category identifier used to indicate the functional region category of the construction scene to which each pixel in the image belongs. The pixel set is the overall set of all pixel locations covered by the detection bounding box of a construction worker. The area ratio is a numerical value used to represent the proportion of pixels with a certain semantic category label in the construction worker pixel set. The preset order is an arrangement rule used to specify the fixed order of various semantic category labels when arranging vectors. The geometric center coordinates are coordinate pairs representing the horizontal and vertical position values ​​of the center point of the construction worker's detection bounding box rectangle in the image plane. The position normalized coordinates are standardized coordinate values ​​used to convert the absolute pixel position of the construction worker in the image into a relative position independent of the image size. The region semantic feature vector is a comprehensive feature description vector used to represent the distribution of various construction semantic regions corresponding to the current position of the construction worker and the relative position of the construction worker in the image.

[0033] In the specific implementation of semantic segmentation of the construction scene and generation of the regional semantic feature vector of the current location of construction workers in each frame of a continuous video image, each frame of the acquired continuous video image is first used as input data and fed into a pre-trained semantic segmentation network frame by frame. This semantic segmentation network performs deep feature extraction and classification calculation on each pixel position of the input image. After forward propagation, it outputs a semantic label map with the same spatial resolution as the input image. The output value at each pixel position in this semantic label map is a category identifier selected from six semantic category labels: driving lane area, shoulder area, construction operation area, material stacking area, machinery operation area, and edge protection area. This allows each pixel in the input image to be assigned a semantic category label for its corresponding construction scene functional area. After semantic segmentation of one frame, the semantic label map corresponding to that frame is obtained.

[0034] Subsequently, the coordinate parameters of the bounding box corresponding to each construction worker in the frame image are obtained from the object detection network. Based on the four parameters of the bounding box in the image—the horizontal and vertical coordinates of the top-left corner, the width, and the height of the rectangle—a rectangular pixel region that completely overlaps with the spatial position of the bounding box is determined on the semantic label map. The entire pixel set consisting of all pixels within this rectangular pixel region is taken as the pixel set occupied by the construction worker. The semantic category label of each pixel in this pixel set is counted, specifically the number of pixels labeled as driving lane area, shoulder area, construction operation area, material stacking area, machinery operation area, and edge protection area. The number of pixels for each semantic category label is divided by the total number of pixels in the pixel set to obtain the area ratio of each semantic category label in the pixel set of the construction worker. This area ratio is represented by a real number between zero and one.

[0035] The calculated area ratios of the driving lane area, shoulder area, construction operation area, material storage area, machinery operation area, and edge protection area are sequentially filled into the corresponding positions of a one-dimensional array according to a pre-set fixed arrangement, forming a regional distribution feature vector composed of six area ratio components. Simultaneously, the geometric center coordinates of the detection bounding box of the construction worker are obtained; that is, the sum of the horizontal coordinate of the upper left corner of the detection bounding box and half the width of the rectangle is used as the geometric center horizontal coordinate, and the sum of the vertical coordinate of the upper left corner of the detection bounding box and half the height of the rectangle is used as the geometric center vertical coordinate. The image dimensions of the current frame image are obtained; that is, the total number of pixels in the horizontal direction is used as the horizontal dimension value, and the total number of pixels in the vertical direction is used as the vertical dimension value. The horizontal coordinate of the geometric center is divided by the horizontal dimension value to obtain the horizontal normalized coordinate value, and the vertical coordinate of the geometric center is divided by the vertical dimension value to obtain the vertical normalized coordinate value. The horizontal and vertical normalized coordinate values ​​are sequentially appended to the end of the above-mentioned regional distribution feature vector to form an augmented vector composed of eight components. The augmented vector is used as the region semantic feature vector of the current construction worker's current position in the image frame.

[0036] The specific steps for determining the semantic segmentation network are as follows: First, a large number of image samples covering different lighting conditions, different construction stages, and different weather conditions are selected from the highway construction site monitoring video database. For each pixel location in each image sample, the labeling personnel manually label the functional area category of the construction scene to which it belongs. The labeled semantic category labels include driving lane area, shoulder area, construction operation area, material stacking area, machinery operation area, and edge protection area. The labeled image samples form a training dataset with pixel-by-pixel semantic category ground truth labels. The image samples in the training dataset are input one by one into the semantic segmentation network to be trained. The semantic segmentation network downsamples the input image through the encoder to extract multi-scale semantic features, and then upsamples the feature map through the decoder to restore the original spatial resolution of the input image. Finally, it outputs a predicted semantic label map with the same size as the input image. Each pixel location in the predicted semantic label map contains the network's classification of the pixel. The predicted probability distribution vectors of various semantic category labels are used. The difference between the predicted probability distribution vectors and the true semantic category labels provided by the annotators for each pixel is used as the pixel-wise classification loss. The average of the classification losses of all pixels is taken as the overall loss value of the image sample. The stochastic gradient descent optimization algorithm is used to calculate the gradient direction of the convolutional kernel parameters of each layer in the semantic segmentation network based on the overall loss value and update the network weight parameters along the gradient descent direction. After completing one round of training by traversing all image samples in the training dataset, the validation dataset is input into the current network to calculate the overall loss value of the validation set. The above image-wise training and validation set evaluation process is repeated. When the overall loss value of the validation set no longer decreases or the decrease is lower than the preset rate of change threshold for several consecutive rounds, the network training is determined to be converged. Training is stopped and the network weight parameters at this time are solidified and stored in the model library. The solidified semantic segmentation network is used as the preset network for performing pixel-wise semantic region category recognition for each frame of continuous video images.

[0037] In step 104, the skeleton feature vector and the appearance feature vector are fused across modes. Based on the fused feature vector, the current behavior of the construction worker is identified by category label, and the category label to which the current behavior of the construction worker belongs is generated. The spatial hazard coefficient of the current behavior of the construction worker is determined according to the category label and the regional semantic feature vector, and then the final safety monitoring alarm signal of the current behavior is generated.

[0038] In some embodiments, the skeleton feature vector and the appearance feature vector are fused across modalities, and the category label of the construction worker's current behavior is identified based on the fused feature vector to generate the category label to which the construction worker's current behavior belongs. This can be achieved by the following steps: The skeleton feature vector is mapped to a skeleton semantic embedding vector through a first fully connected network, and the appearance feature vector is mapped to an appearance semantic embedding vector through a second fully connected network. A cross-modal attention mechanism is used to interactively weight and fuse the skeleton semantic embedding vector and the appearance semantic embedding vector. Specifically, the skeleton semantic embedding vector is used as the query vector and the appearance semantic embedding vector is used as the key vector to calculate the attention weights and obtain the fused feature vector. The fused feature vector is input into the behavior classifier, which outputs the probability that the current behavior of the construction worker belongs to each category in the preset behavior category set. The category with the highest probability is taken as the category label of the current behavior of the construction worker.

[0039] The preset set of behavior categories includes normal walking, standing work, bending over work, climbing, carrying heavy objects, operating hand tools, driving machinery, running, waving to warn, and illegally climbing over guardrails.

[0040] It should be noted that the skeleton feature vector in this application is a vectorized feature description used to characterize the spatial connection relationship and relative geometric configuration between key points of the torso and key points of the hands of construction workers.

[0041] It should be noted that the first fully connected network in this application is a neural network layer structure used to map the input skeleton feature vector to a preset embedding dimension space through linear transformation and nonlinear activation. The skeleton semantic embedding vector is an intermediate feature vector used to represent the compact vectorized representation of the construction worker's posture skeleton information in the semantic space. The second fully connected network is a neural network layer structure used to map the input appearance feature vector to a preset embedding dimension space through linear transformation and nonlinear activation. The appearance semantic embedding vector is an intermediate feature vector used to represent the compact vectorized representation of the construction worker's appearance visual information in the semantic space. The cross-modal attention mechanism is an attention calculation structure used to interactively weight and fuse feature embedding vectors from different modalities. Specifically, it is a calculation process that uses the embedding vector of one modality as query information to perform relevance weighted aggregation on the embedding vector of another modality. The query vector is a feature vector used to provide query basis and measure the relevance between other vectors during the attention calculation process. The key-value vector is a feature vector used as the query object and provide a relevance measurement reference during the attention calculation process. The attention weight is a normalized set of values ​​used to represent the relevance strength between the query vector and each element in the key-value vector. The fused feature vector is a comprehensive feature description vector obtained by interactively and weightedly fusing the semantic embedding vectors of the skeleton modality and the appearance modality through a cross-modal attention mechanism. The behavior classifier is a decision network structure used to map the input fused feature vector to the probability distribution of each behavior category in a preset behavior category set for the current behavior of construction workers. The preset behavior category set is a finite set used to limit all standard behavior categories that construction workers may exhibit at the highway construction site. The category label is a category identifier symbol used to uniquely identify the behavior category to which the current behavior of the construction worker belongs.

[0042] In the specific implementation of cross-modal fusion of skeleton feature vectors and appearance feature vectors to identify the category label of a construction worker's current behavior, the acquired skeleton feature vectors are first fed into a pre-constructed first fully connected network as input data. This first fully connected network contains multiple sequentially stacked fully connected layers. Each layer performs a linear transformation calculation on the input vector, that is, multiplying the input vector by the layer's weight matrix and adding a bias vector. The result of the linear transformation is then input into a non-linear activation function to enhance the network's expressive power. After the skeleton feature vectors are processed sequentially through each layer of the first fully connected network, they are mapped from the original skeleton space dimension to a pre-defined semantic embedding dimension space, outputting a fixed-length compact vector, which is used as the skeleton semantic embedding vector. Simultaneously, the acquired appearance feature vectors are fed into a pre-constructed second fully connected network as input data. The second fully connected network has the same structure as the first fully connected network but its network parameters are independent. After the appearance feature vector is sequentially processed by the linear transformations and nonlinear activations of each layer of the second fully connected network, it is mapped from the original appearance space dimension to the semantic embedding dimension space of the same as the skeleton semantic embedding vector, and a compact vector with the same length as the skeleton semantic embedding vector is output. This compact vector is used as the appearance semantic embedding vector.

[0043] Both the skeleton semantic embedding vector and the appearance semantic embedding vector are input into the cross-modal attention mechanism module for interactive weighted fusion calculation. In this cross-modal attention mechanism, the skeleton semantic embedding vector is designated as the query vector, and the appearance semantic embedding vector is simultaneously designated as the key-value vector. Both the key and value vectors originate from the same appearance semantic embedding vector but participate in different calculation stages. First, the dot product similarity between the query vector and the key vector is calculated to obtain the original attention score vector. Then, a normalized exponential function is applied to normalize this original attention score vector, transforming all score values ​​into attention weights between zero and one, with a sum equal to one. These attention weights reflect the degree of attention the skeleton modality pays to different dimensions of the appearance modality. The normalized attention weights are then multiplied element-wise with the value vector, that is, each weight coefficient in the attention weight is multiplied by the corresponding element in the value vector. All multiplication results are summed to obtain the intermediate feature vector after weighted aggregation of the appearance modality with the skeleton modality as the query condition. Then, the intermediate feature vector is added element by element to the skeleton semantic embedding vector to obtain a fused feature vector that integrates the skeleton modality query information and the appearance modality query information.

[0044] The fused feature vector is input into a pre-built behavior classifier, which contains at least one fully connected layer and a normalized exponential output layer. The fused feature vector is first linearly mapped through the fully connected layer of the behavior classifier, transforming its dimension to the total number of categories in the preset behavior category set, resulting in an unnormalized score for each behavior category. The preset behavior category set predefines ten behavior categories: normal walking, standing work, bending work, climbing, carrying heavy objects, operating hand tools, driving machinery, running, waving warnings, and illegally climbing over guardrails. The normalized exponential output layer of the behavior classifier applies a normalized exponential function to all unnormalized scores, converting each score into a probability value between zero and one, with a total probability of one. Each probability value corresponds to the confidence level that the worker's current behavior belongs to the corresponding behavior category in the preset behavior category set. The largest of these ten probability values ​​is selected as the maximum probability value, and the category identifier corresponding to this maximum probability value is extracted as the category label of the worker's current behavior.

[0045] The specific steps for determining the first fully connected network, the second fully connected network, and the behavior classifier are as follows: First, a large number of image samples containing various behaviors of construction workers are selected from the highway construction site monitoring video database. For each image sample, the corresponding skeleton feature vector and appearance feature vector are obtained as training input data according to the aforementioned steps. At the same time, the annotators label each image sample with the true category label to which the current behavior of the construction worker belongs. The true category label is selected from ten categories in the preset behavior category set: normal walking, standing work, bending over work, climbing, carrying heavy objects, operating hand tools, driving machinery, running, waving warnings, and illegally climbing over guardrails. All annotations are completed. Image samples and their corresponding skeleton feature vectors, appearance feature vectors, and ground truth class labels constitute the behavior recognition training dataset. Each set of skeleton and appearance feature vectors in the training dataset is used as input, sequentially mapped to a skeleton semantic embedding vector via a first fully connected network, and then to an appearance semantic embedding vector via a second fully connected network. The skeleton and appearance semantic embedding vectors are then fused into a fused feature vector using a cross-modal attention mechanism. Finally, the fused feature vector is input to the behavior classifier, which outputs the predicted probability distribution of the construction worker's current behavior belonging to each category in a preset behavior category set. The category corresponding to the highest probability value in the predicted probability distribution is used as the predicted category label. The predicted category label is compared with the true category label provided by the annotator, and the cross-entropy loss value between the two is calculated as the prediction error for the training sample. The prediction error is propagated back layer by layer using the backpropagation algorithm. At the same time, the gradients of the weight parameters of each layer in the first fully connected network, the second fully connected network, and the behavior classifier are calculated, and all trainable parameters of the first fully connected network, the second fully connected network, and the behavior classifier are updated synchronously along the gradient descent direction. After completing one round of training by traversing all samples in the behavior recognition training dataset, the validation dataset is input into the current network to calculate the average cross-entropy loss value on the validation set. The above sample-by-sample training and validation process is repeated. In the evaluation process, when the average cross-entropy loss value of the validation set no longer decreases or the decrease is lower than the preset rate of change threshold, the network training is determined to be converged. Training is stopped, and all weight parameters of the first fully connected network, the second fully connected network, and the behavior classifier are solidified and stored in the model library. The solidified first fully connected network is used as the preset network for mapping skeleton feature vectors to skeleton semantic embedding vectors, the solidified second fully connected network is used as the preset network for mapping appearance feature vectors to appearance semantic embedding vectors, and the solidified behavior classifier is used as the preset network for mapping fused feature vectors to the category label of the current behavior of the construction worker.

[0046] In some embodiments, determining the spatial hazard coefficient of the construction worker's current behavior based on the category label and the regional semantic feature vector, and then generating the final safety monitoring alarm signal for the current behavior, can be achieved through the following steps: A mapping table between behavior categories and risk levels is pre-established, and the basic risk coefficient of the current behavior is obtained by querying the mapping table according to the category label; The area category of the construction worker is determined based on the area ratio of various semantic category labels in the area semantic feature vector. If the construction worker is at the intersection of multiple area categories, the area category with the largest area ratio is taken as the current area category. The basic hazard coefficient is corrected based on the area category. When the current area category does not match the recommended work area corresponding to the category label, a penalty coefficient is added to the basic hazard coefficient to obtain the spatial hazard coefficient of the construction worker's current behavior. The final safety monitoring alarm signal for the current behavior is generated based on the spatial hazard coefficient.

[0047] It should be noted that the mapping table in this application is a data structure used to establish the correspondence between behavior categories and hazard levels, where each behavior category is associated with a pre-defined basic hazard coefficient value. The basic hazard coefficient is a benchmark value used to characterize the inherent hazard level of a certain behavior category in a typical construction scenario. The boundary is used to characterize the spatial position state of the construction worker's pixel set, where multiple semantic category labels are simultaneously present and the area proportion of each category does not exceed a preset majority threshold. The recommended work area is a set of pre-defined functional area categories suitable and safe for a certain behavior category in a construction scenario. The penalty coefficient is a value used to incrementally correct the basic hazard coefficient when the construction worker's current area category does not match the recommended work area corresponding to the current behavior category. The spatial hazard coefficient is a comprehensive value used to characterize the degree of danger posed by the construction worker's current behavior in the current spatial environment.

[0048] In the specific implementation process of determining the spatial hazard coefficient of the construction worker's current behavior based on category labels and regional semantic feature vectors and generating the final safety monitoring alarm signal, the pre-established mapping relationship table between behavior categories and hazard levels is first read from local storage. This mapping relationship table pre-sets corresponding basic hazard coefficient values ​​for ten behavior categories in the preset behavior category set: normal walking, standing work, bending over work, climbing, carrying heavy objects, operating hand tools, driving machinery, running, waving warnings, and illegally climbing over guardrails. Using the category label of the construction worker's current behavior identified in step 105 as the query keyword, a matching search is performed in the mapping relationship table. When a behavior category entry matching the category label is found, the pre-stored basic hazard coefficient value under that entry is extracted and used as the basic hazard coefficient of the current behavior.

[0049] Subsequently, the six area ratio components, located in the first six positions of the region semantic feature vector, are read. These six area ratio components correspond to the area ratio values ​​of the driving lane area, the shoulder area, the construction operation area, the material storage area, the machinery operation area, and the edge protection area, respectively. These six area ratio values ​​are compared, and the largest value is selected. The semantic category label corresponding to this largest area ratio value is determined as the current area category of the construction worker. When multiple area ratio values ​​are equal and all are the maximum value, it indicates that the construction worker is at the boundary of multiple area categories; similarly, the semantic category label with the largest and equal area ratio is used as the current area category.

[0050] Retrieve the recommended work area set corresponding to the current behavior category label. This recommended work area set is a list of construction scenario functional area categories that are allowed for safe operation, pre-defined in the system configuration for each behavior category. Determine whether the current area category of the construction worker, as determined in the previous step, belongs to the recommended work area set corresponding to the current behavior category label. If the current area category belongs to the recommended work area set, no correction operation is performed, and the basic hazard coefficient is directly output as the spatial hazard coefficient. If the current area category does not belong to the recommended work area set, it means that the construction worker is working in an area unsuitable for the current behavior. The pre-set penalty coefficient value is read from the system configuration and added to the basic hazard coefficient to obtain the corrected spatial hazard coefficient.

[0051] The value of the penalty coefficient is positively correlated with the degree of mismatch between the region category and the behavior category. When the degree of mismatch is high, the penalty coefficient takes a larger value. The specific steps for determining the penalty coefficient are as follows: extract all event records where the region category and behavior category do not match from historical security event data, classify and statistically analyze them according to the combination of region category and behavior category, calculate the ratio between the occurrence rate of dangerous events of each mismatch combination and the average occurrence rate of dangerous events of all matching combinations as the initial penalty coefficient, multiply the initial penalty coefficient by the inherent danger weight corresponding to the region category to obtain the weighted penalty coefficient, multiply the weighted penalty coefficient by the scaling factor of the basic danger coefficient of the current behavior category to obtain the scaled penalty coefficient, compare the scaled penalty coefficient with the preset maximum penalty limit value and take the smaller one as the final penalty coefficient, and store the final determined penalty coefficient in the system configuration as the preset penalty coefficient to be called when the basic danger coefficient is subsequently corrected.

[0052] In addition, in some embodiments, generating a final safety monitoring alarm signal for the current behavior based on the spatial hazard factor can be achieved using the following steps: The shortest pixel distance between the current center of gravity of the construction worker and the boundary of the edge protection area in the semantic segmentation image is calculated based on the skeleton feature vector of the construction worker, and the shortest pixel distance is converted into the actual physical distance based on the camera calibration parameters. When the actual physical distance is less than the preset safe distance threshold and the category label belongs to the high-risk behavior category, the spatial hazard coefficient is amplified a second time according to the ratio of the actual physical distance to the preset safe distance threshold to obtain the final hazard coefficient; When the final hazard factor exceeds the preset alarm threshold, a final safety monitoring alarm signal is generated, which includes the location information of the construction personnel, behavior category labels, and hazard factor level.

[0053] It should be noted that the semantic segmentation image in this application is a label map matrix used to represent that each pixel position in a continuous video frame image is assigned a semantic category label of the construction scene. The boundary of the edge protection area is a set of pixel coordinates used to represent the boundary line position between the pixel areas covered by the edge protection area category label and the pixel areas not covered by the edge protection area category label in the semantic segmentation image. The shortest pixel distance is a numerical value used to represent the number of pixels traversed by the shortest line connecting the current center of gravity position of the construction worker to the edge protection area boundary in the image plane. The camera calibration parameters are a set of numerical parameters used to represent the geometric relationship of camera imaging, which establishes a mapping relationship between image pixel coordinates and actual physical space coordinates. The actual physical distance is a numerical value representing the shortest straight-line distance from the current center of gravity position of the construction worker to the edge protection area boundary in real three-dimensional space, expressed in meters. The preset safe distance threshold is a critical safe distance value used to determine whether the construction worker is too close to the edge protection area, expressed in meters. The high-risk behavior category is a set of several behavior categories with a high inherent degree of danger within the preset behavior category set. Secondary magnification is a calculation operation used to additionally increase the spatial hazard factor based on the proximity of construction workers to the edge protection area. The final hazard factor is a comprehensive value characterizing the final hazard level of the construction worker's current behavior after edge distance correction. The preset alarm threshold is the lowest hazard factor critical value used to determine whether the construction worker's current behavior poses a safety risk and requires triggering an alarm. Construction worker location information is a numerical pair characterizing the coordinates of the geometric center of the bounding box detected by the construction worker in continuous video frames. The hazard factor level is a classification identifier used to discretize the continuous values ​​of the final hazard factor into several risk level grades. The final safety monitoring alarm signal is the control command information used to indicate that the construction worker's current behavior poses a safety risk and triggers an on-site warning response.

[0054] The steps for determining the preset alarm threshold are as follows: extract historical safety event data from the safety management database, and use the spatial hazard coefficient values ​​in ascending order as candidate thresholds. Calculate the hazard identification accuracy, hazard event false alarm rate, and safety event false alarm rate corresponding to each candidate threshold. Based on the maximum acceptable false alarm rate and maximum false alarm rate set by the safety management personnel, select all candidate thresholds that meet the conditions. Select the threshold with the highest hazard identification accuracy as the preliminary threshold. Verify the preliminary threshold online during actual operation. Once the verification is successful, determine it as the preset alarm threshold.

[0055] In the specific implementation of generating the final safety monitoring alarm signal for the current behavior based on the spatial hazard coefficient, the coordinates of key points of the construction worker's body are first read from the acquired skeleton feature vector. The center point of the hip is selected as the current center of gravity position of the construction worker. At the same time, the pixel positions of all semantic category labels labeled as edge protection areas are extracted from the generated semantic segmentation image. The set of boundary pixel coordinates between these pixel positions and adjacent non-edge protection area pixels is obtained as the boundary of the edge protection area. The Euclidean distance between the current center of gravity position of the construction worker and each pixel coordinate on the edge protection area boundary is calculated, and the minimum distance value is taken as the shortest pixel distance. The horizontal and vertical pixel equivalent values ​​of the camera are read from the camera calibration parameters, and the average of the two is taken as the pixel equivalent reference value. The shortest pixel distance is divided by the pixel equivalent reference value to obtain the actual physical distance between the current center of gravity position of the construction worker and the edge of the edge protection area. This actual physical distance, in meters, represents the straight-line distance between the construction worker and the edge of the high fall risk area in real space.

[0056] The system determines whether the calculated actual physical distance is less than a preset safety distance threshold, and simultaneously determines whether the identified category label belongs to a predefined set of high-risk behavior categories. If the actual physical distance is less than the preset safety distance threshold and the category label belongs to the high-risk behavior category set, a secondary amplification correction process is initiated. This process involves calculating the ratio of the actual physical distance to the preset safety distance threshold. This ratio represents the proximity of the construction worker to the edge protection area; a smaller ratio indicates greater proximity. An amplification factor is calculated based on this ratio, which monotonically increases as the ratio decreases. The determined spatial hazard coefficient is multiplied by this amplification factor to obtain the corrected final hazard coefficient. If the actual physical distance is not less than the preset safety distance threshold or the category label does not belong to the high-risk behavior category set, no secondary amplification correction is performed, and the spatial hazard coefficient is directly used as the final hazard coefficient.

[0057] Additionally, it should be noted that the high-risk behavior categories in this application include climbing, running, waving warnings, and illegally climbing over guardrails. Furthermore, operating hand tools and driving machinery in hazardous environments such as near edge protection areas may also be classified as high-risk behaviors. These behaviors inherently possess a high degree of danger; for example, climbing and climbing over guardrails can easily lead to falls from heights, running can easily result in collisions or falls in complex construction environments, and waving warnings are often associated with emergency situations. When these behaviors occur near edge protection areas, their risk level further increases, requiring the triggering of safety monitoring alarms.

[0058] After obtaining the final hazard coefficient, it is compared with a preset alarm threshold. When the final hazard coefficient is lower than the preset alarm threshold, it is determined that the current behavior does not pose a significant safety risk, and no alarm signal is generated. When the final hazard coefficient is equal to or higher than the preset alarm threshold, it is determined that the construction worker's current behavior poses a safety risk. The geometric center coordinates of the construction worker's detection bounding box are then read as the construction worker's location information, and the identified category label is read as the behavior category label. Simultaneously, the continuous values ​​of the final hazard coefficient are mapped to the corresponding risk level according to a preset discretization rule to obtain the hazard coefficient level, which includes at least three levels: low risk, medium risk, and high risk. The three information fields—construction worker location information, behavior category label, and hazard coefficient level—are combined and encapsulated into a final safety monitoring alarm signal.

[0059] In some embodiments, reference Figure 3 This figure is a schematic diagram of the workflow of intelligent monitoring of highway construction site safety in some embodiments of this application. As shown in the figure, the image acquisition module on the left acquires continuous video frame images of the construction site through the monitoring camera. The video images are transmitted to the image processing module in the middle via the arrow. This module completes the extraction of the appearance features of construction personnel, the skeleton posture features, and the semantic segmentation of the construction scene to obtain regional semantic features. Then, it performs multimodal feature fusion, personnel behavior recognition, and spatial hazard coefficient determination calculation. The processing results are sent to the alarm signal module on the right, and finally outputs a safety monitoring alarm signal, thus fully realizing the entire process from on-site image acquisition and intelligent visual analysis to safety risk alarm.

[0060] In addition, this application also provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to acquire the code and execute the above-described computer vision-based intelligent monitoring method for highway construction site safety.

[0061] In some embodiments, reference Figure 4 The figure is a schematic diagram of the structure of a computer device implementing a computer vision-based intelligent monitoring method for highway construction site safety, according to some embodiments of this application. The computer vision-based intelligent monitoring method for highway construction site safety in the above embodiments can... Figure 4 The computer device shown is used to implement this, and the computer device 400 includes at least one processor 401, a communication bus 402, a memory 403, and at least one communication interface 404.

[0062] Processor 401 can be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0063] The communication bus 402 can be used to transmit information between the aforementioned components.

[0064] The memory 403 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 403 may exist independently and be connected to the processor 401 via the communication bus 402. The memory 403 may also be integrated with the processor 401.

[0065] The memory 403 stores program code for executing the scheme of this application, and its execution is controlled by the processor 401. The processor 401 executes the program code stored in the memory 403. The program code may include one or more software modules. The method used in the above embodiments can be implemented by the processor 401 and one or more software modules in the program code in the memory 403.

[0066] Communication interface 404 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0067] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0068] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0069] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described computer vision-based intelligent monitoring method for highway construction site safety.

[0070] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0071] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A computer vision-based intelligent monitoring method for safety at highway construction sites, characterized in that, Includes the following steps: Acquire continuous video frame images of the highway construction site; Identify construction workers in the continuous video frame images and extract appearance feature vectors representing the apparent texture of the body and hand areas, tool interaction status, and head protection status, as well as skeleton feature vectors representing the body posture and movement patterns of the construction workers. The construction scene semantic segmentation is performed on each frame of the continuous video frame image to identify multiple construction semantic region categories in the image, and then generate the regional semantic feature vector of the current location of the construction worker. The skeleton feature vector and the appearance feature vector are fused across modes. Based on the fused feature vector, the current behavior of the construction worker is classified and a category label is generated to determine the category label to which the current behavior belongs. The spatial hazard coefficient of the current behavior of the construction worker is determined according to the category label and the regional semantic feature vector, and then the final safety monitoring alarm signal of the current behavior is generated.

2. The method as described in claim 1, characterized in that, The continuous video frame images include multi-view video streams synchronously acquired by multiple panoramic cameras deployed at high points on the highway construction site and local cameras deployed on construction machinery. The multi-view video streams are timestamped and then extracted at a fixed frame rate to obtain continuous video frame images.

3. The method as described in claim 1, characterized in that, The identification of construction workers in the continuous video frame images and the extraction of apparent feature vectors representing the surface texture of the body and hand areas, tool interaction status, and head protection status, as well as skeletal feature vectors representing the body posture and movement patterns of the construction workers specifically include: The continuous video frame images are input into a pre-trained target detection network to identify the detection bounding box of the construction workers in each frame image, and the image of the area where the construction workers are located is cropped according to the detection bounding box. The image of the area where the construction workers are located is input into the human pose estimation network to extract body key points and hand key points, and construct skeleton feature vectors based on the spatial connection relationship between body key points and hand key points. The image of the area where the construction workers are located is input into the appearance feature extraction network, and the appearance texture features, tool interaction status features and head protection status features of the body and hand areas are extracted respectively. The above features are concatenated into an appearance feature vector.

4. The method as described in claim 3, characterized in that, The key points of the torso include the head, shoulders, elbows, wrists, hips, knees, and ankles, and the key points of the hands include the center of the palm and the joints of each finger.

5. The method as described in claim 1, characterized in that, Performing construction scene semantic segmentation on each frame of the continuous video images, identifying multiple construction semantic region categories in the image, and then generating a region semantic feature vector of the current location of the construction worker specifically includes: Each frame of the continuous video frame images is input into a pre-trained semantic segmentation network, and a semantic category label is assigned to each pixel in the image; The set of pixels occupied by the construction workers is determined based on the position of the bounding box of the construction workers in the image, and the area ratio of each semantic category label in the pixel set is calculated. The area ratios of various semantic category labels are arranged into a vector in a preset order, and the ratio of the geometric center coordinates of the bounding box of the construction worker to the image size is appended to the end of the vector as the position normalization coordinates, thus generating the regional semantic feature vector of the construction worker's current position.

6. The method as described in claim 5, characterized in that, The semantic category labels include driving lane area, shoulder area, construction operation area, material storage area, machinery operation area, and edge protection area.

7. The method as described in claim 1, characterized in that, The skeleton feature vector and the appearance feature vector are fused across modalities. Based on the fused feature vector, the current behavior of the construction worker is classified and a category label is generated. Specifically, this includes: The skeleton feature vector is mapped to a skeleton semantic embedding vector through a first fully connected network, and the appearance feature vector is mapped to an appearance semantic embedding vector through a second fully connected network. A cross-modal attention mechanism is used to interactively weight and fuse the skeleton semantic embedding vector and the appearance semantic embedding vector. Specifically, the skeleton semantic embedding vector is used as the query vector and the appearance semantic embedding vector is used as the key vector to calculate the attention weights and obtain the fused feature vector. The fused feature vector is input into the behavior classifier, which outputs the probability that the current behavior of the construction worker belongs to each category in the preset behavior category set. The category with the highest probability is taken as the category label of the current behavior of the construction worker.

8. The method as described in claim 7, characterized in that, The preset set of behavior categories includes normal walking, standing work, bending over work, climbing, carrying heavy objects, operating hand tools, driving machinery, running, waving to warn, and illegally climbing over guardrails.

9. The method as described in claim 1, characterized in that, Determining the spatial hazard coefficient of the construction worker's current behavior based on the category label and the regional semantic feature vector, and then generating the final safety monitoring alarm signal for the current behavior, specifically includes: A mapping table between behavior categories and risk levels is pre-established, and the basic risk coefficient of the current behavior is obtained by querying the mapping table according to the category label; The area category of the construction worker is determined based on the area ratio of various semantic category labels in the area semantic feature vector. If the construction worker is at the intersection of multiple area categories, the area category with the largest area ratio is taken as the current area category. The basic hazard coefficient is corrected based on the area category. When the current area category does not match the recommended work area corresponding to the category label, a penalty coefficient is added to the basic hazard coefficient to obtain the spatial hazard coefficient of the construction worker's current behavior. The final safety monitoring alarm signal for the current behavior is generated based on the spatial hazard coefficient.

10. The method as described in claim 9, characterized in that, The generation of the final safety monitoring alarm signal for the current behavior based on the aforementioned spatial hazard factor specifically includes: The shortest pixel distance between the current center of gravity of the construction worker and the boundary of the edge protection area in the semantic segmentation image is calculated based on the skeleton feature vector of the construction worker, and the shortest pixel distance is converted into the actual physical distance based on the camera calibration parameters. When the actual physical distance is less than the preset safe distance threshold and the category label belongs to the high-risk behavior category, the spatial hazard coefficient is amplified a second time according to the ratio of the actual physical distance to the preset safe distance threshold to obtain the final hazard coefficient; When the final hazard factor exceeds the preset alarm threshold, a final safety monitoring alarm signal is generated, which includes the location information of the construction personnel, behavior category labels, and hazard factor level.

Images