Tunnel construction monitoring analysis method and system based on AI video monitoring

Abstract

The invention provides a tunnel construction monitoring analysis method and system based on AI video monitoring, and relates to the technical field of artificial intelligence. A tunnel construction original video stream data set is acquired through a multi-view video acquisition device, and a spatio-temporal feature sequence is generated through spatio-temporal feature fusion processing; the spatiotemporal feature unit comprises a pixel region movement track, a personnel wearing identifier and a construction equipment contour feature, then extracting dynamic features of the spatiotemporal feature sequence to obtain a dynamic spatiotemporal feature set, and executing construction personnel statistical processing based on the dynamic spatiotemporal feature set; generating a hole entering personnel quantity statistical result, a personnel distribution statistical result of each construction part and a personnel identity matching result, performing work efficiency comparative analysis according to the personnel statistical result and a preset construction progress plan, and generating a construction progress deviation index and resource allocation optimization strategy; and when the progress deviation index exceeds the first threshold value, early warning is triggered, and a strategy is output to the management terminal, so that effective monitoring and management of tunnel construction are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a tunnel construction monitoring and analysis method and system based on AI video monitoring. Background Art

[0002] In the field of tunnel construction, with the continuous expansion of project scale and the increasing complexity of construction, more stringent requirements are put forward for real-time monitoring and efficient management of the construction process. Traditional tunnel construction monitoring methods mainly rely on manual inspections, single-point video monitoring and simple sensor data collection, but these methods have many limitations.

[0003] Manual inspections are not only inefficient and difficult to cover the construction site in all aspects, but are also easily affected by human factors, such as the experience and attention of inspectors, which leads to greater subjectivity and uncertainty in monitoring results. Although single-point video monitoring can provide images of a specific area, it cannot fully capture the dynamic changes in the construction scene due to its single viewing angle. There are a large number of monitoring blind spots, making it difficult to accurately and comprehensively analyze the activity trajectories of construction personnel, the operation of construction equipment, and the overall state of construction surface operations. Simple sensor data collection methods, such as only collecting environmental parameters such as temperature and humidity, or only recording the operating time of construction equipment, have limited data dimensions and cannot reflect the complex spatiotemporal relationships and dynamic characteristics of the construction process, making it difficult to meet the needs of refined management of the construction process.

[0004] In terms of personnel management, traditional monitoring methods make it difficult to accurately count the number of people entering the cave, grasp the distribution of people in each construction site in real time, and cannot quickly and accurately match the identities of people. This makes it difficult to obtain accurate personnel information in a timely manner when a safety accident or emergency occurs, which brings great difficulties to rescue and management work. At the same time, due to the lack of effective data support, the evaluation of construction progress mainly relies on the experience and subjective judgment of managers, making it difficult to accurately quantify the deviation of construction progress, and it is even more impossible to dynamically adjust resource allocation according to real-time construction conditions, resulting in low construction efficiency and serious waste of resources. Summary of the invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a tunnel construction monitoring and analysis method based on AI video monitoring, the method comprising: The original video stream data set of the construction scene is acquired in real time by a multi-view video acquisition device deployed in the tunnel construction area, wherein the original video stream data set includes a plurality of spatially continuous frame sequences aligned with time stamps; Performing spatiotemporal feature fusion processing on the original video stream data set to generate a spatiotemporal feature sequence, wherein each spatiotemporal feature unit in the spatiotemporal feature sequence is composed of a pixel region motion trajectory feature, a personnel wear identification feature, and a construction equipment contour feature of a corresponding frame sequence; Performing dynamic feature extraction processing on the spatiotemporal feature sequence to obtain a dynamic spatiotemporal feature set, wherein the dynamic spatiotemporal feature set includes construction personnel movement path features, construction equipment operation frequency features, and construction surface operation status features; Execute construction personnel statistics processing based on the dynamic spatiotemporal feature set to generate statistics on the number of personnel entering the cave, statistics on personnel distribution at each construction site, and personnel identity matching results, wherein the personnel identity matching results are determined by comparing identification features worn by the personnel with identification templates in a pre-registered identity database; According to the statistical results of the number of people entering the cave, the statistical results of the personnel distribution and the preset construction progress plan, a comparative analysis of work efficiency is performed to generate a construction progress deviation indicator and a resource allocation optimization strategy. When it is detected that the construction progress deviation indicator exceeds a first threshold, a construction early warning instruction is triggered and the resource allocation optimization strategy is output to the management terminal.

[0006] On the other hand, an embodiment of the present invention also provides a tunnel construction monitoring and analysis system based on AI video surveillance, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0007] Based on the above aspects, the embodiment of the present invention converts the spatial continuous frame sequence scattered in the multi-view video stream into a temporal and spatial feature sequence with temporal correlation through spatiotemporal feature fusion processing, which not only breaks through the field of view limitation of traditional single-view monitoring, but also builds a global semantic model of the construction scene through the organic integration of pixel area motion trajectory, personnel wear identification and construction equipment contour features, so that the spatiotemporal evolution law of construction activities can be fully presented. On this basis, the dynamic feature extraction processing further mines key dynamic information such as the movement path of construction personnel, the frequency of equipment operation and the operation status, providing multi-dimensional data support for the management and control of the construction process. Through the statistical processing of construction personnel and the comparative analysis of work efficiency, the number, distribution and identity matching results of personnel entering the tunnel can be output in real time, and the quantitative progress deviation index and resource allocation optimization strategy can be generated in combination with the preset progress plan, so as to trigger the early warning in time and push the targeted adjustment plan when the construction progress is abnormal. Therefore, the deep integration of spatiotemporal feature modeling, dynamic behavior recognition and construction management decision-making not only improves the intelligent level of tunnel construction monitoring, but also significantly improves the construction efficiency and safety through data-driven resource allocation optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the execution flow of the tunnel construction monitoring and analysis method based on AI video monitoring provided in an embodiment of the present invention.

[0009] Figure 2 It is a schematic diagram of exemplary hardware and software components of a tunnel construction monitoring and analysis system based on AI video monitoring provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0010] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 It is a flow chart of a tunnel construction monitoring and analysis method based on AI video monitoring provided by an embodiment of the present invention. The tunnel construction monitoring and analysis method based on AI video monitoring is introduced in detail below.

[0011] Step S110: acquiring an original video stream data set of the construction scene in real time through a multi-view video acquisition device deployed in the tunnel construction area, wherein the original video stream data set includes a plurality of spatially continuous frame sequences aligned with time stamps.

[0012] In this embodiment, in order to comprehensively and accurately monitor the tunnel construction scene, multi-view video acquisition equipment can be deployed in the tunnel construction area. Specifically, cameras are set at different key locations in the tunnel area, such as a camera at the entrance of the tunnel, which is installed at a height of 5 meters from the ground and tilted downward at an angle of 30 degrees to clearly capture the personnel and equipment entering the tunnel; a camera is set every 20 meters in the middle of the tunnel, with an installation height of 4 meters from the ground and a horizontal viewing angle of 120 degrees to ensure full coverage of the construction area; a camera is also set at the exit of the tunnel, with an installation height and angle similar to the camera at the entrance.

[0013] The video stream data collected by each camera can be transmitted to the data processing center in real time through the network. Since there may be slight differences in the startup time of different cameras, in order to ensure the accuracy of subsequent processing, the video stream data collected by each camera can be timestamped. Specifically, timestamp information is added to each video stream data in the data processing center, and then the timestamps of different video stream data are compared, and the timestamp of one camera is used as the benchmark to adjust the time of the video stream data of other cameras. For example, assuming that the timestamp of the camera at the entrance of the tunnel is used as the benchmark, when it is found that the timestamp of the video stream data of a camera in the middle is 0.1 seconds later than that of the camera at the entrance, the timestamps of all video frames of the camera in the middle are uniformly advanced by 0.1 seconds. After such a timestamp alignment operation, a set of original video stream data containing multiple spatially continuous frame sequences is obtained, each of which represents a continuous video frame at a specific perspective, and each spatially continuous frame sequence is aligned in time.

[0014] Step S120: performing spatiotemporal feature fusion processing on the original video stream data set to generate a spatiotemporal feature sequence, wherein each spatiotemporal feature unit in the spatiotemporal feature sequence is composed of pixel area motion trajectory features, personnel wear identification features and construction equipment contour features of the corresponding frame sequence.

[0015] In this embodiment, the original video stream data set is subjected to spatiotemporal feature fusion processing, and this process includes multiple sub-steps.

[0016] Step S121: performing motion trajectory segmentation processing on each spatial continuous frame sequence in the original video stream data set to obtain pixel area motion trajectory features of each construction worker, wherein the pixel area motion trajectory features include independent motion trajectory segments and corresponding trajectory bounding box coordinates.

[0017] In this step, for each spatial continuous frame sequence, a series of processing is performed to obtain the pixel area motion trajectory characteristics of the construction workers. The details are as follows: Step S1211: grayscale each frame image in the spatial continuous frame sequence, convert the color image into a grayscale image, and calculate the pixel brightness difference matrix between adjacent frames based on a preset adjacent time stamp interval.

[0018] For example, taking a spatial continuous frame sequence of a certain perspective as an example, for each color image in the spatial continuous frame sequence, the weighted average method is used to convert it into a grayscale image. Specifically, for each pixel in the color image, its RGB values ​​are R, G, and B respectively, and the corresponding grayscale value is calculated according to the formula grayscale value = 0.299*R+0.587*G+0.114*B. For example, the RGB values ​​of a pixel are (200, 150, 100), then its grayscale value is 0.299*200+0.587*150+0.114*100=153.55, which is rounded to 154. After such processing, the entire color image is converted into a grayscale image.

[0019] Then, the pixel brightness difference matrix between adjacent frames is calculated based on the preset adjacent timestamp interval (assumed to be 0.033 seconds, corresponding to a frame rate of 30 frames per second). Specifically, for two adjacent grayscale image frames, their brightness difference is calculated pixel by pixel. For example, in the nth grayscale image, the grayscale value of a pixel is 120, and in the n+1th grayscale image, the grayscale value of the pixel is 130, then their brightness difference is 130-120=10. In this way, a pixel brightness difference matrix with the same size as the image is obtained.

[0020] Step S1212: extracting a dynamic background model according to the pixel brightness difference matrix, separating the foreground motion target area by background difference method, performing morphological closing operation on the foreground motion target area, eliminating noise interference and filling the internal holes of the target, and generating a connected domain labeling map.

[0021] In this embodiment, a Gaussian mixture model (GMM) can be used for modeling. For each pixel, multiple Gaussian distributions are used to fit the change in its brightness value. For example, for a pixel, it is assumed that three Gaussian distributions are used for fitting, and each Gaussian distribution has its own mean, variance, and weight. In the initial stage, these parameters are randomly initialized, and then these parameters are adjusted to adapt to the change in pixel brightness value by continuously updating the data. After a period of training, a stable Gaussian mixture model is obtained, which is used as a dynamic background model.

[0022] Next, the background difference method is used to separate the foreground moving target area. For each frame of the grayscale image, it is compared with the dynamic background model and the brightness difference of each pixel is calculated. If the difference exceeds the preset threshold (assuming it is 20), the pixel is considered to belong to the foreground moving target area. For example, the grayscale value of a pixel in the current frame is 150, the corresponding mean value in the dynamic background model is 120, and the difference is 30, which exceeds the threshold of 20, then the pixel is marked as part of the foreground moving target area.

[0023] Then, the separated foreground moving target area is processed by morphological closing operation. For example, a dilation operation is first performed, and a 3x3 structure element is used to dilate the foreground moving target area. Specifically, the center of the structure element is aligned with each pixel of the foreground moving target area in turn. If there is a pixel in the area covered by the structure element that belongs to the foreground moving target area, the pixel corresponding to the center of the structure element is marked as the foreground moving target area. Then an erosion operation is performed, and a 3x3 structure element is also used to align the center of the structure element with each pixel of the foreground moving target area in turn. If all the pixels in the area covered by the structure element belong to the foreground moving target area, the pixel corresponding to the center of the structure element is retained as the foreground moving target area. Through the combination of dilation and erosion operations, noise interference is eliminated and the internal holes of the target are filled to generate a connected domain labeling map.

[0024] Step S1213: Fit the minimum bounding rectangle of each foreground moving target based on the connected domain marker map, calculate the centroid coordinates and aspect ratio of each bounding rectangle, map the centroid coordinates to the global coordinate system, and record the pixel area of ​​each bounding rectangle.

[0025] In this embodiment, the rotating caliper algorithm can be used to calculate the minimum circumscribed rectangle. For a connected domain, first find its boundary points, and then find the minimum rectangle that can enclose the connected domain by continuously rotating the tangent of the boundary points. For example, for an irregular connected domain, a rectangle that can just enclose it can be found by the rotating caliper algorithm, and the four sides of the rectangle are tangent to the boundary points of the connected domain.

[0026] Calculate the centroid coordinates of each bounding rectangle. The centroid coordinates are calculated by adding the coordinates of the four vertices of the bounding rectangle and then dividing by 4. For example, if the coordinates of the four vertices of a bounding rectangle are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), then its centroid coordinates are ((x1+x2+x3+x4) / 4, (y1+y2+y3+y4) / 4). At the same time, calculate the length-to-width ratio of the bounding rectangle, that is, the length divided by the width.

[0027] Map the centroid coordinates to the global coordinate system. Since each camera is installed at a different position and angle, there is a certain conversion relationship between the image coordinate system and the global coordinate system. By measuring the installation position and angle of the camera and the actual size of the tunnel construction area in advance, establish a conversion matrix between the image coordinate system and the global coordinate system. For example, assuming that the centroid coordinates of a circumscribed rectangle in the image coordinate system are (x_img, y_img), they can be converted to coordinates in the global coordinate system (x_global, y_global) through the conversion matrix.

[0028] Finally, record the pixel area of ​​each bounding rectangle, which is the length multiplied by the width. For example, if the length of a bounding rectangle is 50 pixels and the width is 30 pixels, then its pixel area is 50*30=1500 pixels.

[0029] Step S1214: Use a multi-target tracking algorithm to associate the centroid coordinates of consecutive timestamps with motion trajectories, calculate the Euclidean distance of the centroid coordinates of the same target at adjacent timestamps, and when the Euclidean distance is less than a preset distance threshold and the length-to-width ratio change is less than a preset ratio threshold, it is determined to be a motion trajectory segment of the same target.

[0030] In this embodiment, a multi-target tracking algorithm (such as a Kalman filter algorithm) can be used to associate the centroid coordinates of consecutive timestamps with motion trajectories. The Kalman filter algorithm tracks the motion trajectory of the target through two steps: prediction and update. In the prediction step, the position of the target at the next timestamp is predicted based on the historical motion information of the target; in the update step, the prediction result is corrected based on the actual measured target position.

[0031] Specifically, for the centroid coordinates of adjacent timestamps, the Euclidean distance between them is calculated. The Euclidean distance is calculated by taking the square root of the sum of the squares of the differences between the horizontal and vertical coordinates of the two centroid coordinates. For example, at the nth timestamp, the centroid coordinates of an object are (x_n, y_n), and at the n+1th timestamp, the centroid coordinates of the object are (x_n+1, y_n+1), then the Euclidean distance between them is the square root of ((x_n+1-x_n)^2+(y_n+1-y_n)^2).

[0032] At the same time, the change range of the length-width ratio of the same target bounding rectangle at adjacent timestamps is calculated. Assuming that at the nth timestamp, the length-width ratio of the target bounding rectangle is r_n, and at the n+1th timestamp, the length-width ratio of the target bounding rectangle is r_n+1, then the change range of the length-width ratio is |r_n+1-r_n| / r_n.

[0033] When the Euclidean distance is less than the preset distance threshold (assuming 15 pixels) and the change in aspect ratio is less than the preset ratio threshold (assuming 0.1), it is determined to be a motion trajectory segment of the same target. For example, at the nth timestamp and the n+1th timestamp, the Euclidean distance of the centroid coordinates of a target is 10 pixels, which is less than the preset distance threshold of 15 pixels, and the change in aspect ratio is 0.05, which is less than the preset ratio threshold of 0.1, then the targets at these two timestamps are determined to be motion trajectory segments of the same target.

[0034] Step S1215: merging the motion trajectory segments of the same target at consecutive timestamps to generate independent motion trajectory segments, and calculating the trajectory bounding box coordinates according to the global coordinate sequence of the circumscribed rectangle, wherein the trajectory bounding box coordinates are the minimum convex polygon vertex coordinate set containing all circumscribed rectangle vertices.

[0035] In this embodiment, all motion trajectory segments belonging to the same target can be connected in time order. For example, if a target has a motion trajectory segment at the 1st to 5th timestamps and another motion trajectory segment at the 6th to 10th timestamps, and if it is determined that the two trajectory segments belong to the same target, they are merged into an independent motion trajectory segment.

[0036] The coordinates of the trajectory bounding box are calculated based on the global coordinate sequence of the bounding rectangle. First, the vertex coordinates of the bounding rectangle of the same target at all timestamps are collected, and then the convex hull algorithm (such as the Graham scan algorithm) is used to calculate the minimum convex polygon containing all these vertices. The basic idea of ​​the Graham scan algorithm is to first find the point with the smallest ordinate among all vertices, then sort the other vertices according to the polar angle, and finally traverse the sorted vertices in turn, and use the cross product to determine whether some vertices need to be deleted, and finally obtain the minimum convex polygon. For example, for a target's set of vertex coordinates of the bounding rectangle at multiple timestamps, a minimum convex polygon can be obtained through the Graham scan algorithm, and the vertex coordinates of the polygon are the coordinates of the trajectory bounding box.

[0037] Step S1216: Verify the integrity of the independent motion trajectory segment, count the coverage time of the trajectory bounding box coordinates on the time axis, and when the coverage time exceeds a preset time threshold and the pixel area change rate is lower than a preset area change threshold, retain the independent motion trajectory segment and the corresponding trajectory bounding box coordinates.

[0038] In this embodiment, the coverage time of the trajectory bounding box coordinates on the time axis is counted, that is, the time difference from the start timestamp to the end timestamp of the independent motion trajectory segment. For example, if the start timestamp of an independent motion trajectory segment is the 10th second and the end timestamp is the 20th second, then its coverage time is 20-10=10 seconds.

[0039] At the same time, the pixel area change rate is calculated. For the pixel area of ​​the circumscribed rectangle at each time stamp in the independent motion trajectory segment, its maximum and minimum values ​​are recorded, and the pixel area change rate is calculated as (maximum value - minimum value) / average value. For example, in an independent motion trajectory segment, the maximum value of the circumscribed rectangle pixel area is 2000 pixels, the minimum value is 1500 pixels, and the average value is (2000+1500) / 2=1750 pixels, then the pixel area change rate is (2000-1500) / 1750=0.286.

[0040] When the coverage duration exceeds the preset duration threshold (assuming 8 seconds) and the pixel area change rate is lower than the preset area change threshold (assuming 0.3), the independent motion trajectory segment and the corresponding trajectory bounding box coordinates are retained. For example, if the coverage duration of an independent motion trajectory segment is 12 seconds, which exceeds the preset duration threshold of 8 seconds, and the pixel area change rate is 0.2, which is lower than the preset area change threshold of 0.3, the independent motion trajectory segment and the corresponding trajectory bounding box coordinates are retained.

[0041] Step S122: extracting a color space distribution histogram in the pixel area corresponding to the coordinates of the trajectory bounding box, classifying the color space distribution histogram based on a pre-trained wear identification classification model to obtain personnel wear identification features, wherein the wear identification classification model is trained by a training data set labeled with safety helmets, reflective clothing, and work type numbers.

[0042] In this embodiment, first, the image of the pixel area is converted to the HSV color space, because the HSV color space can better reflect the essential characteristics of the color. Then, the three channels of H (hue), S (saturation), and V (brightness) are quantized respectively, assuming that the H channel is quantized into 18 intervals, the S channel is quantized into 4 intervals, and the V channel is quantized into 3 intervals. For each pixel point in the pixel area, the interval to which it belongs is determined according to its HSV value, and then the number of pixels in each interval is counted to obtain a color space distribution histogram. For example, for a pixel point, its HSV value is (30, 0.5, 0.8), the H channel belongs to the third interval after quantization, the S channel belongs to the second interval after quantization, and the V channel belongs to the third interval after quantization, then a count is added in the corresponding interval.

[0043] Then, the color space distribution histogram is classified based on the pre-trained wear mark classification model. The wear mark classification model is trained with a training data set labeled with safety helmets, reflective clothing, and work type numbers. The training data set contains a large number of image samples, each of which is labeled with information such as whether it contains a safety helmet, reflective clothing, and work type number. During the training process, a convolutional neural network (CNN) is used as the model architecture, and the parameters of the convolutional neural network are continuously adjusted through the back-propagation algorithm, so that the convolutional neural network can accurately identify the wear marks in the image.

[0044] Next, the extracted color space distribution histogram is input into the pre-trained wear identification classification model, which will output the wear identification features of the person in the pixel area, such as whether he is wearing a helmet, whether he is wearing reflective clothing, and the type of work number. For example, the output result of the wear identification classification model shows that the person in the pixel area is wearing a red helmet, a yellow reflective clothing, and the type of work number is 007.

[0045] Step S123: Perform edge enhancement processing on the construction equipment contour in the spatial continuous frame sequence, extract the polygon vertex coordinate set of the enhanced contour, perform shape matching based on the polygon vertex coordinate set and the pre-stored equipment template library, and determine the type of construction equipment and the corresponding operating status.

[0046] In this embodiment, edge enhancement processing is performed on the outline of the construction equipment in the spatial continuous frame sequence, and the specific steps are as follows: Step S1231: convert each frame image of the spatial continuous frame sequence into a grayscale image, and perform binary segmentation on the grayscale image based on a preset fixed grayscale threshold to obtain an initial binary image.

[0047] Each color image of the spatial continuous frame sequence is converted into a grayscale image using the same method as in step S1211, that is, using the weighted average method. For example, for a color image, the RGB value of each pixel is converted into a grayscale value according to the formula grayscale value = 0.299*R+0.587*G+0.114*B to obtain a grayscale image.

[0048] The grayscale image is binarized based on a preset fixed grayscale threshold (assuming it is 128). For each pixel in the grayscale image, if its grayscale value is greater than or equal to 128, its value is set to 255 (white); if its grayscale value is less than 128, its value is set to 0 (black). Through this process, the initial binary image is obtained.

[0049] Step S1232: Perform Gaussian filtering on the initial binary image to generate a smoothed binary image after eliminating high-frequency noise interference, and use the Canny edge detection algorithm to extract edge pixels from the smoothed binary image to generate an initial edge pixel set.

[0050] Perform Gaussian filtering on the initial binary image to eliminate high-frequency noise interference. Gaussian filtering is a linear smoothing filter that is implemented by convolving a two-dimensional Gaussian kernel with the image. Assume that a 3x3 Gaussian kernel is used, and its coefficients are [[1, 2, 1], [2, 4, 2], [1, 2, 1]] / 16. Convolve the Gaussian kernel with each pixel of the initial binary image in turn to obtain a smoothed binary image.

[0051] The Canny edge detection algorithm is used to extract edge pixels from the smoothed binary image. The Canny edge detection algorithm includes the following steps: first, Gaussian smoothing is performed. Since Gaussian filtering has been performed before, it can be omitted here; then the gradient amplitude and direction of the image are calculated. The gradient of the image in the x and y directions are calculated by the Sobel operator respectively, and then the gradient amplitude is calculated according to the square root of the formula gradient amplitude = (square of the gradient in the x direction + square of the gradient in the y direction), and the gradient direction is calculated according to the formula gradient direction = arctan (gradient in the y direction / gradient in the x direction); then non-maximum suppression is performed. In the gradient direction, the gradient amplitude of each pixel is compared with the gradient amplitude of the adjacent pixel points. If the gradient amplitude of the pixel point is not the local maximum, its value is set to 0; finally, double threshold detection and edge connection are performed, and two thresholds are set (assuming that the low threshold is 50 and the high threshold is 150). If the gradient amplitude of a pixel point is greater than the high threshold, it is marked as a strong edge point; if the gradient amplitude of a pixel point is greater than the low threshold and is adjacent to a strong edge point, it is marked as a weak edge point, and the initial edge pixel set is obtained by connecting the strong edge point and the weak edge point. For example, in a smoothed binary image, if the gradient amplitude of a pixel is 180, which is greater than the high threshold of 150, it will be marked as a strong edge point; if the gradient amplitude of another pixel is 80, which is greater than the low threshold of 50 and is adjacent to a strong edge point, it will be marked as a weak edge point.

[0052] Step S1233: performing a morphological gradient operation on the initial edge pixel set, enhancing edge continuity by calculating pixel differences between the dilation operation and the erosion operation, and generating an edge enhanced image.

[0053] In this embodiment, a dilation operation is first performed, and a 3x3 structure element can be used to dilate the initial edge pixel set. The center of the structure element is aligned with each pixel of the initial edge pixel set in turn. If there is a pixel in the area covered by the structure element that belongs to the initial edge pixel set, the pixel corresponding to the center of the structure element is marked as an edge pixel. Then an erosion operation is performed, and the 3x3 structure element is also used to align the center of the structure element with each pixel of the initial edge pixel set in turn. If all pixels in the area covered by the structure element belong to the initial edge pixel set, the pixel corresponding to the center of the structure element is retained as an edge pixel. The difference between the pixel set obtained by the dilation operation and the pixel set obtained by the erosion operation is the morphological gradient. In this way, the continuity of the edge is enhanced to generate an edge enhanced image. For example, in the dilation operation, a pixel that was not originally in the edge pixel set is marked as an edge pixel after the dilation operation because its adjacent pixel points belong to the edge pixel set; while in the erosion operation, some edge pixels are removed because their adjacent pixel points do not all belong to the edge pixel set. By calculating the difference between the two, the edge is made more continuous.

[0054] Step S1234: performing contour tracking processing based on the edge enhanced image, traversing adjacent edge pixel points in a clockwise direction to generate a closed contour point set, and eliminating non-closed contour point sets whose pixel number is less than a preset contour point number threshold.

[0055] In this embodiment, an edge pixel point that has not been visited can be selected from the edge enhanced image as the starting point, and then its adjacent edge pixel points can be traversed in a clockwise direction. For example, starting from the starting point, first check the pixel point above it. If the pixel point is an edge pixel point and has not been visited, then continue to traverse it as the next point; if the pixel point above is not an edge pixel point, then check the pixel point to the upper right, and so on. By continuously traversing adjacent edge pixel points until returning to the starting point, a closed contour point set is formed.

[0056] During the traversal process, the number of pixels of each closed contour point set is recorded. A threshold value of the number of contour points is preset (assuming it is 20). If the number of pixels of a closed contour point set is less than 20, it will be removed. For example, if a closed contour point set contains only 15 pixels, which is less than the preset threshold value of 20, the closed contour point set will be removed from the result.

[0057] Step S1235: Execute the Douglas-Peucker polygon approximation algorithm on the retained closed contour point set, calculate the maximum vertical distance error between each closed contour point set and the approximation polygon, and when the maximum vertical distance error is less than a preset shape error threshold, extract the vertex coordinates of the approximation polygon as a set of candidate vertex coordinates.

[0058] In this embodiment, the basic idea of ​​the Douglas-Peucker polygon approximation algorithm is to first connect the starting point and the end point of the closed contour point set to form a straight line, and then calculate the vertical distances from all points in the closed contour point set to the straight line to find the point with the largest vertical distance. If the maximum vertical distance is greater than a preset shape error threshold (assuming it is 3 pixels), then the point is used as a new segmentation point, the closed contour point set is divided into two parts, and the above process is repeated for the two parts respectively; if the maximum vertical distance is less than or equal to the preset shape error threshold, then the closed contour point set of that part is approximately replaced by the straight line connecting the starting point and the end point.

[0059] For example, for a closed contour point set, the starting point is A and the end point is B. The vertical distances from all points in the point set to the straight line AB are calculated. Assuming that the point corresponding to the maximum vertical distance is C, and the vertical distance is 4 pixels, which is greater than the preset shape error threshold of 3 pixels, point C is used as the segmentation point, and the closed contour point set is divided into AC segment and CB segment, and the two segments are processed separately.

[0060] Through continuous iteration, the approximation polygon is finally obtained. The maximum vertical distance error between each closed contour point set and the approximation polygon is calculated. If the error is less than the preset shape error threshold, the vertex coordinates of the approximation polygon are extracted as the candidate vertex coordinate set.

[0061] Step S1236: Calculate the spacing between vertex sampling points based on the candidate vertex coordinate set, count the pixel spacing between adjacent vertices in the image coordinate system, convert the pixel spacing into a proportional distance based on the actual construction equipment size, eliminate redundant vertices whose proportional distance is less than a preset minimum spacing threshold, and generate an optimized vertex coordinate set.

[0062] In this embodiment, for adjacent vertices in the candidate vertex coordinate set, their pixel spacing in the image coordinate system is calculated. For example, the coordinates of adjacent vertex A are (x1, y1), and the coordinates of vertex B are (x2, y2), then their pixel spacing is the square root of ((x2-x1)^2+(y2-y1)^2).

[0063] Convert pixel spacing to proportional distance based on the actual size of construction machinery. Measure the actual size of the construction machinery in advance. For example, the actual length of an excavator is 5 meters. In the image, the corresponding pixel length of the excavator is 500 pixels. The ratio of pixels to actual size is 5 meters / 500 pixels = 0.01 meters / pixel. Multiply the pixel spacing of adjacent vertices by this ratio to get the proportional distance based on the actual size of the construction machinery.

[0064] A minimum spacing threshold is preset (assuming it is 0.1 meters). If the proportional distance between adjacent vertices is less than 0.1 meters, the vertex is considered to be a redundant vertex and is removed. For example, if the pixel spacing between adjacent vertices is 8 pixels, the corresponding proportional distance is 8 pixels * 0.01 meters / pixel = 0.08 meters, which is less than the preset minimum spacing threshold of 0.1 meters. In this case, the vertex is removed from the candidate vertex coordinate set to generate an optimized vertex coordinate set.

[0065] Step S1237: normalize the optimized vertex coordinate set, convert the vertex coordinates into a normalized coordinate set based on the actual space coordinate system according to the ratio coefficient between the image resolution and the actual size of the construction equipment, and generate a polygon vertex coordinate set of the enhanced contour.

[0066] In this embodiment, the conversion can be performed based on the ratio coefficient between the image resolution and the actual size of the construction machine. Assuming that the image resolution is 1920x1080 pixels, the actual length of the construction machine is 5 meters, and the width is 2 meters, and the pixel length corresponding to the construction machine in the image is 500 pixels, and the width is 200 pixels, then the ratio coefficient in the length direction is 5 meters / 500 pixels=0.01 meters / pixel, and the ratio coefficient in the width direction is 2 meters / 200 pixels=0.01 meters / pixel.

[0067] For each vertex coordinate in the optimized vertex coordinate set, multiply its horizontal coordinate by the scale factor in the length direction and its vertical coordinate by the scale factor in the width direction to convert the vertex coordinate into a coordinate based on the real space coordinate system. For example, the image coordinates of a vertex are (200, 150) pixels, and the converted real space coordinates are (200 pixels * 0.01 meters / pixel, 150 pixels * 0.01 meters / pixel) = (2 meters, 1.5 meters).

[0068] Through such normalization processing, the vertex coordinates are converted into a normalized coordinate set based on the actual space coordinate system, and a polygon vertex coordinate set of the enhanced contour is generated.

[0069] Shape matching is performed based on the polygon vertex coordinate set of the generated enhanced contour and the pre-stored equipment template library. The pre-stored equipment template library contains standard polygon vertex coordinate sets of various construction equipment, as well as the corresponding equipment type and operation status information. A shape matching algorithm (such as a matching algorithm based on Hausdorff distance) is used to calculate the similarity between the polygon vertex coordinate set of the enhanced contour and each template in the template library. Hausdorff distance refers to the maximum one-way distance between two point sets, that is, the maximum value of the minimum distance from each point in one point set to another point set. For example, the Hausdorff distance between the polygon vertex coordinate set A of the enhanced contour and template B in the template library is calculated. If the distance is less than the preset similarity threshold (assuming it is 0.2 meters), the two are considered to match, and the type of construction equipment and the corresponding operation status are determined based on the information corresponding to template B.

[0070] Step S124: performing feature splicing on the motion trajectory features of the pixel area, the identification features of the personnel wearing and the type of the construction equipment at the same timestamp to generate a spatiotemporal feature unit, and arranging all the spatiotemporal feature units in chronological order to obtain a spatiotemporal feature sequence.

[0071] In this embodiment, the motion trajectory features of the pixel area include independent motion trajectory segments and corresponding trajectory bounding box coordinates, the personnel wear identification features include whether they wear a helmet, whether they wear reflective clothing, and the type of work number, etc., and the construction equipment types include specific types such as excavators and loaders. These features are spliced ​​in a set order, for example, the trajectory bounding box coordinates are spliced ​​first, then the personnel wear identification features are spliced, and finally the construction equipment types are spliced.

[0072] For example, at a certain moment, the coordinates of a construction worker's trajectory bounding box are [(100, 200), (150, 250), (200, 200), (150, 150)], the worker's wearing identification features are "red safety helmet, yellow reflective clothing, work type number 007", and the type of construction equipment is "excavator". These features are spliced ​​together to form a spatiotemporal feature unit.

[0073] Arrange all spatiotemporal feature units in chronological order, and arrange the spatiotemporal feature units corresponding to each timestamp in sequence to obtain a spatiotemporal feature sequence. For example, arrange the spatiotemporal feature units at different times in order from early to late to form a sequence containing multiple spatiotemporal feature units.

[0074] Step S130: performing dynamic feature extraction processing on the spatiotemporal feature sequence to obtain a dynamic spatiotemporal feature set, wherein the dynamic spatiotemporal feature set includes construction personnel movement path features, construction equipment operation frequency features, and construction surface operation status features.

[0075] A dynamic feature extraction model including convolutional neural network, recurrent neural network and graph convolutional network is constructed, and the spatiotemporal feature sequence is input into the dynamic feature extraction model for dynamic feature extraction processing. The specific steps are as follows: Step S131: construct a dynamic feature extraction model including a convolutional neural network, a recurrent neural network and a graph convolutional network, and input the spatiotemporal feature sequence into the dynamic feature extraction model.

[0076] In this embodiment, the dynamic feature extraction model is composed of a convolutional neural network (CNN), a recurrent neural network (RNN) and a graph convolutional network (GCN). The convolutional neural network is used to extract the local features of each spatiotemporal feature unit in the spatiotemporal feature sequence, the recurrent neural network is used to capture the feature dependencies of continuous timestamps, and the graph convolutional network is used to model the spatial relationship of the motion trajectory features of the pixel area.

[0077] The spatiotemporal feature sequence is input into the dynamic feature extraction model to start feature extraction processing. For example, the spatiotemporal feature sequence containing multiple spatiotemporal feature units is sequentially input into the input layer of the convolutional neural network.

[0078] Step S132: Perform local feature convolution operation on each spatiotemporal feature unit through the convolutional neural network to extract texture change gradient features and equipment operation action segment features of the construction surface working area.

[0079] In this embodiment, the convolutional neural network includes multiple convolutional layers and pooling layers. In the convolutional layer, a convolution kernel is used to perform a local feature convolution operation on each spatiotemporal feature unit. The convolution kernel is a small matrix, such as a 3x3 matrix. The convolution kernel is slid on the spatiotemporal feature unit, and the area covered by the convolution kernel is weighted and summed to obtain a new eigenvalue. By continuously sliding the convolution kernel, a new set of feature maps is obtained.

[0080] For example, for a spatiotemporal feature unit, a 3x3 convolution kernel is used for convolution operation, and the coefficients of the convolution kernel are [[1, 0, -1], [2, 0, -2], [1, 0, -1]]. The center of the convolution kernel is aligned with each pixel of the spatiotemporal feature unit in turn, and the weighted sum is taken for the area covered by the convolution kernel. Assuming that the pixel values ​​of the area covered by the convolution kernel are [[10, 20, 30], [40, 50, 60], [70, 80, 90]], the convolution result is 1*10+0*20+(-1)*30+2*40+0*50+(-2)*60+1*70+0*80+(-1)*90=-10.

[0081] Through the alternating operation of multiple convolutional layers and pooling layers, the texture change gradient features of the construction surface working area and the characteristics of the machine operation action fragments are extracted. The pooling layer is used to reduce the dimension of the feature map. Commonly used pooling methods include maximum pooling and average pooling. For example, using a 2x2 maximum pooling kernel, the feature map is divided into multiple 2x2 areas, and the maximum value of each area is taken as the new feature value to obtain the pooled feature map.

[0082] Step S133: The texture change gradient features of continuous timestamps and the equipment operation action fragment features are modeled based on the time-dependent nature of the recurrent neural network to generate construction equipment operation frequency features and construction surface operation status features.

[0083] In this embodiment, a recurrent neural network (such as a bidirectional long short-term memory unit, Bi-LSTM) can capture the feature dependencies of consecutive timestamps.

[0084] The bidirectional long short-term memory unit contains memory units in both the forward and backward directions. At each time step, the forward memory unit updates the current hidden state based on the current input and the hidden state of the previous time step, and the backward memory unit updates the current hidden state based on the current input and the hidden state of the next time step. By splicing the forward and backward hidden states together, a richer feature representation is obtained.

[0085] For example, at the tth time step, the forward memory unit updates the current hidden state according to the input texture change gradient features and the equipment operation action fragment features and the hidden state of the t-1th time step, and the backward memory unit updates the current hidden state according to the input and the hidden state of the t+1th time step. After the forward and backward hidden states are concatenated, they are processed by the fully connected layer and the activation function to output the equipment operation frequency prediction features and the construction surface operation state prediction features of the time step.

[0086] By processing the entire time series, the construction equipment operation frequency characteristics and construction surface operation status characteristics are generated. The construction equipment operation frequency characteristics can be obtained by counting the equipment operation frequency prediction characteristics of each time step, and the construction surface operation status characteristics can be obtained by classifying the construction surface operation status prediction characteristics of each time step.

[0087] Step S134: The spatial relationship of the motion trajectory features of the pixel area is modeled through the graph convolution network, and a construction movement graph structure is constructed with the centroid of the trajectory bounding box as the node and the distance between adjacent trajectories as the edge weight. A speed-direction joint analysis is performed on the construction movement graph structure, and the displacement vector modulus of each construction worker per unit time and the angular covariance of the adjacent worker trajectories are calculated to generate the movement path features of the construction workers.

[0088] In this embodiment, first, the centroids of the trajectory bounding boxes are taken as nodes, and the distances between the centroids of adjacent trajectory bounding boxes are calculated as edge weights to construct the construction movement graph structure. For example, for the centroids A and B of two trajectory bounding boxes, the Euclidean distance between them is calculated as the edge weight.

[0089] A speed-direction joint analysis is performed on the construction movement graph structure. The displacement vector modulus of each construction worker per unit time is calculated. The displacement vector modulus can be obtained by calculating the square root of the sum of the squares of the centroid coordinates of the trajectory bounding box of adjacent timestamps. For example, at the t-th timestamp, the centroid coordinates of the trajectory bounding box of a construction worker are (x1, y1), and at the t+1-th timestamp, the centroid coordinates are (x2, y2), then the displacement vector modulus is the square root of ((x2-x1)^2+(y2-y1)^2).

[0090] Calculate the angular covariance of adjacent personnel trajectories. Angular covariance can reflect the directional correlation of adjacent personnel trajectories. Angular covariance is measured by calculating the cosine value of the angle between the displacement vectors of adjacent personnel. For example, for two adjacent construction workers, their displacement vectors are vector A and vector B, respectively. The dot product of vector A and vector B is calculated and divided by the product of their modulus lengths to obtain the cosine value of the angle.

[0091] By analyzing the structure of the construction movement graph, the construction personnel movement path characteristics are generated. The construction personnel movement path characteristics include the displacement vector modulus of each construction personnel, the angle covariance of the adjacent personnel trajectories and other information.

[0092] Step S135: Aggregate the construction personnel's movement path features, the construction equipment operation frequency features and the construction surface operation status features to generate a dynamic spatiotemporal feature set.

[0093] In this embodiment, these features can be combined in a splicing manner to form a higher-dimensional feature vector. For example, the construction personnel movement path feature is a vector containing 10 elements, the construction equipment operation frequency feature is a vector containing 5 elements, and the construction surface operation status feature is a vector containing 3 elements. Then they are spliced ​​together to form a dynamic spatiotemporal feature set containing 10+5+3=18 elements.

[0094] Step S140: Perform construction personnel statistics processing based on the dynamic spatiotemporal feature set to generate statistics on the number of personnel entering the cave, personnel distribution statistics at each construction site, and personnel identity matching results. The personnel identity matching results are determined by comparing the identification features worn by the personnel with identification templates in the pre-registered identity database.

[0095] The construction personnel statistics processing is performed based on the dynamic spatiotemporal feature set. The specific steps are as follows: Step S141: Determine the moving direction of the construction workers according to the displacement vector modulus in the construction workers' moving path characteristics and the angular covariance of the trajectories of adjacent workers. When the displacement vector modulus is greater than the second threshold and the moving direction corresponding to the angular covariance points to the tunnel entrance, it is marked as a tunnel entry behavior. The number of tunnel entry behaviors per unit time is accumulated to generate a statistical result of the number of tunnel entry personnel.

[0096] In this embodiment, the displacement vector modulus represents the moving distance of the construction personnel in a unit time, and the angle covariance reflects the directional correlation of the trajectories of adjacent personnel.

[0097] A second threshold is preset (assuming it is 2 m / s). When the displacement vector modulus is greater than 2 m / s and the moving direction corresponding to the angle covariance points to the tunnel entrance, it is marked as entering the tunnel. For example, if the displacement vector modulus of a construction worker is 2.5 m / s, and by analyzing the angle covariance of the trajectories of adjacent personnel, it is determined that the moving direction points to the tunnel entrance, the movement of the construction worker is marked as entering the tunnel.

[0098] The number of cave-entry behaviors in a unit of time (assuming 1 hour) is accumulated to generate the number of cave-entry statistics. For example, if 10 cave-entry behaviors are counted within 1 hour, the number of cave-entry statistics is 10.

[0099] Step S142: Based on the predefined three-dimensional coordinate model of the tunnel construction site, the coordinates of the trajectory bounding box of each construction worker are mapped to the three-dimensional coordinate model, the frequency of occurrence of the trajectory bounding box within the coordinate interval of each construction site is counted, and the personnel distribution statistics of each construction site are generated.

[0100] The predefined 3D coordinate model of the tunnel construction site is established based on the actual structure and construction plan of the tunnel, which accurately divides the coordinate range of each construction site in 3D space. For example, the tunnel is divided into different parts such as the portal construction area, the face construction area, and the lining construction area, and each part has a clear 3D coordinate range.

[0101] For the coordinates of the trajectory bounding box of each construction worker, it is necessary to map it from the image coordinate system to the three-dimensional coordinate system of the three-dimensional coordinate model of the tunnel construction site. This mapping process is based on the conversion relationship between the image coordinate system and the global coordinate system established in step S1213, and is further converted in combination with the coordinate reference of the three-dimensional coordinate model of the tunnel construction site. Assuming that the centroid coordinates of the trajectory bounding box of a construction worker in the image coordinate system are (x_img, y_img), it is first converted to the coordinates in the global coordinate system (x_global, y_global) through the conversion matrix, and then the global coordinates are converted to the coordinates in the three-dimensional coordinate model (x_3d, y_3d, z_3d) according to the origin position and coordinate axis direction of the three-dimensional coordinate model of the tunnel construction site.

[0102] Next, count the frequency of occurrence of the trajectory bounding box within the coordinate interval of each construction site. For each construction worker's trajectory bounding box, determine whether the three-dimensional coordinates of its center of mass fall within the coordinate interval of a certain construction site. If it falls within a certain interval, the frequency of occurrence of the trajectory bounding box of the construction site is increased by 1. For example, within a period of time, the center of mass coordinates of the trajectory bounding boxes of 5 construction workers fall within the coordinate interval of the face construction area, then the frequency of occurrence of the trajectory bounding box of the face construction area is 5 times. By performing such statistics on the trajectory bounding boxes of all construction workers, the statistical results of the personnel distribution of each construction site can be generated. For example, the statistical results of the personnel distribution in the portal construction area are 3 people, 5 people in the face construction area, 2 people in the lining construction area, etc.

[0103] Step S143: perform optical character recognition on the job number area in the identification feature worn by the person, extract the job number string, perform fuzzy matching on the job number string with the pre-registered identity database, and generate a personnel identity matching result when the matching similarity exceeds a third threshold.

[0104] In this embodiment, the identification features of the personnel wear include the information of the work type number area. Optical character recognition (OCR) is performed on the area to extract the work type number string therein. First, the image portion containing the work type number area is preprocessed, including grayscale, binarization, denoising and other operations to improve the accuracy of character recognition. For example, a grayscale method similar to that in step S1211 is used to convert a color image into a grayscale image, and then binarization segmentation is performed using a preset fixed grayscale threshold to remove noise interference in the image.

[0105] Then, a mature OCR algorithm is used to perform character recognition on the preprocessed image. For example, a deep learning-based OCR model is used. The model has been trained with a large number of character samples and can accurately recognize the characters in the image. Assuming that the work type number on the personnel wear identification is "007", the string is successfully extracted after OCR processing.

[0106] Perform fuzzy matching on the extracted work type number string with the pre-registered identity database. The pre-registered identity database stores the identity information of all construction workers, including work type number, name, position, etc. Fuzzy matching is to deal with possible recognition errors, such as individual character errors in OCR recognition results. Use a string similarity algorithm (such as the edit distance algorithm) to calculate the similarity between the extracted work type number string and each work type number in the database. The edit distance refers to the minimum number of editing operations (insertion, deletion, replacement) required to convert one string into another. The similarity can be calculated by subtracting the ratio of the edit distance to the length of the longer string from 1.

[0107] A third threshold is preset (assuming it is 0.8). When the matching similarity exceeds 0.8, it is considered that the matching personnel identity information is found. For example, the extracted work type number string is "007". A record with the work type number "007" is found in the pre-registered identity database. Its matching similarity is 1, which exceeds the third threshold of 0.8. Then, a personnel identity matching result is generated based on the record, including the name, position and other detailed information of the construction worker.

[0108] Step S150: Perform work efficiency comparison analysis based on the statistical results of the number of people entering the cave, the statistical results of the personnel distribution and the preset construction progress plan, generate a construction progress deviation index and a resource allocation optimization strategy, and when it is detected that the construction progress deviation index exceeds a first threshold, trigger a construction early warning instruction and output the resource allocation optimization strategy to the management terminal.

[0109] According to the statistical results of the number of people entering the cave, the statistical results of the distribution of people and the preset construction schedule, the work efficiency comparison analysis is carried out. The specific steps are as follows: Step S151: Obtain a target staffing table for the current construction phase from the preset construction schedule, wherein the target staffing table includes theoretical staffing numbers and work type combination requirements for each construction site.

[0110] In this embodiment, the preset construction schedule is a detailed plan formulated according to the overall planning and time arrangement of the tunnel construction, which includes the specific requirements of each construction stage. The target staffing table for the current construction stage is obtained from the plan. For example, the current construction stage is the initial support stage of the tunnel. The target staffing table stipulates that the theoretical number of personnel in the tunnel entrance construction area is 4, and the work type combination requirements are 2 blasters, 1 support worker and 1 safety officer; the theoretical number of personnel in the face construction area is 6, and the work type combination requirements are 3 excavators, 2 slag workers and 1 surveyor, etc.

[0111] Step S152: Calculate the absolute difference between the actual number of personnel in the personnel distribution statistics of each construction site and the theoretical number of personnel to generate a personnel number deviation index.

[0112] In this embodiment, for each construction site, the absolute difference between the actual number of personnel and the theoretical number of personnel is calculated. For example, the actual number of personnel in the tunnel entrance construction area is 3 people, and the theoretical number of personnel is 4 people, so the deviation of the number of personnel in this site is |3-4|=1 person; the actual number of personnel in the face construction area is 5 people, and the theoretical number of personnel is 6 people, and the deviation of the number of personnel is |5-6|=1 person. The deviation of the number of personnel in each construction site is summarized to obtain a personnel number deviation index. Assuming that there are 3 construction sites in the tunnel, and the deviations of the number of personnel are 1 person, 1 person, and 0 people respectively, the deviation index of the number of personnel is 1+1+0=2 people.

[0113] Step S153: Compare the job type combinations in the actual number of personnel with the job type combination requirements, count the missing job types and redundant job types, and generate a job type matching deviation index.

[0114] In this embodiment, the job type combination in the actual number of personnel in each construction site is compared with the job type combination requirements in the target personnel configuration table. For each construction site, check whether the actual types and numbers of jobs that exist meet the requirements. For example, in the tunnel entrance construction area, the target job type combination requires 2 blasters, 1 support worker, and 1 safety officer, but there are only 1 blaster, 1 support worker, and 1 safety officer in the actual personnel, then the missing job type is 1 blaster; if there are 3 blasters, 1 support worker, and 1 safety officer in the actual personnel, then the redundant job type is 1 blaster.

[0115] Count the missing and redundant work types of all construction sites to generate a work type matching deviation index. The number of missing and redundant work types can be summarized. For example, in the entire tunnel construction, there are 2 missing work types and 1 redundant work type, so the work type matching deviation index is 2+1=3.

[0116] Step S154: Calculate the construction progress deviation index according to the personnel quantity deviation index and the work type matching deviation index. The calculation formula of the construction progress deviation index is: deviation index = α×personnel quantity deviation+β×work type matching deviation, wherein α and β are weight coefficients dynamically adjusted according to the construction stage.

[0117] In this embodiment, the construction progress deviation index is calculated based on the personnel quantity deviation index and the work type matching deviation index. The weight coefficients α and β are dynamically adjusted according to the construction stage. Different construction stages have different requirements for the number of personnel and the combination of work types, so the weight coefficients will also change accordingly. For example, in the initial support stage of the tunnel, the adequacy of the number of personnel has a greater impact on the construction progress. At this time, α can be set to 0.6 and β to 0.4; in the later decoration stage of the tunnel, the rationality of the work type combination is more critical. α can be set to 0.4 and β to 0.6.

[0118] Assuming that in the current construction stage α=0.6, β=0.4, the personnel quantity deviation index is 2 people, and the work type matching deviation index is 3, then the construction progress deviation index = 0.6×2+0.4×3=2.4.

[0119] Step S155: When the construction progress deviation index exceeds a first threshold, a resource allocation optimization strategy including personnel transfer suggestions and recruitment needs is generated based on the missing job types and the redundant job types.

[0120] The first threshold is preset (assuming it is 2). When the construction progress deviation index exceeds 2, it means that there may be a large deviation in the construction progress and resource allocation optimization is needed. Based on the missing and redundant job types obtained by statistics, a resource allocation optimization strategy including personnel transfer suggestions and recruitment needs is generated.

[0121] For example, if the statistical results show that at a certain construction stage, the face construction area is short of 2 tunneling workers, while there are 3 redundant support workers in the tunnel entrance construction area, then the resource allocation optimization strategy can suggest transferring the 2 support workers in the tunnel entrance construction area to the face construction area for tunneling training before taking up the post; if the missing jobs cannot be deployed among the existing personnel, recruitment needs will be proposed based on the type and number of missing jobs, such as recruiting 2 tunneling workers.

[0122] When it is detected that the construction progress deviation index exceeds the first threshold, the construction early warning instruction is triggered. The construction early warning instruction can be an audible and visual alarm signal, or a message notification automatically sent by the system. At the same time, the generated resource allocation optimization strategy is output to the management terminal, which can be a computer, mobile phone and other devices, so that the management personnel can timely understand the construction progress deviation and take corresponding measures. For example, after the management terminal receives the construction early warning instruction and resource allocation optimization strategy, the management personnel can make personnel transfer arrangements and formulate recruitment plans according to the strategy to ensure that the tunnel construction can proceed smoothly according to the scheduled progress.

[0123] Step S210: The training method of the dynamic feature extraction model includes obtaining a training data set containing a spatiotemporal feature sequence and a corresponding annotation data set, wherein the annotation data set includes construction equipment operation frequency annotations, construction surface operation status annotations, and construction personnel movement path annotations.

[0124] In this embodiment, the training data set can be obtained by collecting video data in multiple tunnel construction scenes, and obtaining a spatiotemporal feature sequence through a processing process similar to steps S110-S120. For example, multiple cameras are installed at different tunnel construction sites to collect videos, and then the collected video data is subjected to spatiotemporal feature fusion processing to generate a spatiotemporal feature sequence.

[0125] The corresponding annotated data set needs to be manually annotated. For the operation frequency annotation of construction machinery, it is necessary to count the number of operations of each construction machinery in the video and record it. For example, in a video, the excavator is operated 10 times and the loader is operated 8 times. For the annotation of the construction surface operation status, classification annotation is performed according to the actual operation conditions of the construction surface, such as normal operation, shutdown for maintenance, waiting for materials, etc. For the annotation of the movement path of construction personnel, the movement trajectory of each construction personnel is manually tracked and their location information at different timestamps is recorded.

[0126] Step S220: Perform multi-channel feature normalization processing on the spatiotemporal feature sequence, convert the pixel area motion trajectory features, personnel wear identification features and construction equipment contour features into normalized feature vectors with zero mean and unit variance, ensuring that the dimensional units of different feature dimensions are consistent.

[0127] In this embodiment, standardization operations are performed on the pixel area motion trajectory features, personnel wear identification features, and construction equipment contour features. Taking the pixel area motion trajectory features as an example, the mean and standard deviation of the feature in the training data set are first calculated. Assuming that the pixel area motion trajectory feature is a vector containing multiple coordinate values, the mean and standard deviation of the coordinate values ​​corresponding to all vectors are calculated. For example, for the horizontal coordinate of the trajectory bounding box, the mean and standard deviation of the horizontal coordinates of all trajectory bounding boxes are calculated.

[0128] Then, subtract the mean from each coordinate value of each vector and divide it by the standard deviation to obtain a normalized feature vector with zero mean and unit variance. The same method is used to standardize the wear identification features of personnel and the contour features of construction equipment. For example, the wear identification feature of personnel may be a vector containing information such as the color of the helmet and the type of reflective clothing, and the contour feature of the construction equipment may be a vector containing the coordinates of the polygon vertices. They are standardized separately to ensure that the dimensional units of different feature dimensions are consistent so that the model can better learn these features.

[0129] Step S230: Initialize the convolution kernel weight matrix of the convolution neural network, the hidden state transfer matrix of the recurrent neural network, and the adjacency matrix update function of the graph convolution network, wherein the convolution kernel size matches the spatial size of the pixel area of ​​the spatiotemporal feature unit, the hidden state dimension matches the time step of the instrument operation action fragment feature, and the edge weight dimension of the adjacency matrix matches the actual physical distance unit of the center of mass spacing of the trajectory bounding box.

[0130] In this embodiment, for the convolutional neural network, the convolution kernel size needs to match the spatial size of the pixel area of ​​the spatiotemporal feature unit. For example, if the pixel area of ​​the spatiotemporal feature unit is a 32x32 image area, then the size of the convolution kernel can be set to a suitable size such as 3x3 or 5x5 to ensure that local features can be effectively extracted. The initial value of the convolution kernel weight matrix can be obtained by random initialization, such as random sampling from a normal distribution.

[0131] For recurrent neural networks, the hidden state dimension needs to match the time step of the machine operation action clip feature. Assuming that the time step of the machine operation action clip feature is 10, the hidden state dimension can be set to 100 to capture sufficient time-dependent information. The initial value of the hidden state transfer matrix can also be randomly initialized.

[0132] For graph convolutional networks, the edge weight dimension of the adjacency matrix needs to match the actual physical distance unit of the track bounding box centroids. For example, if the actual physical distance unit of the track bounding box centroids is meters, then the edge weights of the adjacency matrix should also be initialized in meters. The adjacency matrix update function can take some common function forms, such as linear functions or nonlinear activation functions, and the initial parameters can be randomly initialized.

[0133] Step S240: Slice the standardized spatiotemporal feature sequence according to the time window to generate a time segment sequence, each time segment sequence containing a fixed number of continuous spatiotemporal feature units.

[0134] In this embodiment, the time window size is preset (assuming it is 5), that is, each time segment sequence contains 5 continuous space-time feature units. Starting from the starting position of the space-time feature sequence, 5 continuous space-time feature units are selected in sequence to form a time segment sequence, and then move back one time step, and then select 5 continuous space-time feature units to form the next time segment sequence, until the entire space-time feature sequence is traversed. For example, if the space-time feature sequence contains 100 space-time feature units, and it is sliced ​​according to the time window size of 5, 100-5+1=96 time segment sequences can be obtained.

[0135] Step S250: Input the time segment sequence into the convolutional neural network, and extract a local texture feature map through alternating operations of multiple convolutional layers and pooling layers. The local texture feature map includes texture change gradient features of the construction surface working area and equipment operation action segment features.

[0136] In this embodiment, in the convolution layer, the initialized convolution kernel weight matrix is ​​used to perform convolution operation on each spatiotemporal feature unit in the time segment sequence. For example, a 3x3 convolution kernel is used to convolve the spatiotemporal feature unit, and the coefficients of the convolution kernel are continuously weighted and summed with the pixel values ​​of the spatiotemporal feature unit to obtain new feature values. Through the operation of multiple convolution layers, more advanced local features are gradually extracted.

[0137] In the pooling layer, the maximum pooling or average pooling method is used to reduce the dimension of the feature map output by the convolution layer. For example, a 2x2 maximum pooling kernel is used to divide the feature map into multiple 2x2 regions, and the maximum value of each region is taken as the new feature value to obtain the pooled feature map. Through the alternating operation of multiple convolutional layers and pooling layers, a local texture feature map is extracted, which contains the texture change gradient features of the construction surface working area and the machine operation action segment features.

[0138] Step S260: Input the local texture feature map into the recurrent neural network in chronological order, capture the feature dependency of adjacent time slices through a bidirectional long short-term memory unit, and output the predicted features of the machine operation frequency and the predicted features of the construction surface operation status of each time slice.

[0139] In this embodiment, the local texture feature map is input into the bidirectional long short-term memory unit (Bi-LSTM) of the recurrent neural network in chronological order. At each time step, the forward memory unit of the Bi-LSTM updates the current hidden state according to the current input local texture feature map and the hidden state of the previous time step, and the backward memory unit updates the current hidden state according to the current input and the hidden state of the next time step. By splicing the forward and backward hidden states together, a richer feature representation is obtained.

[0140] After being processed by the fully connected layer and the activation function, the predicted features of the machine operation frequency and the predicted features of the construction surface operation status of each time slice are output. For example, at the tth time step, Bi-LSTM outputs a vector, part of which represents the predicted features of the machine operation frequency, and the other part represents the predicted features of the construction surface operation status. By processing the entire time series, the predicted features of each time slice are obtained.

[0141] Step S270: Input the pixel area motion trajectory features in the spatiotemporal feature sequence into the graph convolutional network, construct a dynamic adjacency matrix based on the centroid coordinates of the trajectory bounding box, and generate a motion path prediction feature with enhanced spatial relationship through node feature aggregation and edge weight update.

[0142] In this embodiment, a dynamic adjacency matrix is ​​constructed based on the coordinates of the centroids of the trajectory bounding box, and the elements of the adjacency matrix represent the connection relationship between nodes (centroids of the trajectory bounding box) and edge weights (distance between centroids of the trajectory bounding box). For example, for two trajectory bounding box centroids A and B, if the distance between them is less than a certain threshold, it is considered that there is a connection between them, and the element value of the corresponding position in the adjacency matrix is ​​their distance; if the distance is greater than the threshold, it is considered that there is no connection between them, and the element value of the corresponding position in the adjacency matrix is ​​0.

[0143] In the graph convolutional network, the mobile path prediction features with enhanced spatial relationships are generated through node feature aggregation and edge weight update. Node feature aggregation refers to the weighted sum of the features of adjacent nodes, and edge weight update refers to the adjustment of edge weights based on node features and connection relationships. For example, the features of a node can be updated through the features of its adjacent nodes and edge weights, and the formula can be expressed as node new features = weighted sum (adjacent node features * edge weight). By continuously performing node feature aggregation and edge weight update, the mobile path prediction features with enhanced spatial relationships are obtained.

[0144] Step S280: splicing the equipment operation frequency prediction feature, the construction surface operation state prediction feature and the moving path prediction feature to generate a dynamic spatiotemporal feature prediction set.

[0145] In this embodiment, for example, the machine operation frequency prediction feature is a vector containing 5 elements, the construction surface operation state prediction feature is a vector containing 3 elements, and the moving path prediction feature is a vector containing 10 elements. They are spliced ​​together in sequence to form a dynamic spatiotemporal feature prediction set containing 5+3+10=18 elements. This splicing method can integrate prediction features from different aspects and provide more comprehensive information for subsequent loss calculation and model optimization.

[0146] Step S290: Calculate the multi-task loss function between the dynamic spatiotemporal feature prediction set and the labeled data set, wherein the multi-task loss function includes the weighted sum of the mean square error of the machine operation frequency, the cross entropy loss of the construction surface operation state, and the cosine similarity loss of the moving path direction.

[0147] Calculate the multi-task loss function between the dynamic spatiotemporal feature prediction set and the labeled data set, which specifically includes the following parts: Step S291: Perform time dimension average pooling on the equipment operation frequency prediction features to generate a global operation frequency scalar for each construction equipment, and calculate the mean square error between the global operation frequency scalar and the actual operation frequency of the corresponding equipment in the labeled data.

[0148] In this embodiment, the machine operation frequency prediction feature is a sequence with multiple values ​​in the time dimension. For example, there are 10 time steps in a time segment sequence, and each time step has a machine operation frequency prediction value. These 10 values ​​are averaged and pooled, that is, they are added and divided by the number of time steps to obtain the global operation frequency scalar of each construction machine. For example, for the machine operation frequency prediction feature of the excavator, the predicted values ​​in 10 time steps are 3, 4, 3, 5, 4, 3, 4, 5, 4, 3, respectively. These values ​​are added to get 38, and then divided by 10 to get the global operation frequency scalar of 3.8.

[0149] Calculate the mean square error between the global operation frequency scalar and the actual operation frequency of the corresponding equipment in the labeled data. Assuming that the actual operation frequency of the excavator in the labeled data is 4, the mean square error is calculated by first calculating the difference between the two, that is, 3.8-4=-0.2, and then squaring the difference to get (-0.2)^2=0.04. By performing this calculation for all construction equipment, the mean square error of the equipment operation frequency is obtained.

[0150] Step S292: Perform multi-category probability normalization on the construction surface operation state prediction features, generate the operation state probability distribution of each time slice, and calculate the cross entropy loss between the operation state probability distribution and the corresponding state label in the annotated data.

[0151] In this embodiment, the construction surface operation state prediction feature is usually a vector, whose elements represent the prediction scores of different operation states. The scores are normalized using the softmax function and converted into probability distributions. For example, the construction surface operation state has three states: normal operation, shutdown and maintenance, and waiting for materials. The construction surface operation state prediction feature vector is [2, 1, 3]. The probability distribution calculated by the softmax function is [0.2, 0.1, 0.7], indicating that the probability of normal operation is 0.2, the probability of shutdown and maintenance is 0.1, and the probability of waiting for materials is 0.7.

[0152] Calculate the cross entropy loss between the probability distribution of the work status and the corresponding state label in the annotated data. Assume that the work status label of this time slice in the annotated data is waiting for materials, and the corresponding probability distribution is [0, 0, 1]. The calculation method of the cross entropy loss is to multiply the probability distribution of the annotated data with the corresponding element of the predicted probability distribution and take the negative logarithm and then sum them. That is, -(0*log(0.2)+0*log(0.1)+1*log(0.7))=-log(0.7). By performing such calculations for all time slices, the cross entropy loss of the work status of the construction surface is obtained.

[0153] Step S293: Decomposing the movement path prediction feature by a unit time displacement vector, extracting the displacement direction angle value, and calculating the cosine similarity between the displacement direction angle value and the corresponding path direction angle value in the annotation data.

[0154] In this embodiment, the mobile path prediction feature can be represented as a series of coordinate points, and the unit time displacement vector is obtained by calculating the difference between adjacent coordinate points. For example, the coordinates at the t-th time step are (x1, y1), and the coordinates at the t+1-th time step are (x2, y2), then the unit time displacement vector is (x2-x1, y2-y1).

[0155] Extract the displacement direction angle value by calculating the angle between the displacement vector and the positive direction of the x-axis. For example, if the displacement vector is (3, 4), the inverse tangent function can be used to calculate the angle, that is, arctan(4 / 3), to obtain the displacement direction angle value.

[0156] Calculate the cosine similarity between the displacement direction angle value and the corresponding path direction angle value in the annotation data. Assuming that the predicted displacement direction angle value is 60 degrees and the corresponding path direction angle value in the annotation data is 50 degrees, convert the angle value into vector form, and then calculate the dot product of the two vectors divided by the product of their modulus lengths to get the cosine similarity. By performing such calculations on the movement paths of all construction workers, the cosine similarity loss of the movement path direction is obtained.

[0157] Step S294: Set the equipment operation frequency loss weight, operation status loss weight and movement path similarity weight according to the construction monitoring task priority, and add the weighted sub-losses to generate a multi-task loss function.

[0158] In this embodiment, for example, in certain construction stages, the frequency of machine operation has a greater impact on the construction progress, so the machine operation frequency loss weight can be set to 0.5; accurate judgment of the operating status is also very important, and the operating status loss weight can be set to 0.3; the moving path similarity is relatively less important, and the moving path similarity weight can be set to 0.2.

[0159] The weighted sub-losses are added together to generate the multi-task loss function. That is, the multi-task loss function = the weight of the machine operation frequency loss * the mean square error of the machine operation frequency + the weight of the operation state loss * the cross entropy loss of the construction surface operation state + the weight of the moving path similarity * the cosine similarity loss of the moving path direction. For example, the mean square error of the machine operation frequency is 0.04, the cross entropy loss of the construction surface operation state is 0.1, and the cosine similarity loss of the moving path direction is 0.05. According to the above weights, the multi-task loss function is calculated to be 0.5*0.04+0.3*0.1+0.2*0.05=0.02+0.03+0.01=0.06.

[0160] Step S295: During the back propagation process, the local texture feature map gradient of the convolutional neural network, the state transfer gradient of the recurrent neural network, and the adjacency matrix gradient of the graph convolutional network are chain-derived according to the multi-task loss function so that the dimension of each sub-loss gradient matches the corresponding network parameter update step size.

[0161] In this embodiment, chain derivation is based on the composite function derivation rule, starting from the multi-task loss function, and gradually calculating the gradient of each network layer.

[0162] For convolutional neural networks, the gradient of the local texture feature map is calculated according to the multi-task loss function, and then the gradient of the convolution kernel weight matrix is ​​obtained through chain derivation. For example, the gradient of the local texture feature map output by the multi-task loss function for a convolution layer is [0.1, 0.2, 0.3]. The gradient of the convolution kernel weight matrix of the convolution layer can be calculated through chain derivation, so that the convolution kernel weight matrix can be updated according to the gradient.

[0163] For recurrent neural networks, the state transfer gradient is calculated based on the multi-task loss function, that is, the gradient of the multi-task loss function to the hidden state transfer matrix is ​​calculated. For example, the gradient of the multi-task loss function to the hidden state at a certain time step is obtained through back propagation calculation, and then the gradient of the hidden state transfer matrix is ​​calculated based on the relationship between the hidden state and the hidden state transfer matrix.

[0164] For graph convolutional networks, the adjacency matrix gradient is calculated based on the multi-task loss function. The gradient of the adjacency matrix is ​​calculated by analyzing the relationship between the multi-task loss function and the adjacency matrix update function. For example, the adjacency matrix update function is a complex function, and the gradient of the multi-task loss function to the adjacency matrix elements can be obtained by chain derivation.

[0165] Through this chain derivation process, the dimensions of each sub-loss gradient match the corresponding network parameter update step size, ensuring that the model can accurately update parameters according to the gradient, thereby continuously optimizing the performance of the model. After each back propagation, an optimization algorithm (such as the stochastic gradient descent algorithm) is used to update the convolution kernel weight matrix of the convolutional neural network, the hidden state transfer matrix of the recurrent neural network, and the adjacency matrix update function parameters of the graph convolutional network according to the calculated gradient until the loss function converges, that is, the value of the multi-task loss function no longer has a significant downward trend, at which point the model training is completed.

[0166] Figure 2 A schematic diagram of exemplary hardware and software components of a tunnel construction monitoring and analysis system 100 based on AI video monitoring that can implement the concept of the present invention is shown in some embodiments of the present invention. For example, the processor 120 can be used in the tunnel construction monitoring and analysis system 100 based on AI video monitoring and is used to perform the functions of the present invention.

[0167] The tunnel construction monitoring and analysis system 100 based on AI video monitoring can be a general server or a special-purpose server, both of which can be used to implement the tunnel construction monitoring and analysis method based on AI video monitoring of the present invention. Although the present invention only shows one server, for convenience, the functions described in the present invention can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0168] For example, the tunnel construction monitoring and analysis system 100 based on AI video monitoring may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the tunnel construction monitoring and analysis system 100 based on AI video monitoring may also include program instructions stored in ROM, RAM, or other types of non-temporary storage media, or any combination thereof. The method of the present invention can be implemented according to these program instructions. The tunnel construction monitoring and analysis system 100 based on AI video monitoring also includes an I / O interface 150 between a computer and other input and output devices.

[0169] For ease of explanation, only one processor is described in the tunnel construction monitoring and analysis system 100 based on AI video surveillance. However, it should be noted that the tunnel construction monitoring and analysis system 100 based on AI video surveillance in the present invention may also include multiple processors, so the steps performed by one processor described in the present invention may also be performed jointly or individually by multiple processors. For example, if the processor of the tunnel construction monitoring and analysis system 100 based on AI video surveillance executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0170] In addition, an embodiment of the present invention also provides a readable storage medium, in which computer executable instructions are preset. When the processor executes the computer executable instructions, the above-mentioned tunnel construction monitoring and analysis method based on AI video monitoring is implemented.

[0171] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. A tunnel construction monitoring and analysis method based on AI video monitoring, characterized in that: The method comprises: The original video stream data set of the construction scene is acquired in real time by a multi-view video acquisition device deployed in the tunnel construction area, wherein the original video stream data set includes a plurality of spatially continuous frame sequences aligned with time stamps; Performing spatiotemporal feature fusion processing on the original video stream data set to generate a spatiotemporal feature sequence, wherein each spatiotemporal feature unit in the spatiotemporal feature sequence is composed of a pixel region motion trajectory feature, a personnel wear identification feature, and a construction equipment contour feature of a corresponding frame sequence; Performing dynamic feature extraction processing on the spatiotemporal feature sequence to obtain a dynamic spatiotemporal feature set, wherein the dynamic spatiotemporal feature set includes construction personnel movement path features, construction equipment operation frequency features, and construction surface operation status features; Execute construction personnel statistics processing based on the dynamic spatiotemporal feature set to generate statistics on the number of personnel entering the cave, statistics on personnel distribution at each construction site, and personnel identity matching results, wherein the personnel identity matching results are determined by comparing identification features worn by the personnel with identification templates in a pre-registered identity database; According to the statistical results of the number of people entering the cave, the statistical results of the personnel distribution and the preset construction progress plan, a comparative analysis of work efficiency is performed to generate a construction progress deviation indicator and a resource allocation optimization strategy. When it is detected that the construction progress deviation indicator exceeds a first threshold, a construction early warning instruction is triggered and the resource allocation optimization strategy is output to the management terminal.

2. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 1 is characterized in that: The spatiotemporal feature fusion processing is performed on the original video stream data set to generate a spatiotemporal feature sequence, wherein each spatiotemporal feature unit in the spatiotemporal feature sequence is composed of the pixel area motion trajectory feature of the corresponding frame sequence, the personnel wear identification feature and the construction equipment contour feature. Composition, including: Performing motion trajectory segmentation processing on each spatial continuous frame sequence in the original video stream data set to obtain pixel area motion trajectory features of each construction worker, wherein the pixel area motion trajectory features include independent motion trajectory segments and corresponding trajectory bounding box coordinates; A color space distribution histogram is extracted in the pixel area corresponding to the coordinates of the trajectory bounding box, and the color space distribution histogram is classified based on a pre-trained wear identification classification model to obtain a wear identification feature of the person, wherein the wear identification classification model is trained by a training data set labeled with a safety helmet, a reflective clothing, and a work type number; Performing edge enhancement processing on the construction machine outline in the spatial continuous frame sequence, extracting a polygon vertex coordinate set of the enhanced outline, performing shape matching based on the polygon vertex coordinate set and a pre-stored machine template library, and determining the type of construction machine and the corresponding operation state; The motion trajectory features of the pixel area, the identification features of the personnel wearing and the type of construction equipment at the same timestamp are spliced ​​to generate spatiotemporal feature units, and all spatiotemporal feature units are arranged in chronological order to obtain a spatiotemporal feature sequence.

3. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 1 is characterized in that: The step of performing motion trajectory segmentation processing on each spatial continuous frame sequence in the original video stream data set to obtain an independent motion trajectory segment of each construction worker and a corresponding trajectory bounding box coordinate includes: Performing grayscale processing on each frame image in the spatial continuous frame sequence, converting the color image into a grayscale image, and calculating a pixel brightness difference matrix between adjacent frames based on a preset adjacent time stamp interval; Extracting a dynamic background model according to the pixel brightness difference matrix, separating a foreground moving target region by a background difference method, performing a morphological closing operation on the foreground moving target region, eliminating noise interference and filling holes inside the target, and generating a connected domain labeling map; Fitting the minimum bounding rectangle of each foreground moving object based on the connected domain marker graph, calculating the centroid coordinates and aspect ratio of each bounding rectangle, mapping the centroid coordinates to the global coordinate system, and recording the pixel area of ​​each bounding rectangle; A multi-target tracking algorithm is used to associate the centroid coordinates of consecutive timestamps with motion trajectories, and the Euclidean distance of the centroid coordinates of the same target at adjacent timestamps is calculated. When the Euclidean distance is less than a preset distance threshold and the length-to-width ratio change is less than a preset ratio threshold, it is determined to be a motion trajectory segment of the same target; Merge the motion trajectory segments of the same target at consecutive timestamps to generate independent motion trajectory segments, and calculate the trajectory bounding box coordinates according to the global coordinate sequence of the bounding rectangle, wherein the trajectory bounding box coordinates are the minimum convex polygon vertex coordinate set containing all the vertices of the bounding rectangle; The independent motion trajectory segment is verified for trajectory integrity, and the coverage duration of the trajectory bounding box coordinates on the time axis is counted. When the coverage duration exceeds a preset duration threshold and the pixel area change rate is lower than a preset area change threshold, the independent motion trajectory segment and the corresponding trajectory bounding box coordinates are retained.

4. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 2 is characterized in that: The edge enhancement process is performed on the construction machine contour in the spatial continuous frame sequence to extract a polygon vertex coordinate set of the enhanced contour, including: Convert each frame image of the spatial continuous frame sequence into a grayscale image, and perform binary segmentation on the grayscale image based on a preset fixed grayscale threshold to obtain an initial binary image; Performing Gaussian filtering on the initial binary image to eliminate high-frequency noise interference to generate a smoothed binary image, and using the Canny edge detection algorithm to extract edge pixels from the smoothed binary image to generate an initial edge pixel set; Performing a morphological gradient operation on the initial edge pixel set, enhancing edge continuity by calculating pixel differences between a dilation operation and an erosion operation, and generating an edge enhanced image; Performing contour tracking processing based on the edge enhanced image, traversing adjacent edge pixel points in a clockwise direction to generate a closed contour point set, and eliminating a non-closed contour point set whose number of pixels is less than a preset contour point number threshold; Execute the Douglas-Peucker polygon approximation algorithm on the retained closed contour point set, calculate the maximum vertical distance error between each closed contour point set and the approximation polygon, and when the maximum vertical distance error is less than a preset shape error threshold, extract the vertex coordinates of the approximation polygon as a candidate vertex coordinate set; Calculate the vertex sampling point spacing based on the candidate vertex coordinate set, count the pixel spacing between adjacent vertices in the image coordinate system, convert the pixel spacing into a proportional distance based on the actual construction machine size, eliminate redundant vertices whose proportional distance is less than a preset minimum spacing threshold, and generate an optimized vertex coordinate set; The optimized vertex coordinate set is normalized, and according to the ratio coefficient between the image resolution and the actual size of the construction equipment, the vertex coordinates are converted into a normalized coordinate set based on the actual space coordinate system to generate a polygon vertex coordinate set of the enhanced contour.

5. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 2 is characterized in that: The step of performing dynamic feature extraction processing on the spatiotemporal feature sequence to obtain a dynamic spatiotemporal feature set includes: Constructing a dynamic feature extraction model including a convolutional neural network, a recurrent neural network and a graph convolutional network, and inputting the spatiotemporal feature sequence into the dynamic feature extraction model; The convolutional neural network is used to perform local feature convolution operations on each spatiotemporal feature unit to extract the texture change gradient features and the equipment operation action segment features of the construction surface operation area; The texture change gradient features of continuous timestamps and the equipment operation action fragment features are modeled for time sequence dependency through the recurrent neural network to generate construction equipment operation frequency features and construction surface operation status features; The spatial relationship of the motion trajectory features of the pixel area is modeled by the graph convolution network, and a construction movement graph structure is constructed with the centroid of the trajectory bounding box as a node and the distance between adjacent trajectories as an edge weight. A speed-direction joint analysis is performed on the construction movement graph structure, and the displacement vector modulus of each construction worker in unit time and the angular covariance of the trajectories of adjacent workers are calculated to generate the movement path features of the construction workers; The construction personnel's movement path characteristics, the construction equipment operation frequency characteristics and the construction surface operation status characteristics are aggregated to generate a dynamic spatiotemporal feature set.

6. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 5 is characterized in that: The training method of the dynamic feature extraction model includes: Acquire a training data set including a spatiotemporal feature sequence and a corresponding annotated data set, wherein the annotated data set includes annotations of construction equipment operation frequency, construction surface operation status, and construction personnel movement path; Perform multi-channel feature normalization processing on the spatiotemporal feature sequence, convert the pixel area motion trajectory features, personnel wear identification features and construction equipment contour features into normalized feature vectors with zero mean and unit variance, ensuring that the dimensional units of different feature dimensions are consistent; Initialize the convolution kernel weight matrix of the convolution neural network, the hidden state transfer matrix of the recurrent neural network, and the adjacency matrix update function of the graph convolution network, wherein the convolution kernel size matches the pixel region spatial size of the spatiotemporal feature unit, the hidden state dimension matches the time step of the instrument operation action segment feature, and the edge weight dimension of the adjacency matrix matches the actual physical distance unit of the centroid distance of the trajectory bounding box; Slice the standardized spatiotemporal feature sequence according to the time window to generate a time segment sequence, each of which contains a fixed number of continuous spatiotemporal feature units; Inputting the time segment sequence into the convolutional neural network, extracting a local texture feature map through alternating operations of multiple convolutional layers and pooling layers, wherein the local texture feature map includes texture change gradient features of the construction surface working area and equipment operation action segment features; The local texture feature map is input into the recurrent neural network in chronological order, the feature dependency relationship of adjacent time slices is captured through a bidirectional long short-term memory unit, and the predicted features of the machine operation frequency and the predicted features of the construction surface operation status of each time slice are output; Inputting the pixel area motion trajectory features in the spatiotemporal feature sequence into the graph convolutional network, constructing a dynamic adjacency matrix based on the centroid coordinates of the trajectory bounding box, and generating a motion path prediction feature with enhanced spatial relationship through node feature aggregation and edge weight update; Splicing the equipment operation frequency prediction feature, the construction surface operation state prediction feature and the moving path prediction feature to generate a dynamic spatiotemporal feature prediction set; Calculating a multi-task loss function between the dynamic spatiotemporal feature prediction set and the annotated data set, the multi-task loss function comprising a weighted sum of a mean square error of the machine operation frequency, a cross entropy loss of the construction surface operation state, and a cosine similarity loss of the moving path direction; Back propagation optimization is performed based on the multi-task loss function, and the convolution kernel weight matrix of the convolutional neural network, the hidden state transfer matrix of the recurrent neural network, and the adjacency matrix update function of the graph convolutional network are synchronously updated until the loss function converges.

7. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 6 is characterized in that: The calculating of the multi-task loss function between the dynamic spatiotemporal feature prediction set and the labeled data set includes: Performing time dimension average pooling on the prediction features of the machine operation frequency, generating a global operation frequency scalar for each construction machine, and calculating the mean square error between the global operation frequency scalar and the actual operation frequency of the corresponding machine in the labeled data; Performing multi-category probability normalization on the construction surface operation state prediction features, generating an operation state probability distribution for each time slice, and calculating the cross entropy loss between the operation state probability distribution and the corresponding state label in the annotated data; Decomposing the movement path prediction feature by a unit time displacement vector, extracting the displacement direction angle value, and calculating the cosine similarity between the displacement direction angle value and the corresponding path direction angle value in the annotation data; According to the priority of the construction monitoring task, the machine operation frequency loss weight, the operation status loss weight and the movement path similarity weight are set, and the weighted sub-losses are added together to generate a multi-task loss function; During the back propagation process, the local texture feature map gradient of the convolutional neural network, the state transfer gradient of the recurrent neural network, and the adjacency matrix gradient of the graph convolutional network are chain-derived according to the multi-task loss function so that the dimension of each sub-loss gradient matches the corresponding network parameter update step size.

8. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 1 is characterized in that: The construction personnel statistics processing is performed based on the dynamic spatiotemporal feature set to generate the statistics results of the number of personnel entering the cave, the personnel distribution statistics results of each construction site and the personnel identity matching results, including: Determine the moving direction of the construction personnel according to the displacement vector modulus in the construction personnel moving path characteristics and the angular covariance of the adjacent personnel trajectories; when the displacement vector modulus is greater than the second threshold and the moving direction corresponding to the angular covariance points to the tunnel entrance, mark it as a tunnel entry behavior; accumulate the number of tunnel entry behaviors per unit time to generate a statistical result of the number of tunnel entry personnel; Based on a predefined three-dimensional coordinate model of the tunnel construction site, the coordinates of the trajectory bounding box of each construction worker are mapped to the three-dimensional coordinate model, the frequency of occurrence of the trajectory bounding box within the coordinate interval of each construction site is counted, and the statistical results of the personnel distribution at each construction site are generated; Optical character recognition is performed on the job number area in the identification feature worn by the person, the job number string is extracted, the job number string is fuzzy matched with the pre-registered identity database, and a personnel identity matching result is generated when the matching similarity exceeds a third threshold.

9. The tunnel construction monitoring and analysis method based on AI video monitoring according to claim 1 is characterized in that: The comparative analysis of work efficiency based on the statistical results of the number of personnel entering the cave, the statistical results of the personnel distribution and the preset construction progress plan is used to generate a construction progress deviation index and a resource allocation optimization strategy, including: Obtaining a target staffing table for the current construction phase from the preset construction schedule, wherein the target staffing table includes theoretical staffing numbers and work type combination requirements for each construction site; Calculate the absolute difference between the actual number of personnel in the personnel distribution statistics of each construction site and the theoretical number of personnel to generate a personnel number deviation index; Compare the job type combinations in the actual number of personnel with the job type combination requirements, count missing job types and redundant job types, and generate job type matching deviation indicators; The construction progress deviation index is calculated according to the personnel quantity deviation index and the work type matching deviation index. The calculation formula of the construction progress deviation index is: deviation index = α×personnel quantity deviation+β×work type matching deviation, where α and β are weight coefficients dynamically adjusted according to the construction stage; When the construction progress deviation index exceeds a first threshold, a resource allocation optimization strategy including personnel transfer suggestions and recruitment needs is generated based on the missing job types and the redundant job types.

10. A tunnel construction monitoring and analysis system based on AI video monitoring, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the tunnel construction monitoring and analysis method based on AI video monitoring as described in any one of claims 1 to 9.

Images