Real-time abnormal behavior recognition method and system for tunnel construction videos

Abstract

The present invention discloses a method and system for real-time abnormal behavior recognition in tunnel construction videos. The method constructs a wide-angle field of view and a telephoto field of view respectively with the help of wide-angle video images and telephoto video images captured by a wide-angle main camera and a telephoto auxiliary camera, and combines the wide-angle field of view with the telephoto field of view to obtain a complementary field of view area. At the same time, target detection is performed on the target person in the telephoto video image, and the detected target person is tracked to obtain the motion area of ​​the target person. If the motion area does not completely fall within the complementary field of view area, it indicates that there is a motion blind spot, and the motion blind spot is supplemented by the telephoto auxiliary camera. The method also extracts the behavioral features of the target person in the updated complementary field of view area based on a multi-head attention convolutional network, and inputs the behavioral features into a pre-trained behavior prediction and classification model to identify whether the behavior of the target person is abnormal, thereby effectively solving the blind spot vision problem.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for real-time abnormal behavior recognition in tunnel construction videos. Background Art

[0002] The narrow and enclosed environment of tunnel construction creates significant blind spots, making it difficult to detect hidden dangers in tunnel construction. Currently, tunnel construction monitoring typically uses multiple cameras. However, due to the relatively fixed field of view of the cameras, this still presents difficulties in addressing blind spots. Furthermore, using a large number of cameras inevitably increases construction costs. Therefore, current tunnel construction monitoring methods, due to blind spot issues, suffer from poor accuracy and a high missed detection rate, making it difficult to ensure the normal operation of tunnel construction. Summary of the Invention

[0003] In view of this, the present invention provides a method and system for real-time abnormal behavior recognition in tunnel construction videos, which solves the technical problems of the current tunnel construction monitoring method, which has poor monitoring accuracy and high missed detection rate due to blind spot vision problems.

[0004] A first aspect of the present invention provides a method for real-time abnormal behavior recognition in tunnel construction videos, wherein a wide-angle main camera and a telephoto auxiliary camera are set in any one of multiple target construction areas of the tunnel, comprising:

[0005] respectively collecting a wide-angle video image and a telephoto video image of the target construction area by using the wide-angle main camera and the telephoto auxiliary camera;

[0006] Determining a wide-angle field of view and a telephoto field of view according to the wide-angle video image and the telephoto video image, and combining the wide-angle field of view and the telephoto field of view to obtain a complementary field of view area;

[0007] Performing target detection on a target person in the telephoto video image, and tracking the detected target person to obtain a motion area of ​​the target person;

[0008] Determining whether the motion area completely falls within the complementary field of view area;

[0009] When the motion area does not completely fall within the complementary field of view area, the telephoto auxiliary camera is used to retake the motion blind area, and the retaken motion blind area is updated to the complementary field of view area; wherein the motion blind area is the portion of the motion area that does not fall within the complementary field of view area;

[0010] A multi-head attention convolutional network is used to extract the behavioral characteristics of the target person in the updated complementary area of ​​the field of view;

[0011] The behavioral characteristics are input into a pre-trained behavior prediction classification model to identify whether the behavior of the target person is abnormal.

[0012] Optionally, determining a wide-angle field of view and a telephoto field of view based on the wide-angle video image and the telephoto video image, and combining the wide-angle field of view and the telephoto field of view to obtain a complementary field of view area includes:

[0013] According to the wide-angle video image, a wide-angle field of view is established in a polar coordinate system with the optical center as the origin;

[0014] According to the telephoto video image, a telephoto field of view is established in a rectangular coordinate system with the optical center as the origin;

[0015] Determining an overlapping area and a non-overlapping area according to the wide-angle field of view and the telephoto field of view;

[0016] Adjusting the focal length parameter of the wide-angle main camera according to the area ratio of the overlapping area to the non-overlapping area, and updating the wide-angle field of view according to the adjusted focal length parameter, and obtaining an updated wide-angle field of view until the area ratio is greater than a preset area ratio threshold;

[0017] The overlapping area and the non-overlapping area are re-determined according to the updated wide-angle field of view, and the overlapping area and the non-overlapping area are integrated into the field of view complementary area.

[0018] Optionally, the non-overlapping area includes a wide-angle exclusive area and a telephoto exclusive area;

[0019] The determining of overlapping areas and non-overlapping areas according to the wide-angle field of view and the telephoto field of view includes:

[0020] Converting the updated wide-angle field of view to a rectangular coordinate system, and fusing the overlapping area of ​​the wide-angle field of view and the telephoto field of view in the rectangular coordinate system by weighted averaging to obtain a fused overlapping area;

[0021] Mapping each pixel point in the wide-angle exclusive area to a rectangular coordinate system using an initial mapping matrix to obtain a first pixel point; and inversely mapping each pixel point in the telephoto exclusive area to a polar coordinate system using the initial mapping matrix to obtain a second pixel point;

[0022] Performing weighted fusion on the first pixel and the second pixel using an initial fusion weight to obtain a non-overlapping area after initial fusion;

[0023] Constructing a global energy constraint function based on the non-overlapping areas after the initial fusion; wherein the global energy constraint function is used to constrain the smoothness and consistency of the image fusion process;

[0024] By optimizing and solving the global energy constraint function, a fusion weight and a mapping matrix are obtained under minimization of the global energy constraint function;

[0025] The non-overlapping area after the initial fusion is optimized according to the fusion weight and the mapping matrix to obtain an optimized non-overlapping area.

[0026] Optionally, the performing target detection on the target person in the telephoto video image and tracking the detected target person to obtain a motion area of ​​the target person includes:

[0027] The YOLO target detection algorithm is used to detect the target person in the telephoto video image and obtain the target person's location information and bounding box;

[0028] Calculating an optical flow vector of the target person between adjacent frames in the telephoto video image using an optical flow method based on the position information and the bounding box;

[0029] Tracking the bounding box displacement trajectory points of the target person detected by the target according to the optical flow vector, and determining the motion trajectory of the target person according to the bounding box displacement; wherein the motion trajectory includes a plurality of bounding box displacement trajectory points;

[0030] performing expansion calculation on each of the bounding box displacement trajectory points according to the motion trajectory to obtain an expansion neighborhood of each of the bounding box displacement trajectory points;

[0031] Connected domain connection processing is performed on the expanded neighborhood of each of the bounding box displacement trajectory points to obtain an expanded motion area, and a valid area in the expanded motion area is screened using a density threshold to obtain the motion area of ​​the target person.

[0032] Optionally, performing expansion calculation on each of the bounding box displacement trajectory points according to the motion trajectory to obtain an expansion neighborhood of each of the bounding box displacement trajectory points includes:

[0033] According to each of the bounding box displacement trajectory points, convert the bounding box displacement trajectory point into a spatiotemporal point in the spatiotemporal domain, and determine the density of the spatiotemporal point;

[0034] determining an expansion amplitude of the bounding box displacement trajectory points according to the density of the space-time points;

[0035] determining an expansion radius of the bounding box displacement trajectory point according to the expansion amplitude and a velocity vector in the optical flow vector;

[0036] According to the expansion radius, an expansion neighborhood corresponding to the bounding box displacement trajectory point is determined; wherein the expansion neighborhood is a circular area drawn with the bounding box displacement trajectory point as the center and the expansion radius.

[0037] Optionally, the multi-head attention-based convolutional network extracts behavioral features of the target person in the updated complementary area of ​​the field of view, including:

[0038] Inputting the updated target person image within the complementary area of ​​the field of view into a multi-head attention-based convolutional network model; wherein the multi-head attention-based convolutional network model includes a plurality of self-attention heads including a convolutional network;

[0039] Calculating the correlation weights between different trajectory points in the target person image through each of the self-attention heads to obtain a correlation weight matrix;

[0040] Extracting features of the target person at each of the trajectory points from the target person image through the convolutional network, and performing weighted summation on all features in the target person image according to the correlation weight matrix to obtain a weighted feature map;

[0041] The weighted feature maps are spliced ​​and processed through a fully connected layer to obtain a behavior feature vector of the target person; wherein the behavior feature vector is used to characterize the behavior pattern of the target person.

[0042] Optionally, the method further comprises:

[0043] Collect historical behavioral feature samples of workers in tunnel construction videos and behavioral anomaly categories corresponding to the historical behavioral feature samples;

[0044] Constructing a training set and a test set based on the historical behavior feature samples and the behavior abnormality categories corresponding to the historical behavior feature samples;

[0045] Constructing an initial behavior prediction classification model, wherein the initial behavior prediction classification model adopts a deep learning algorithm;

[0046] Using the training set to train the behavior prediction classification model to obtain a behavior prediction classification model;

[0047] The behavior prediction classification model is tested using a test set, and based on the test results, the network parameters of the behavior prediction classification model are optimized to obtain an optimized behavior prediction classification model.

[0048] In a second aspect, the present invention further provides a real-time abnormal behavior recognition system for tunnel construction videos, which includes a wide-angle main camera and a telephoto auxiliary camera set in any of multiple target construction areas of the tunnel, including:

[0049] An image acquisition module, configured to respectively capture a wide-angle video image and a telephoto video image of the target construction area through the wide-angle main camera and the telephoto auxiliary camera;

[0050] a field of view combining module, configured to determine a wide-angle field of view and a telephoto field of view based on the wide-angle video image and the telephoto video image, and combine the wide-angle field of view and the telephoto field of view to obtain a complementary field of view area;

[0051] a motion region determination module, configured to detect a target person in the telephoto video image and track the detected target person to obtain a motion region of the target person;

[0052] An area determination module, configured to determine whether the motion area completely falls within the complementary field of view area;

[0053] a blind spot re-shooting module, configured to re-shoot the motion blind spot using the telephoto auxiliary camera when the motion area does not completely fall within the complementary field of view area, and update the re-shot motion blind spot to the complementary field of view area; wherein the motion blind spot is the portion of the motion area that does not fall within the complementary field of view area;

[0054] A feature extraction module, configured to extract the behavioral features of the target person within the updated complementary area of ​​the field of view based on a multi-head attention convolutional network;

[0055] The behavior anomaly identification module is used to input the behavior characteristics into a pre-trained behavior prediction classification model to identify whether the behavior of the target person is abnormal.

[0056] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for real-time abnormal behavior identification in tunnel construction videos as described in the first aspect.

[0057] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the method for real-time abnormal behavior identification in tunnel construction videos as described in the first aspect.

[0058] It can be seen from the above technical solutions that the present invention, with the help of the respective advantages of the wide-angle main camera and the telephoto auxiliary camera, constructs a wide-angle field of view and a telephoto field of view respectively from the wide-angle video images and telephoto video images captured by the two, and combines the wide-angle field of view and the telephoto field of view to obtain a field of view complementary area, thereby forming a complete and continuous field of view complementary area. At the same time, the target person in the telephoto video image is detected, and the detected target person is tracked to obtain the motion area of ​​the target person, and it is judged whether the motion area completely falls within the field of view complementary area. If the motion area does not completely fall within the field of view complementary area, it indicates that there is a motion blind spot, and the motion blind spot is re-shot by the telephoto auxiliary camera, and the re-shot motion blind spot is updated to the field of view complementary area. The behavioral characteristics of the target person in the updated field of view complementary area are extracted based on the multi-head attention convolutional network, and the behavioral characteristics are input into a pre-trained behavior prediction classification model to identify whether the behavior of the target person is abnormal, thereby effectively solving the blind spot vision problem, improving the accuracy of real-time monitoring of tunnel construction video, and effectively avoiding target omission. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 A flowchart of a method for real-time abnormal behavior recognition in tunnel construction videos provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of the structure of a real-time abnormal behavior recognition system for tunnel construction videos provided by an embodiment of the present invention;

[0062] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0064] An embodiment of the present application provides a method for real-time abnormal behavior recognition in tunnel construction videos, wherein a wide-angle main camera and a telephoto auxiliary camera are set in any one of multiple target construction areas in the tunnel.

[0065] Because tunnels are long and narrow, this application divides the tunnel into multiple target construction areas along its length, and sets up a wide-angle main camera and a telephoto auxiliary camera in each area. The wide-angle main camera is responsible for capturing a wide range of video images to cover the entire target construction area and ensure a broad field of view; while the telephoto auxiliary camera focuses on capturing details at a distance, allowing for more detailed observation of specific areas or targets. Through the collaborative work of the wide-angle main camera and the telephoto auxiliary camera, comprehensive monitoring of the tunnel construction area can be achieved, effectively reducing blind spot vision issues.

[0066] In specific implementation, Figure 1 As shown, the embodiment of the present application provides a method for real-time abnormal behavior recognition in tunnel construction videos, comprising:

[0067] 101. Use the wide-angle main camera and the telephoto auxiliary camera to collect wide-angle video images and telephoto video images in the target construction area respectively.

[0068] The positioning of the wide-angle main camera and telephoto auxiliary camera is flexibly adjusted based on the actual conditions of the tunnel construction area, ensuring that both wide-angle and telephoto video images fully cover the target construction area and effectively reduce blind spots. The wide-angle main camera is typically positioned higher to capture a wider field of view, while the telephoto auxiliary camera is positioned to capture key details based on specific monitoring needs. This arrangement allows the wide-angle main camera and telephoto auxiliary camera to complement each other and achieve comprehensive monitoring of the tunnel construction area.

[0069] 102. Determine a wide-angle field of view and a telephoto field of view based on the wide-angle video image and the telephoto video image, and combine the wide-angle field of view and the telephoto field of view to obtain a complementary field of view area.

[0070] The wide-angle field of view is formed by the video images captured by the wide-angle main camera. Its characteristic is its wide field of view, which can cover a large construction area. The telephoto field of view is formed by the video images captured by the telephoto auxiliary camera. Its advantage is that it can capture details at a distance and conduct precise observation of specific targets. By combining the wide-angle field of view and the telephoto field of view, a complementary field of view area that is both broad and detailed can be obtained. This area not only covers the entire target construction area, but also can clearly present key details, thereby effectively improving the accuracy and comprehensiveness of monitoring. Generally speaking, the wide-angle field of view is wider than the telephoto field of view.

[0071] The field of view integration process takes into account both the overlapping and non-overlapping areas of the wide-angle and telephoto fields of view. By organically combining information from the two fields of view, a complete and continuous complementary field of view is formed. This enables comprehensive monitoring of the tunnel construction area, effectively reducing blind spots and improving construction safety and efficiency.

[0072] 103. Perform target detection on the target person in the telephoto video image, and track the detected target person to obtain a motion area of ​​the target person.

[0073] The target detection algorithm used can be a deep learning algorithm, such as the YOLO (You Only Look Once) family of algorithms. These algorithms offer fast detection speed and high accuracy, making them suitable for real-time detection of target personnel in tunnel construction videos. During target detection, the algorithm traverses each pixel in the video image and calculates its eigenvalues ​​to determine whether the pixel belongs to the target person. Once a target person is detected, the algorithm immediately records their location and bounding box.

[0074] Target tracking is to continuously track the detected target person to obtain his / her motion trajectory. In the present invention, the optical flow method can be used to achieve target tracking. The optical flow method is a method based on the spatiotemporal gradient of pixel intensity data in an image sequence, which can calculate the motion information of the target in the image. By calculating the optical flow vector of the target person between adjacent frames, the target person's bounding box displacement trajectory points can be tracked to obtain his / her motion trajectory. The acquisition of the motion trajectory is crucial for the subsequent identification of behavioral anomalies because it can reflect the motion state and behavior pattern of the target person.

[0075] 104. Determine whether the motion area completely falls within the complementary field of view area.

[0076] 105. When the motion area does not completely fall into the complementary field of view area, the telephoto auxiliary camera is used to retake the motion blind area, and the retaken motion blind area is updated to the complementary field of view area; wherein the motion blind area is the part of the motion area that does not fall into the complementary field of view area.

[0077] If the movement area does not completely fall within the complementary field of view area, it indicates a motion blind spot, meaning that part of the target person's movement trajectory is not monitored. In this case, it is necessary to adjust the parameters of the telephoto auxiliary camera to recapture the motion blind spot to obtain complete motion information. The recaptured motion blind spot image will be updated to the complementary field of view area, thereby achieving comprehensive monitoring of the target person's movement trajectory. This design can effectively avoid safety hazards caused by monitoring blind spots and improve the safety and efficiency of tunnel construction.

[0078] 106. The multi-head attention-based convolutional network extracts the behavioral features of the target person in the updated complementary area of ​​the field of view.

[0079] Among them, multi-head attention is a feature extraction method that effectively captures behavioral details of the target person. It extracts key behavioral features by performing in-depth analysis of the target person's image within the updated complementary field of view. In practice, the multi-head attention mechanism segments the target person's image into multiple regions, each of which independently calculates its correlation weight with the other regions. These weights reflect the degree of association between different regions, helping the model to more accurately understand the target person's behavioral patterns. Through steps such as weighted summation and feature concatenation, the multi-head attention mechanism generates a feature vector rich in behavioral information, providing strong support for subsequent identification of behavioral anomalies. The application of this method enables more accurate identification of abnormal behavior of the target person, improving the accuracy and reliability of monitoring.

[0080] 107. Input the behavioral characteristics into the pre-trained behavior prediction classification model to identify whether the target person's behavior is abnormal.

[0081] The behavior prediction and classification model, built based on a deep learning algorithm, accurately classifies input behavioral features to determine whether the target individual's behavior is abnormal. During the training phase, the model learns from a large number of historical behavioral feature samples and their corresponding abnormal behavior categories, continuously optimizing network parameters to improve classification accuracy and generalization. In practical application, the extracted behavioral features of the target individual are input into the model, which quickly outputs a judgment result indicating whether the target individual's behavior is abnormal. If abnormal behavior is identified, the system immediately triggers an alarm mechanism, notifying relevant personnel for timely action, thereby effectively preventing safety accidents and ensuring the safe and smooth progress of tunnel construction.

[0082] It should be noted that this application uses the respective advantages of the wide-angle main camera and the telephoto auxiliary camera to construct a wide-angle field of view and a telephoto field of view respectively from the wide-angle video images and telephoto video images captured by the two, and combines the wide-angle field of view and the telephoto field of view to obtain a field of view complementary area, thereby forming a complete and continuous field of view complementary area. At the same time, the target person in the telephoto video image is detected, and the detected target person is tracked to obtain the motion area of ​​the target person, and it is judged whether the motion area completely falls within the field of view complementary area. If the motion area does not completely fall within the field of view complementary area, it means that there is a motion blind spot, and the motion blind spot is re-shot by the telephoto auxiliary camera, and the re-shot motion blind spot is updated to the field of view complementary area. The behavioral characteristics of the target person in the updated field of view complementary area are extracted based on the multi-head attention convolutional network, and the behavioral characteristics are input into the pre-trained behavior prediction classification model to identify whether the behavior of the target person is abnormal, thereby effectively solving the blind spot vision problem, improving the accuracy of real-time monitoring of tunnel construction video, and effectively avoiding target omission.

[0083] In some embodiments, the wide-angle field of view and the telephoto field of view are determined based on the wide-angle video image and the telephoto video image, and the wide-angle field of view and the telephoto field of view are combined to obtain a complementary field of view area, including:

[0084] 201. Based on the wide-angle video image, a wide-angle field of view is established in a polar coordinate system with the optical center as the origin.

[0085] Among them, with the optical center (H1, 0) as the origin, the fan-shaped wide-angle field of view in the polar coordinate system is established as: Where, For wide-angle field of view, is the radius, is the angle, is the initial angle.

[0086] 202. Based on the telephoto video image, a telephoto field of view is established in a rectangular coordinate system with the optical center as the origin.

[0087] Among them, the optical center (X c , Y c ) as the origin, the rectangular telephoto field of view in the rectangular coordinate system is established as: Where, For the telephoto field of view, are the horizontal and vertical coordinates of the telephoto field of view, is the lower limit of the horizontal axis, 、 are the lower and upper limits of the vertical axis respectively.

[0088] 203. Determine overlapping areas and non-overlapping areas based on the wide-angle field of view and the telephoto field of view.

[0089] The overlapping area is the portion covered by both the wide-angle and telephoto fields of view, while the non-overlapping area is the unique field of view of each camera. This precise demarcation of overlapping and non-overlapping areas provides a clearer understanding of the respective monitoring ranges of the wide-angle main camera and the telephoto auxiliary camera, as well as their complementary relationship.

[0090] Specifically, step 203 includes:

[0091] Step 2031: convert the updated wide-angle field of view into a rectangular coordinate system, and fuse the overlapping area of ​​the wide-angle field of view and the telephoto field of view in the rectangular coordinate system by weighted averaging to obtain a fused overlapping area.

[0092] The updated wide-angle field of view is converted from a polar coordinate system to a rectangular coordinate system, facilitating integration with the telephoto field of view in the same coordinate system. Once the wide-angle field of view is obtained in the rectangular coordinate system, it is compared with the telephoto field of view to determine their overlapping area. This overlapping area is the area covered by both the wide-angle and telephoto fields of view. This area allows for both the wide-angle field of view's expansive view and the fine details of the telephoto field of view, making it extremely valuable for surveillance.

[0093] For overlapping areas, the present invention uses a weighted average fusion method. Specifically, the pixel values ​​of the wide-angle and telephoto fields of view in the overlapping area are weighted averaged according to certain weights to obtain the fused overlapping area. This method can fully utilize the information of both fields of view, so that the fused overlapping area can achieve better performance in terms of wide field of view and detail clarity.

[0094] Step 2032: Map each pixel point in the wide-angle exclusive area to a rectangular coordinate system using an initial mapping matrix to obtain a first pixel point; and de-map each pixel point in the telephoto exclusive area to a polar coordinate system using the initial mapping matrix to obtain a second pixel point.

[0095] Here, each pixel in the wide-angle exclusive area is set as I p ( ) The telephoto exclusive area in the rectangular coordinate system is I c (x c ,y c ), the non-overlapping area is defined as:

[0096] Wide-angle exclusive area:

[0097] Telephoto exclusive area:

[0098] By introducing the initial mapping matrix, each pixel in the wide-angle exclusive area is mapped to the rectangular coordinate system, and the first pixel is obtained as: Where, is the first pixel, is the initial mapping matrix, and T is the matrix transpose.

[0099] By introducing the initial mapping matrix, each pixel point in the telephoto exclusive area is inversely mapped to the polar coordinates, and the second pixel point is obtained as: Where, is the second pixel.

[0100] Among them, by introducing the scale adjustment parameter ∈(0,1],construct the initial mapping matrix as: Where,

[0101] is the scale scaling factor, is the rotation angle adjustment coefficient, 、 is the translation parameter, is the scale adjustment parameter and is determined by the spatial location of the non-overlapping regions.

[0102] in, Where, 、 They are the initial scaling factor and the target scaling factor respectively.

[0103] Where, is the initial angle, is the angle adjustment amount.

[0104] Step 2033: Perform weighted fusion on the first pixel and the second pixel using the initial fusion weight to obtain a non-overlapping area after initial fusion.

[0105] Among them, the initial fusion weight is calculated by combining the gradient and distance information. The initial fusion weight is: Where, is the wide-angle field of view fusion weight, is the telephoto field of view fusion weight, 、 is the gradient threshold adjustment parameter, 、 is the image gradient, representing the texture complexity, is the distance from the polar coordinate point to the boundary of the overlapping area, is the distance attenuation coefficient, is the distance from the rectangular coordinate point to the boundary of the overlapping area.

[0106] Perform weighted fusion through the initial fusion weight to obtain the non-overlapping area after initial fusion The set of pixels in is: Where, The pixels in the telephoto exclusive area are mapped back to the pixel images in polar coordinates. is the pixel image of the telephoto exclusive area in the rectangular coordinate system, The pixels in the wide-angle exclusive area are mapped to the pixel image in the rectangular coordinate system. It is the pixel image of the wide-angle exclusive area in polar coordinates.

[0107] Step 2034: Construct a global energy constraint function based on the non-overlapping areas after the initial fusion.

[0108] Among them, the global energy constraint function is used to constrain the smoothness and consistency of the image fusion process.

[0109] The global energy constraint function is:

[0110]

[0111] Where, is the global energy, is the regularization coefficient, which controls the gradient smoothness.

[0112] Step 2035: By optimizing and solving the global energy constraint function, the fusion weight and mapping matrix under the condition of minimizing the global energy constraint function are obtained.

[0113] An iterative optimization algorithm, such as gradient descent or the conjugate gradient method, is used to optimize the global energy constraint function. During each iteration, the global energy is calculated based on the current fusion weights and mapping matrix. These weights and mapping matrix are then updated using gradient information to gradually reduce the global energy until convergence is achieved or the preset number of iterations is reached. Ultimately, the optimal fusion weights and mapping matrix are obtained while minimizing the global energy constraint function. These optimal parameters ensure smoothness and consistency during the image fusion process, resulting in a fused image with superior visual field breadth and detail clarity.

[0114] Step 2036: Optimize the non-overlapping area after the initial fusion according to the fusion weight and the mapping matrix to obtain an optimized non-overlapping area.

[0115] Among them, according to the final fusion weights and mapping matrix, the non-overlapping areas after the initial fusion are further optimized. The purpose of this step is to ensure that the fusion effect of the non-overlapping areas is more natural and accurate, avoiding obvious stitching marks or information loss. By fine-tuning the fusion weights, the information contribution of the wide-angle field of view and the telephoto field of view in the non-overlapping areas can be balanced, making the fused image more consistent and coherent in the overall visual effect. At the same time, using the mapping matrix to accurately map the pixels can further reduce the errors introduced by coordinate transformation and improve the accuracy of image fusion. The non-overlapping areas after optimization will be able to better merge with the overlapping areas to form a complete and high-quality field of view complementary area.

[0116] 204. Adjust the focal length parameters of the wide-angle main camera according to the area ratio of the overlapping area and the non-overlapping area, and update the wide-angle field of view according to the adjusted focal length parameters until the area ratio is greater than a preset area ratio threshold, thereby obtaining an updated wide-angle field of view.

[0117] Since the overlapping area represents the area covered by both the wide-angle main camera and the telephoto auxiliary camera, its area ratio reflects the degree of overlap between the two cameras' fields of view. To achieve better field-of-view complementarity, the present invention dynamically adjusts the focal length parameters of the wide-angle main camera based on the area ratio of the overlapping and non-overlapping areas.

[0118] Specifically, the area ratio of the overlapping area and the non-overlapping area is first calculated, and then a determination is made as to whether the ratio meets a preset area ratio threshold. If not (the area ratio threshold is small, indicating a small overlapping area, meaning the wide-angle main camera's shooting needs to be optimized), the focal length parameters of the wide-angle main camera are adjusted based on the difference in area ratio to expand or reduce the range of the wide-angle field of view, thereby increasing or reducing the area of ​​the overlapping area. After adjusting the focal length parameters, the wide-angle field of view is updated, and the area ratio is recalculated until the area ratio is greater than the preset area ratio threshold. In this way, by continuously adjusting the focal length parameters of the wide-angle main camera, the area ratio of the overlapping area and the non-overlapping area can be optimized, thereby ensuring the quality and monitoring effect of the complementary field of view area. After obtaining the updated wide-angle field of view, it is combined again with the telephoto field of view to obtain an updated complementary field of view area, providing more accurate and reliable image information for subsequent target detection and behavioral anomaly identification.

[0119] 205. Redetermine the overlapping area and the non-overlapping area based on the updated wide-angle field of view, and integrate the overlapping area and the non-overlapping area into a complementary field of view area.

[0120] Among them, after obtaining the updated wide-angle field of view, it is necessary to re-determine the overlapping and non-overlapping areas. The purpose of this step is to ensure that the division of overlapping and non-overlapping areas remains accurate after the focal length parameters are adjusted, so as to provide a reliable image basis for subsequent target detection and behavioral anomaly recognition. Specifically, based on the relative positional relationship between the updated wide-angle field of view and the telephoto field of view, the overlapping and non-overlapping parts between them are recalculated, and the overlapping and non-overlapping areas are spliced ​​together to obtain the complementary field of view area.

[0121] In some embodiments, detecting a target person in a telephoto video image and tracking the detected target person to obtain a motion area of ​​the target person may include:

[0122] Step S301: Use the YOLO target detection algorithm to detect the target person in the telephoto video image to obtain the target person's position information and bounding box.

[0123] The YOLO (You Only Look Once) object detection algorithm is a deep learning-based object detection algorithm that can simultaneously predict the location and category of an object in a single forward propagation, offering the advantages of fast detection speed and high accuracy. In this paper, the YOLO object detection algorithm is used to detect objects in telephoto video images, enabling rapid and accurate identification of target persons in telephoto video images and obtaining their location information and bounding boxes.

[0124] Step S302: Calculate the optical flow vector of the target person between adjacent frames in the telephoto video image using the optical flow method based on the position information and the bounding box.

[0125] Among them, the optical flow method is a method for describing the movement of pixels in an image. By analyzing the movement trajectory of pixels in an image sequence, it can calculate the movement speed and direction of the target person between adjacent frames. In the present invention, the optical flow method is used to calculate the optical flow vector of the target person between adjacent frames in a telephoto video image, which can track the movement trajectory of the target person and provide basic data for subsequent behavioral abnormality identification. Specifically, first, based on the position information and bounding box of the target person, the target area to be tracked is determined, and then the optical flow method is used to calculate the optical flow vector of the pixel points in the area to obtain the movement trajectory of the target person. The optical flow vector includes the optical flow direction and the optical flow speed.

[0126] Step S303: Track the bounding box displacement trajectory points of the target person detected by the target according to the optical flow vector, and determine the motion trajectory of the target person according to the bounding box displacement; wherein the motion trajectory includes multiple bounding box displacement trajectory points;

[0127] After tracking the target person's bounding box displacement trajectory points, these points need to be connected to form the target person's motion trajectory. The motion trajectory is the path of the target person's continuous movement in the video image, which reflects the target person's behavioral patterns and movement characteristics. By analyzing the motion trajectory, we can further determine whether the target person has exhibited any abnormal behavior, such as sudden acceleration, deceleration, or changes in direction.

[0128] Step S304: performing expansion calculation on each bounding box displacement trajectory point according to the motion trajectory to obtain the expansion neighborhood of each bounding box displacement trajectory point.

[0129] Dilation is a morphological operation that expands the target region to fill holes or small gaps within it, making it more complete and continuous. In this paper, dilation is performed on each bounding box displacement trajectory point to obtain its dilated neighborhood, further expanding the detection range of the target person and improving the accuracy and robustness of target detection.

[0130] Specifically, an expansion radius is set around each bounding box displacement trajectory point, and all pixels within this radius are considered part of the target person, resulting in an expanded target area. The expanded neighborhood is the area between the expanded target area and the original bounding box. By analyzing the expanded neighborhood, we can further determine the target person's behavior and motion characteristics, such as whether the target person has collided with other objects or entered a prohibited area.

[0131] Step S305: Perform connected domain connection processing on the expanded neighborhood of each bounding box displacement trajectory point to obtain an expanded motion area, and filter the valid area in the expanded motion area using a density threshold to obtain the motion area of ​​the target person.

[0132] Among them, connected domain connection processing is an image processing technology that can connect adjacent pixels or areas with similar attributes in an image to form a larger connected area.

[0133] In this paper, the expanded neighborhood of each bounding box displacement trajectory point is connected by connecting connected regions. This connects the target person's motion trajectory in the video image to form a complete motion region. However, due to the presence of noise or interference in the image, the expanded motion region may contain some invalid or false regions.

[0134] Therefore, it is necessary to remove these invalid areas through density threshold screening to obtain the real motion area of ​​the target person.

[0135] Specifically, the density of the expanded motion region—the number of pixels per unit area—is first calculated. Regions with a density below a preset threshold are then deemed invalid and removed. This results in the target person's true motion region, further improving the accuracy and robustness of target detection and providing more reliable image information for subsequent behavioral anomaly identification. Once the target person's motion region is determined, further analysis and processing can be performed, such as calculating characteristic parameters such as speed, direction, and trajectory, to comprehensively describe and identify the target person's behavioral patterns and motion characteristics. These characteristic parameters can then serve as input data for subsequent behavioral anomaly identification, determining whether the target person is exhibiting abnormal behavior.

[0136] In some embodiments, an expansion calculation is performed on each bounding box displacement trajectory point according to the motion trajectory to obtain an expansion neighborhood of each bounding box displacement trajectory point, including:

[0137] Step S3041: According to each bounding box displacement trajectory point, the bounding box displacement trajectory point is converted into a spatiotemporal point in the spatiotemporal domain, and the density of the spatiotemporal point is determined.

[0138] Among them, the movement trajectory of the target person is assumed to be: Where, is the position in rectangular coordinates, is the timestamp, and N is the number of trajectory points.

[0139] The influence range of the trajectory point in the spatiotemporal domain is calculated by the spatiotemporal joint Gaussian kernel function: Where, is the influence range in the space-time domain, 、 is the spatial dimension bandwidth parameter, which controls the attenuation rate of density due to spatial distance. is the bandwidth parameter in the time dimension, is the horizontal coordinate difference between the i-th trajectory point and the i-1-th trajectory point, is the difference in ordinate between the i-th trajectory point and the i-1-th trajectory point, is the time difference between the i-th trajectory point and the i-1-th trajectory point.

[0140] Among them, the kernel function smoothly diffuses the spatiotemporal influence of the trajectory points through Gaussian distribution, where the spatial dimension affects the intensity by scale attenuation, and the time dimension assigns different weights according to the time interval, ensuring that the trajectory points closer to the current moment contribute more to the density calculation, thereby accurately characterizing the aggregation characteristics of the trajectory points in the spatiotemporal domain.

[0141] For any point in space and time , its density is estimated as: Where, Spacetime point density.

[0142] Step S3042: Determine the expansion amplitude of the bounding box displacement trajectory points according to the density of the space-time points.

[0143] The expansion amplitude refers to the size of the expanded area when the bounding box displacement trajectory points are expanded. In the present invention, the expansion amplitude is determined according to the density of the space-time points, which can achieve adaptive expansion processing of areas with different densities. Specifically, a higher-density area indicates that the target person appears more frequently in the area and may have more complex motion patterns or behavioral characteristics. Therefore, a larger expansion amplitude is required to fully cover the target person's motion range; while a lower-density area indicates that the target person appears less frequently in the area and has a relatively simple motion pattern. Therefore, a smaller expansion amplitude can be used to avoid inaccurate target detection caused by over-expansion.

[0144] The expansion amplitude is calculated as:

[0145] Where, is the expansion amplitude, is the time weight sharpening parameter, which controls the weight increase rate of recent trajectories; is the latest timestamp in the trajectory, ensuring that the more recent trajectories have a greater impact on the expansion.

[0146] Step S3043: Determine the expansion radius of the bounding box displacement trajectory point according to the expansion amplitude and the velocity vector in the optical flow vector.

[0147] Among them, the expansion radius refers to the radius of the circular area expanded with the point as the center when the bounding box displacement trajectory point is expanded. In the present invention, the expansion radius is determined based on the expansion amplitude and the velocity vector in the optical flow vector, which can achieve accurate description and expansion of the target person's motion trajectory. Specifically, the size of the expansion radius should match the target person's movement speed and direction to ensure that the expanded movement area can accurately cover the target person's actual movement range. At the same time, since the target person's movement speed and direction may change over time, it is necessary to dynamically adjust the expansion radius according to the velocity vector in the optical flow vector to adapt to the target person's movement changes.

[0148] The expansion radius of each trajectory point is determined by the density and historical velocity: Where, For trajectory points The expansion radius, is the base expansion radius (e.g. 1.5 times the average target size by default), 、 are the density and velocity weight coefficients, is the velocity vector in the optical flow vector, which is estimated by the aforementioned optical flow method.

[0149] Step S3044: Determine the expansion neighborhood corresponding to the bounding box displacement trajectory point according to the expansion radius; wherein the expansion neighborhood is a circular area drawn with the bounding box displacement trajectory point as the center and the expansion radius.

[0150] Among them, since people move dynamically, the potential motion range of the target person in the video image is determined by expanding the neighborhood for subsequent behavioral anomaly recognition.

[0151] In some embodiments, the multi-head attention-based convolutional network extracts behavioral features of the target person within the updated complementary field of view, including:

[0152] Step 601: input the updated target person image within the complementary field of view into a multi-head attention-based convolutional network model; wherein the multi-head attention-based convolutional network model includes a plurality of self-attention heads including a convolutional network;

[0153] Step 602: Calculate the correlation weights between different trajectory points in the target person image through each self-attention head to obtain a correlation weight matrix;

[0154] Among them, the self-attention head generates a correlation weight matrix by calculating the correlation score between any two positions in the input data.

[0155] In the present invention, each self-attention head calculates the correlation weights between different positions in the target person image, thereby obtaining a correlation weight matrix. These matrices reflect the degree of association between different positions in the target person image.

[0156] Specifically, each self-attention head calculates the dot product similarity between any two trajectory point positions based on the trajectory point representation of the target person image and normalizes it into a correlation score through the softmax function. These scores are then used as weights, and a correlation weight matrix is ​​constructed based on the weight corresponding to each trajectory point.

[0157] Step 603: extract the features of the target person at each trajectory point from the target person image through a convolutional network, and perform weighted summation on all features in the target person image according to the correlation weight matrix to obtain a weighted feature map.

[0158] The convolutional network may be a 3D convolutional network (Two-Stream 3D CNN), including a spatial stream network and a temporal stream network. The spatial stream network extracts the static posture features of the target person at each trajectory point, and the temporal stream network extracts the dynamic movement features of the target person at each trajectory point.

[0159] The weighted feature map reflects the importance of features at different locations in the target person's image. In this invention, a weighted summation of behavioral features in the target person's image is performed according to the correlation weight matrix. This can highlight key areas related to the target person's behavioral features and suppress irrelevant background information, thereby improving the accuracy and robustness of behavioral feature extraction.

[0160] Specifically, for each location in the target person's image, the feature value at that location is multiplied by the corresponding weight based on its corresponding correlation weight. The feature values ​​at all locations are summed to produce a weighted feature map. This weighted feature map not only contains the target person's features at each trajectory point, but also incorporates the correlation information between the target person's locations at different trajectory points, providing a richer and more accurate feature representation for subsequent behavioral anomaly identification. By analyzing and processing this weighted feature map, the target person's behavioral characteristics, such as posture and movement, can be further extracted, thereby achieving a comprehensive description and identification of the target person's behavioral patterns.

[0161] Step 604: concatenate the weighted feature maps and process them through a fully connected layer to obtain a behavioral feature vector of the target person; wherein the behavioral feature vector is used to characterize the behavioral pattern of the target person.

[0162] The fully connected layer is a neural network layer that maps the input feature map to a fixed-size output vector. In this invention, the weighted feature maps are concatenated and processed through the fully connected layer to obtain the target person's behavioral feature vector. This vector contains the target person's behavioral characteristics, such as posture and movement, and can be used for subsequent behavioral anomaly identification.

[0163] In some embodiments, the method further comprises:

[0164] Step S11: Collect historical behavior feature samples of workers in the tunnel construction video and behavioral abnormality categories corresponding to the historical behavior feature samples;

[0165] Step S12: construct a training set and a test set based on the historical behavior feature samples and the behavior abnormality categories corresponding to the historical behavior feature samples;

[0166] Step S13: constructing an initial behavior prediction classification model, wherein the initial behavior prediction classification model adopts a deep learning algorithm;

[0167] Step S14: training the behavior prediction classification model using the training set to obtain a behavior prediction classification model;

[0168] Step S15: Use the test set to test the behavior prediction classification model, and optimize the network parameters of the behavior prediction classification model based on the test results to obtain an optimized behavior prediction classification model.

[0169] The behavior prediction classification model is used to classify and predict the behavior of target personnel in tunnel construction videos. In this paper, a deep learning algorithm is used to construct the initial behavior prediction classification model. This is because deep learning algorithms can automatically learn feature representations from large amounts of data and accurately classify and predict target personnel behavior.

[0170] Specifically, deep learning algorithms can automatically extract characteristic information related to the target person's behavior by learning from historical behavioral feature samples in a training set, and then build a classifier capable of distinguishing different types of behavioral anomalies. During the training process, deep learning algorithms continuously adjust the model's parameters to minimize the error between predicted and actual results, thereby improving the model's classification and prediction performance.

[0171] like Figure 2 As shown, the embodiment of the present application provides a real-time abnormal behavior recognition system for tunnel construction videos, which sets a wide-angle main camera and a telephoto auxiliary camera in any of multiple target construction areas of the tunnel, including:

[0172] The image acquisition module 100 is used to respectively capture wide-angle video images and long-focus video images of the target construction area through a wide-angle main camera and a long-focus auxiliary camera;

[0173] The field of view combining module 200 is used to determine the wide-angle field of view and the telephoto field of view based on the wide-angle video image and the telephoto video image, and combine the wide-angle field of view and the telephoto field of view to obtain a complementary field of view area;

[0174] The motion region determination module 300 is used to detect the target person in the telephoto video image and track the detected target person to obtain the motion region of the target person;

[0175] The region determination module 400 is used to determine whether the motion region completely falls within the complementary field of view region;

[0176] The blind spot re-shooting module 500 is configured to re-shoot the blind spot using the telephoto auxiliary camera when the motion area does not completely fall within the complementary field of view area, and update the re-shot blind spot to the complementary field of view area; wherein the blind spot is the portion of the motion area that does not fall within the complementary field of view area;

[0177] A feature extraction module 600 is used to extract the behavioral features of the target person in the updated complementary area of ​​the field of view based on a multi-head attention convolutional network;

[0178] The behavior anomaly identification module 700 is used to input the behavior characteristics into a pre-trained behavior prediction classification model to identify whether the behavior of the target person is abnormal.

[0179] In some embodiments, the field of view uniting module 200 is configured to:

[0180] According to the wide-angle video image, a wide-angle field of view in a polar coordinate system is established with the optical center as the origin;

[0181] According to the telephoto video image, a telephoto field of view in a rectangular coordinate system is established with the optical center as the origin;

[0182] Determine overlapping areas and non-overlapping areas based on the wide-angle field of view and the telephoto field of view;

[0183] Adjusting the focal length parameters of the wide-angle main camera according to the area ratio of the overlapping area and the non-overlapping area, and updating the wide-angle field of view according to the adjusted focal length parameters until the area ratio is greater than a preset area ratio threshold, thereby obtaining an updated wide-angle field of view;

[0184] The overlapping area and the non-overlapping area are re-determined based on the updated wide-angle field of view, and the overlapping area and the non-overlapping area are integrated into a complementary field of view area.

[0185] In some embodiments, the non-overlapping area includes a wide-angle exclusive area and a telephoto exclusive area;

[0186] Based on the wide-angle field of view and the telephoto field of view, determine the overlapping and non-overlapping areas, including:

[0187] The updated wide-angle field of view is converted to a rectangular coordinate system, and the overlapping area of ​​the wide-angle field of view and the telephoto field of view in the rectangular coordinate system is fused by weighted averaging to obtain a fused overlapping area;

[0188] Mapping each pixel point in the wide-angle exclusive area to a rectangular coordinate system using an initial mapping matrix to obtain a first pixel point; and inversely mapping each pixel point in the telephoto exclusive area to a polar coordinate system using the initial mapping matrix to obtain a second pixel point;

[0189] Perform weighted fusion on the first pixel and the second pixel using an initial fusion weight to obtain a non-overlapping area after initial fusion;

[0190] A global energy constraint function is constructed based on the non-overlapping areas after the initial fusion. The global energy constraint function is used to constrain the smoothness and consistency of the image fusion process.

[0191] By optimizing and solving the global energy constraint function, the fusion weight and mapping matrix under the minimum global energy constraint function are obtained;

[0192] According to the fusion weight and the mapping matrix, the non-overlapping area after the initial fusion is optimized to obtain the optimized non-overlapping area.

[0193] In some embodiments, the motion region determination module 300 is configured to:

[0194] The YOLO target detection algorithm is used to detect the target person in the telephoto video image and obtain the target person's location information and bounding box;

[0195] Based on the position information and bounding box, the optical flow method is used to calculate the optical flow vector of the target person between adjacent frames in the telephoto video image;

[0196] According to the optical flow vector, the bounding box displacement trajectory points of the target person detected by the target are tracked, and the motion trajectory of the target person is determined according to the bounding box displacement; wherein the motion trajectory includes multiple bounding box displacement trajectory points;

[0197] According to the motion trajectory, the expansion calculation is performed on each bounding box displacement trajectory point to obtain the expansion neighborhood of each bounding box displacement trajectory point;

[0198] The expanded neighborhood of each bounding box displacement trajectory point is connected to the connected domain to obtain the expanded motion area, and the valid area in the expanded motion area is filtered by the density threshold to obtain the motion area of ​​the target person.

[0199] In some embodiments, an expansion calculation is performed on each bounding box displacement trajectory point according to the motion trajectory to obtain an expansion neighborhood of each bounding box displacement trajectory point, including:

[0200] According to each bounding box displacement trajectory point, the bounding box displacement trajectory point is converted to a spatiotemporal point in the spatiotemporal domain, and the density of the spatiotemporal point is determined;

[0201] Determine the expansion amplitude of the bounding box displacement trajectory points based on the density of space-time points;

[0202] Determine the expansion radius of the bounding box displacement trajectory point based on the expansion amplitude and the velocity vector in the optical flow vector;

[0203] According to the expansion radius, the expansion neighborhood corresponding to the bounding box displacement trajectory point is determined; wherein the expansion neighborhood is a circular area drawn with the bounding box displacement trajectory point as the center and the expansion radius.

[0204] In some embodiments, the feature extraction module 600 is configured to:

[0205] Inputting the updated target person image within the complementary area of ​​the field of view into a multi-head attention-based convolutional network model; wherein the multi-head attention-based convolutional network model includes multiple self-attention heads including a convolutional network;

[0206] The correlation weights between different trajectory points in the target person image are calculated through each self-attention head to obtain the correlation weight matrix;

[0207] The convolutional network is used to extract the features of the target person at each trajectory point from the target person image, and all the features in the target person image are weighted and summed according to the correlation weight matrix to obtain a weighted feature map;

[0208] The weighted feature maps are spliced ​​and processed through a fully connected layer to obtain the target person's behavioral feature vector; the behavioral feature vector is used to characterize the target person's behavioral pattern.

[0209] In some embodiments, the system further includes a model training module for:

[0210] Collect historical behavioral feature samples of workers in tunnel construction videos and the behavioral anomaly categories corresponding to the historical behavioral feature samples;

[0211] Construct training and test sets based on historical behavior feature samples and the behavioral anomaly categories corresponding to the historical behavior feature samples;

[0212] Construct an initial behavior prediction classification model using a deep learning algorithm;

[0213] The behavior prediction classification model is trained using the training set to obtain the behavior prediction classification model;

[0214] The behavior prediction classification model is tested using the test set. Based on the test results, the network parameters of the behavior prediction classification model are optimized to obtain the optimized behavior prediction classification model.

[0215] like Figure 3 As shown, an embodiment of the present application provides an electronic device, the electronic device 10 includes a memory 20 and a processor 30, the memory 20 stores a computer program, and when the computer program is executed by the processor 30, the processor 30 executes the steps of the real-time abnormal behavior identification method for tunnel construction video in the above embodiment.

[0216] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, the steps of the method for real-time abnormal behavior identification in tunnel construction video as described in the above embodiment are implemented.

[0217] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, electronic devices, and computer storage media can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0218] It should be noted that the user information (including but not limited to user images, user portrait information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0219] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0220] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0221] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, electronic devices, computer storage media and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0222] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0223] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0224] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the method described in each embodiment of the present invention via a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0225] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for real-time abnormal behavior recognition in tunnel construction videos, characterized in that: A wide-angle main camera and a telephoto auxiliary camera are set up in any of the multiple target construction areas of the tunnel, including: respectively collecting a wide-angle video image and a telephoto video image of the target construction area by using the wide-angle main camera and the telephoto auxiliary camera; Determining a wide-angle field of view and a telephoto field of view according to the wide-angle video image and the telephoto video image, and combining the wide-angle field of view and the telephoto field of view to obtain a complementary field of view area, including: According to the wide-angle video image, a wide-angle field of view is established in a polar coordinate system with the optical center as the origin; According to the telephoto video image, a telephoto field of view is established in a rectangular coordinate system with the optical center as the origin; Determining an overlapping area and a non-overlapping area according to the wide-angle field of view and the telephoto field of view; Adjusting the focal length parameter of the wide-angle main camera according to the area ratio of the overlapping area to the non-overlapping area, and updating the wide-angle field of view according to the adjusted focal length parameter, and obtaining an updated wide-angle field of view until the area ratio is greater than a preset area ratio threshold; re-determining the overlapping area and the non-overlapping area according to the updated wide-angle field of view, and integrating the overlapping area and the non-overlapping area into the field of view complementary area; Performing target detection on a target person in the telephoto video image, and tracking the detected target person to obtain a motion area of ​​the target person; Determining whether the motion area completely falls within the complementary field of view area; When the motion area does not completely fall within the complementary field of view area, the telephoto auxiliary camera is used to retake the motion blind area, and the retaken motion blind area is updated to the complementary field of view area; wherein the motion blind area is the portion of the motion area that does not fall within the complementary field of view area; A multi-head attention convolutional network is used to extract the behavioral characteristics of the target person in the updated complementary area of ​​the field of view; The behavioral characteristics are input into a pre-trained behavior prediction classification model to identify whether the behavior of the target person is abnormal.

2. The method for real-time abnormal behavior recognition in tunnel construction videos according to claim 1 is characterized in that: The non-overlapping area includes a wide-angle exclusive area and a telephoto exclusive area; The determining of overlapping areas and non-overlapping areas according to the wide-angle field of view and the telephoto field of view includes: Converting the updated wide-angle field of view to a rectangular coordinate system, and fusing the overlapping area of ​​the wide-angle field of view and the telephoto field of view in the rectangular coordinate system by weighted averaging to obtain a fused overlapping area; Mapping each pixel point in the wide-angle exclusive area to a rectangular coordinate system using an initial mapping matrix to obtain a first pixel point; and inversely mapping each pixel point in the telephoto exclusive area to a polar coordinate system using the initial mapping matrix to obtain a second pixel point; Performing weighted fusion on the first pixel and the second pixel using an initial fusion weight to obtain a non-overlapping area after initial fusion; Constructing a global energy constraint function based on the non-overlapping areas after the initial fusion; wherein the global energy constraint function is used to constrain the smoothness and consistency of the image fusion process; By optimizing and solving the global energy constraint function, a fusion weight and a mapping matrix are obtained under minimization of the global energy constraint function; The non-overlapping area after the initial fusion is optimized according to the fusion weight and the mapping matrix to obtain an optimized non-overlapping area.

3. The method for real-time abnormal behavior recognition in tunnel construction videos according to claim 1 is characterized in that: The detecting the target person in the telephoto video image and tracking the detected target person to obtain the motion area of ​​the target person includes: The YOLO target detection algorithm is used to detect the target person in the telephoto video image and obtain the target person's location information and bounding box; Calculating an optical flow vector of the target person between adjacent frames in the telephoto video image using an optical flow method based on the position information and the bounding box; Tracking the bounding box displacement trajectory points of the target person detected by the target according to the optical flow vector, and determining the motion trajectory of the target person according to the bounding box displacement; wherein the motion trajectory includes a plurality of bounding box displacement trajectory points; performing expansion calculation on each of the bounding box displacement trajectory points according to the motion trajectory to obtain an expansion neighborhood of each of the bounding box displacement trajectory points; Connected domain connection processing is performed on the expanded neighborhood of each of the bounding box displacement trajectory points to obtain an expanded motion area, and a valid area in the expanded motion area is screened using a density threshold to obtain the motion area of ​​the target person.

4. The method for real-time abnormal behavior recognition in tunnel construction videos according to claim 3 is characterized in that: The step of performing expansion calculation on each of the bounding box displacement trajectory points according to the motion trajectory to obtain an expansion neighborhood of each of the bounding box displacement trajectory points includes: According to each of the bounding box displacement trajectory points, convert the bounding box displacement trajectory point into a spatiotemporal point in the spatiotemporal domain, and determine the density of the spatiotemporal point; determining an expansion amplitude of the bounding box displacement trajectory points according to the density of the space-time points; determining an expansion radius of the bounding box displacement trajectory point according to the expansion amplitude and a velocity vector in the optical flow vector; According to the expansion radius, an expansion neighborhood corresponding to the bounding box displacement trajectory point is determined; wherein the expansion neighborhood is a circular area drawn with the bounding box displacement trajectory point as the center and the expansion radius.

5. The method for real-time abnormal behavior recognition in tunnel construction videos according to claim 1 is characterized in that: The multi-head attention-based convolutional network extracts the behavioral features of the target person in the updated complementary area of ​​the field of view, including: Inputting the updated target person image within the complementary area of ​​the field of view into a multi-head attention-based convolutional network model; wherein the multi-head attention-based convolutional network model includes a plurality of self-attention heads including a convolutional network; Calculating the correlation weights between different trajectory points in the target person image through each of the self-attention heads to obtain a correlation weight matrix; Extracting features of the target person at each of the trajectory points from the target person image through the convolutional network, and performing weighted summation on all features in the target person image according to the correlation weight matrix to obtain a weighted feature map; The weighted feature maps are spliced ​​and processed through a fully connected layer to obtain a behavior feature vector of the target person; wherein the behavior feature vector is used to characterize the behavior pattern of the target person.

6. The method for real-time abnormal behavior recognition in tunnel construction videos according to claim 1, characterized in that: Also includes: Collect historical behavioral feature samples of workers in tunnel construction videos and behavioral anomaly categories corresponding to the historical behavioral feature samples; Constructing a training set and a test set based on the historical behavior feature samples and the behavior abnormality categories corresponding to the historical behavior feature samples; Constructing an initial behavior prediction classification model, wherein the initial behavior prediction classification model adopts a deep learning algorithm; Using the training set to train the behavior prediction classification model to obtain a behavior prediction classification model; The behavior prediction classification model is tested using a test set, and based on the test results, the network parameters of the behavior prediction classification model are optimized to obtain an optimized behavior prediction classification model.

7. A real-time abnormal behavior recognition system for tunnel construction videos, characterized by: A wide-angle main camera and a telephoto auxiliary camera are set up in any of the multiple target construction areas of the tunnel, including: An image acquisition module, configured to respectively capture a wide-angle video image and a telephoto video image of the target construction area through the wide-angle main camera and the telephoto auxiliary camera; a field of view combining module, configured to determine a wide-angle field of view and a telephoto field of view based on the wide-angle video image and the telephoto video image, and combine the wide-angle field of view and the telephoto field of view to obtain a complementary field of view area; Field of view joint module for: According to the wide-angle video image, a wide-angle field of view in a polar coordinate system is established with the optical center as the origin; According to the telephoto video image, a telephoto field of view in a rectangular coordinate system is established with the optical center as the origin; Determine overlapping areas and non-overlapping areas based on the wide-angle field of view and the telephoto field of view; Adjusting the focal length parameters of the wide-angle main camera according to the area ratio of the overlapping area and the non-overlapping area, and updating the wide-angle field of view according to the adjusted focal length parameters until the area ratio is greater than a preset area ratio threshold, thereby obtaining an updated wide-angle field of view; Redetermine the overlapping and non-overlapping areas based on the updated wide-angle field of view, and integrate the overlapping and non-overlapping areas into a complementary field of view area; a motion region determination module, configured to detect a target person in the telephoto video image and track the detected target person to obtain a motion region of the target person; An area determination module, configured to determine whether the motion area completely falls within the complementary field of view area; a blind spot re-shooting module, configured to re-shoot the motion blind spot using the telephoto auxiliary camera when the motion area does not completely fall within the complementary field of view area, and update the re-shot motion blind spot to the complementary field of view area; wherein the motion blind spot is the portion of the motion area that does not fall within the complementary field of view area; A feature extraction module, configured to extract the behavioral features of the target person within the updated complementary area of ​​the field of view based on a multi-head attention convolutional network; The behavior anomaly identification module is used to input the behavior characteristics into a pre-trained behavior prediction classification model to identify whether the behavior of the target person is abnormal.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor performs the steps of the method for real-time abnormal behavior identification in tunnel construction video according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the steps of the method for real-time abnormal behavior recognition in tunnel construction video according to any one of claims 1 to 6 are implemented.

Images