Platform jump abnormal behavior identification method and device, equipment and medium

Abstract

The invention provides a platform jump abnormal behavior identification method and device, equipment and a medium, and the method comprises the steps: obtaining continuous video frames, screening candidate segments containing vertical displacement motions based on the obtained continuous video frames, and carrying out the adaptive frame sampling of the candidate segments, and generating an input tensor; performing feature extraction on the input tensor by using an optimized video Swin converter to obtain a multi-scale spatial-temporal feature map; carrying out feature fusion on the obtained multi-scale spatio-temporal feature maps to obtain a comprehensive spatio-temporal feature map; predicting a drop point coordinate of the jump behavior through a drop point rationality network by integrating the spatial-temporal feature map; and matching the coordinates of the drop point with the geometric boundary mask graph, and judging whether the current drop point is located in a preset dangerous area or not so as to identify the jump abnormal behavior.

Description

Technical Field

[0001] This invention relates to the field of anomaly recognition technology, specifically to a method, apparatus, device, and medium for recognizing platform jumping anomaly behavior. Background Technology

[0002] In current industrial settings, the automatic identification of high-risk abnormal behaviors such as personnel jumping on platforms is a critical requirement for safety management. Traditional rule-based or simple motion detection methods struggle to distinguish between jumps and normal movements. While deep learning methods have made progress, they generally suffer from computational complexity, weak generalization, and a lack of behavioral physical semantics. In recent years, VisionTransformer has demonstrated powerful performance in image recognition, and Video Swin Transformer has further extended it to the video domain, achieving efficient spatiotemporal modeling through a 3D sliding window mechanism. However, its general architecture is not optimized for specific high-risk behaviors, resulting in insufficient feature extraction in key spatiotemporal regions, making it difficult to meet the industrial-grade requirements for high-precision, low-latency anomaly identification. Summary of the Invention

[0003] The main objective of this invention is to solve the technical problem in the prior art that the feature extraction of jumping behavior in key spatiotemporal regions is insufficient, making it difficult to meet the industrial-grade requirements for high-precision and low-latency anomaly identification.

[0004] The first aspect of this invention provides a method for identifying abnormal platform jumping behavior, comprising: Acquire continuous video frames, filter candidate segments containing vertical displacement actions based on the acquired continuous video frames, and generate input tensors by adaptive frame sampling of the candidate segments; A multi-scale spatiotemporal feature map is obtained by extracting features from the input tensor using an optimized video Swing transformer. The obtained multi-scale spatiotemporal feature maps are fused to obtain a comprehensive spatiotemporal feature map. The comprehensive spatiotemporal feature map is used to predict the landing coordinates of the jump behavior through the landing point rationality network. The landing coordinates are matched with the geometric boundary mask map to determine whether the current landing point is located in the preset danger zone, thereby identifying abnormal jump behavior.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of acquiring continuous video frames, filtering candidate segments containing vertical displacement actions based on the acquired continuous video frames, and performing adaptive frame sampling on the candidate segments to generate an input tensor includes: Acquire continuous video streams through surveillance cameras; Each frame of video image in the acquired continuous video stream is converted into a standard digital image format that meets preset requirements. Each converted frame contains the complete platform scene area. The inter-frame difference variance method is used to obtain the inter-frame difference variance value for the converted video frames; the inter-frame difference variance value is compared with a set threshold. When the inter-frame difference variance value exceeds the set threshold, it is determined that there is a vertical displacement action between the two consecutive frames; the process is repeatedly triggered to obtain the frame sequence containing the vertical displacement action, and the frame sequence containing the vertical displacement action is marked as a candidate segment; T-frame images are extracted from the candidate segments through adaptive sampling. The size of each frame in the extracted T-frame images is adjusted to H×W, and its RGB three-channel information is preserved. Finally, they are combined to generate an input tensor of T×H×W×3 dimensions.

[0006] Optionally, in a second implementation of the first aspect of the present invention, obtaining the inter-frame difference variance value for the converted video frame using the inter-frame difference variance method includes: selecting two adjacent video images for the converted video frame and converting them into grayscale images; calculating the difference in grayscale values ​​of corresponding pixels in the two grayscale images; and summing the squared values ​​of all differences to obtain the inter-frame difference variance value.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of using an optimized video Swin transformer to extract features from the input tensor to obtain a multi-scale spatiotemporal feature map includes: According to the preset window size, a 3D sliding window is constructed in the spatiotemporal dimension of the input tensor; during the sliding process, the window moves in the time and space dimensions according to the preset step size, dividing the entire input tensor into multiple non-overlapping or partially overlapping 3D window feature blocks. A spatiotemporal jump trajectory template is generated based on a physical parabolic model. Based on the jump trajectory template, for each 3D window feature block, the similarity between each spatiotemporal position within the block and the corresponding position on the trajectory template is calculated, and a spatiotemporal attention bias value is generated based on the similarity. Calculate the original self-attention weights for all feature vectors within each 3D window feature block, and then superimpose the attention bias values ​​onto the original self-attention weights to obtain the enhanced self-attention weights. Based on the enhanced self-attention weights, the feature vectors within the 3D window feature block are weighted and summed to obtain the local spatiotemporal features of the focus jump-takeoff-landing point. The obtained local spatiotemporal features focusing on takeoff-flight-landing point are processed through layer normalization and fully connected layers to obtain enhanced local spatiotemporal features; The enhanced local spatiotemporal features are sampled sequentially using an asymmetric spatiotemporal downsampling strategy and a symmetric downsampling strategy to obtain a multi-scale spatiotemporal feature map; the multi-scale spatiotemporal feature map includes a high-level feature map and a low-level feature map.

[0008] Optionally, in the fourth implementation of the first aspect of the present invention, the step of generating a jump trajectory template based on a physical parabolic model in a spatiotemporal dimension includes: the physical parabolic model is combined with gravitational acceleration and a preset initial jump velocity to calculate a complete spatial coordinate change curve from the take-off point to the highest point of the flight and then to the landing point, and the curve is mapped to the pixel coordinate system of the image to form a jump trajectory template in a spatiotemporal dimension.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of performing feature fusion on the obtained multi-scale spatiotemporal feature maps to obtain a comprehensive spatiotemporal feature map includes: The high-level feature maps in the multi-scale spatiotemporal feature map are upsampled to the same size as the low-level feature maps; and the feature maps of the corresponding levels are added or concatenated element by element, and integrated through a convolutional layer to obtain a fused spatiotemporal feature map of multi-scale information.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the integrated spatiotemporal feature map predicts the landing coordinates of the jump behavior through a landing point rationality network; the landing coordinates are matched with a geometric boundary mask map to determine whether the current landing point is located within a preset danger zone, thereby identifying abnormal jump behavior, including: Obtain the platform's CAD model or pre-calibrated geometric boundary data, and construct a geometric boundary mask based on the obtained platform CAD model or pre-calibrated geometric boundary data; Constructing a landing point rationality network based on convolutional and recurrent neural networks; The comprehensive spatiotemporal feature map is input into the landing point rationality network to predict the landing point coordinates of the jump behavior; The spatiotemporal feature map is processed by global average pooling to obtain a one-dimensional visual feature vector. At the same time, the landing point rationality judgment result is converted into numerical features. The two are concatenated and input into a fully connected layer. The fully connected layer calculates the confidence score of the jump behavior based on the visual feature vector. Combined with whether the landing point is in the danger zone and the risk level of the danger zone, the corresponding danger level is generated.

[0011] A second aspect of the present invention provides a platform jump abnormal behavior recognition device, comprising: The input tensor generation module is used to acquire continuous video frames, filter candidate segments containing vertical displacement actions based on the acquired continuous video frames, and perform adaptive frame sampling on the candidate segments to generate input tensors. The multi-scale spatiotemporal feature map acquisition module is used to extract features from the input tensor using an optimized video Swin transformer to obtain a multi-scale spatiotemporal feature map. The jump abnormal behavior recognition module is used to fuse the obtained multi-scale spatiotemporal feature maps to obtain a comprehensive spatiotemporal feature map; the comprehensive spatiotemporal feature map predicts the landing coordinates of the jump behavior through the landing point rationality network; the landing coordinates are matched with the geometric boundary mask map to determine whether the current landing point is located within the preset danger area, thereby identifying the jump abnormal behavior.

[0012] Optionally, in a first implementation of the second aspect of the present invention, the input tensor generation module includes: Acquire continuous video streams through surveillance cameras; Each frame of video image in the acquired continuous video stream is converted into a standard digital image format that meets preset requirements. Each converted frame contains the complete platform scene area. The inter-frame difference variance method is used to obtain the inter-frame difference variance value for the converted video frames; the inter-frame difference variance value is compared with a set threshold. When the inter-frame difference variance value exceeds the set threshold, it is determined that there is a vertical displacement action between the two consecutive frames; the process is repeatedly triggered to obtain the frame sequence containing the vertical displacement action, and the frame sequence containing the vertical displacement action is marked as a candidate segment; T-frame images are extracted from the candidate segments through adaptive sampling. The size of each frame in the extracted T-frame images is adjusted to H×W, and its RGB three-channel information is preserved. Finally, they are combined to generate an input tensor of T×H×W×3 dimensions.

[0013] Optionally, in a second implementation of the second aspect of the present invention, obtaining the inter-frame difference variance value for the converted video frame using the inter-frame difference variance method includes: selecting two adjacent video images for the converted video frame and converting them into grayscale images; calculating the difference in grayscale values ​​of corresponding pixels in the two grayscale images; and summing the squared values ​​of all differences to obtain the inter-frame difference variance value.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the multi-scale spatiotemporal feature map acquisition module includes: According to the preset window size, a 3D sliding window is constructed in the spatiotemporal dimension of the input tensor; during the sliding process, the window moves in the time and space dimensions according to the preset step size, dividing the entire input tensor into multiple non-overlapping or partially overlapping 3D window feature blocks. A spatiotemporal jump trajectory template is generated based on a physical parabolic model. Based on the jump trajectory template, for each 3D window feature block, the similarity between each spatiotemporal position within the block and the corresponding position on the trajectory template is calculated, and a spatiotemporal attention bias value is generated based on the similarity. Calculate the original self-attention weights for all feature vectors within each 3D window feature block, and then superimpose the attention bias values ​​onto the original self-attention weights to obtain the enhanced self-attention weights. Based on the enhanced self-attention weights, the feature vectors within the 3D window feature block are weighted and summed to obtain the local spatiotemporal features of the focus jump-takeoff-landing point. The obtained local spatiotemporal features focusing on takeoff-flight-landing point are processed through layer normalization and fully connected layers to obtain enhanced local spatiotemporal features; The enhanced local spatiotemporal features are sampled sequentially using an asymmetric spatiotemporal downsampling strategy and a symmetric downsampling strategy to obtain a multi-scale spatiotemporal feature map; the multi-scale spatiotemporal feature map includes a high-level feature map and a low-level feature map.

[0015] Optionally, in the fourth implementation of the second aspect of the present invention, the step of generating a spatiotemporal jump trajectory template based on a physical parabolic model includes: calculating a complete spatial coordinate change curve from the take-off point to the highest point of take-off and then to the landing point using the physical parabolic model, including gravitational acceleration and a preset initial jump velocity, and mapping it to the pixel coordinate system of the image to form a spatiotemporal jump trajectory template.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the step of fusing the obtained multi-scale spatiotemporal feature maps to obtain a comprehensive spatiotemporal feature map includes: The high-level feature maps in the multi-scale spatiotemporal feature map are upsampled to the same size as the low-level feature maps; and the feature maps of the corresponding levels are added or concatenated element by element, and integrated through a convolutional layer to obtain a fused spatiotemporal feature map of multi-scale information.

[0017] Optionally, in a sixth implementation of the second aspect of the present invention, the integrated spatiotemporal feature map predicts the landing coordinates of the jump behavior through a landing point rationality network; the landing coordinates are matched with a geometric boundary mask map to determine whether the current landing point is located within a preset danger zone, thereby identifying abnormal jump behavior, including: Obtain the platform's CAD model or pre-calibrated geometric boundary data, and construct a geometric boundary mask based on the obtained platform CAD model or pre-calibrated geometric boundary data; Constructing a landing point rationality network based on convolutional and recurrent neural networks; The comprehensive spatiotemporal feature map is input into the landing point rationality network to predict the landing point coordinates of the jump behavior; The spatiotemporal feature map is processed by global average pooling to obtain a one-dimensional visual feature vector. At the same time, the landing point rationality judgment result is converted into numerical features. The two are concatenated and input into a fully connected layer. The fully connected layer calculates the confidence score of the jump behavior based on the visual feature vector. Combined with whether the landing point is in the danger zone and the risk level of the danger zone, the corresponding danger level is generated.

[0018] A third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the platform jump abnormal behavior recognition method described above.

[0019] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the platform jump abnormal behavior recognition method described above.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention integrates visual features and platform geometric constraints, and accurately distinguishes dangerous jumps from normal actions by judging the rationality of the landing point, avoiding missed high-risk behaviors and improving recognition accuracy; 2. This invention reduces redundant data input by screening vertical displacement candidate segments and using adaptive frame sampling, combined with an asymmetric spatiotemporal downsampling strategy, thereby reducing computational overhead while preserving key details and solving the pain points of complex and costly computation of traditional optical flow methods and global Transformer models. 3. This invention forms a closed-loop response mechanism of automatic identification, hierarchical alarm, rapid handling and data review, which transforms the response from passive to proactive warning, effectively reducing the risk of safety accidents such as falling from heights and falling into dangerous areas caused by platform jumping, and helping to standardize safety production management. Attached Figure Description

[0021] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a first flowchart of a platform jump abnormal behavior recognition method provided in an embodiment of the present invention.

[0022] Figure 2 This is a second flowchart of the platform jump abnormal behavior recognition method provided in the embodiments of the present invention.

[0023] Figure 3 This is a schematic diagram of a platform jump abnormal behavior recognition device provided in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] This invention provides a method, apparatus, device, and medium for identifying platform jump anomaly behavior, comprising: acquiring continuous video frames, filtering candidate segments containing vertical displacement actions based on the acquired continuous video frames, and generating an input tensor by adaptive frame sampling of the candidate segments; extracting features from the input tensor using an optimized video Swing transformer to obtain a multi-scale spatiotemporal feature map; fusing the obtained multi-scale spatiotemporal feature map to obtain a comprehensive spatiotemporal feature map; predicting the landing coordinates of the jump behavior using a landing point rationality network based on the comprehensive spatiotemporal feature map; matching the landing coordinates with a geometric boundary mask map to determine whether the current landing point is located within a preset danger zone, thereby identifying the jump anomaly behavior. This invention solves the technical problem of insufficient feature extraction in key spatiotemporal regions of jump behavior, making it difficult to meet the industrial-grade requirements for high-precision, low-latency anomaly identification.

[0026] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the platform jump abnormal behavior identification method in this invention includes: 101. Obtain continuous video frames, and filter candidate segments containing vertical displacement actions based on the obtained continuous video frames. Perform adaptive frame sampling on the candidate segments to generate input tensors. In this embodiment, the surveillance camera device is activated, a video data transmission channel is established, and continuous video stream data is continuously received according to the camera's default frame rate; Each frame of video image in a continuous video stream is converted into a standard digital image format that meets preset requirements. Each converted frame contains the complete platform scene area. The inter-frame difference variance method is used to process continuous video frames. Specifically, firstly, two adjacent video frames are selected and converted into grayscale images to simplify the calculation. Then, the difference between the grayscale values ​​of corresponding pixels in the two frames is calculated. After squaring all the differences, the inter-frame difference variance value is obtained by summing them. An empirical variance threshold is set. When the variance difference between a frame and the previous frame exceeds this threshold, significant motion is considered to exist between the two frames. Frame sequences containing significant motion are continuously tracked. When motion matching vertical displacement characteristics appears in the sequence, that frame sequence is marked as a candidate segment. The motion density of the candidate segments is analyzed. When the motion within a segment is intense, a smaller sampling interval is used; when the motion is relatively gentle, a larger sampling interval is used, ensuring that key motion information is preserved while controlling the amount of data. T frames are extracted from the candidate segments using this adaptive sampling method. The size of each frame is uniformly adjusted to H×W, and its RGB three-channel information is preserved. Finally, these are combined to generate an input tensor of T frames × H × W × 3 dimensions.

[0028] 102. Use an optimized video Swin transformer to extract features from the input tensor to obtain a multi-scale spatiotemporal feature map; In this embodiment, a 3D sliding window is constructed in the spatiotemporal dimension of the input tensor according to a preset window size, such as T1×H1×W1, where T1 is the time dimension window length and H1 and W1 are the spatial dimension window lengths. During the sliding process, the window moves in the time and spatial dimensions according to a preset step size, dividing the entire input tensor into multiple non-overlapping or partially overlapping 3D window feature blocks. A spatiotemporal jump trajectory template is generated based on a physical parabolic model. Specifically, the physical parabolic model combines parameters such as gravitational acceleration and common initial jump velocities to calculate the complete spatial coordinate change curve from the take-off point to the highest point of flight and then to the landing point, and maps it to the pixel coordinate system of the image to form a spatiotemporal jump trajectory template.

[0029] Based on the jump trajectory template, for each 3D window feature block, the similarity between each spatiotemporal position within the block and the corresponding position on the trajectory template is calculated. Spatiotemporal attention bias values ​​are generated based on the similarity, with higher similarity positions corresponding to larger bias values. For all feature vectors within each 3D window feature block, the original self-attention weights are calculated, and the attention bias value is superimposed on the original self-attention weights to obtain the enhanced self-attention weights. Specifically, in each Swing Block, the generated spatiotemporal attention bias is embedded into the self-attention calculation process. When calculating the self-attention weights for all feature vectors within each 3D window feature block, the attention bias is superimposed on the original weights, thereby dynamically enhancing the model's attention to the features of the take-off-flight-landing region covered by the trajectory template.

[0030] After completing the self-attention calculation, the features are transformed through layer normalization and fully connected layers to obtain the enhanced local spatiotemporal features; The enhanced local spatiotemporal features are sequentially sampled using an asymmetric spatiotemporal downsampling strategy and a symmetric downsampling strategy to obtain multi-scale spatiotemporal feature maps. These multi-scale spatiotemporal feature maps include high-level and low-level feature maps. Specifically, in the first two stages, an asymmetric downsampling strategy is used to process the enhanced spatiotemporal features. The downsampling stride for the time dimension is set to 1, preserving temporal details by not changing the feature length in the time dimension, while the downsampling stride for the spatial dimension is set to 2, reducing the spatial size to half its original size through convolution operations. In subsequent stages, conventional symmetric downsampling is used for feature dimensionality reduction, progressively extracting more abstract high-level spatiotemporal features.

[0031] 103. The obtained multi-scale spatiotemporal feature maps are fused to obtain a comprehensive spatiotemporal feature map; the comprehensive spatiotemporal feature map is used to predict the landing coordinates of the jump behavior through the landing point rationality network; the landing coordinates are matched with the geometric boundary mask map to determine whether the current landing point is located in the preset danger zone, thereby identifying abnormal jump behavior.

[0032] In this embodiment, multi-scale spatiotemporal feature maps of different stages are used, corresponding to spatiotemporal information from fine-grained to coarse-grained, respectively. A top-down feature fusion strategy is adopted, upsampling the high-level feature maps to the same size as the low-level feature maps, then adding or concatenating the feature maps of corresponding levels element by element, and integrating the features through convolutional layers to obtain a comprehensive spatiotemporal feature map that integrates multi-scale information. The platform's CAD model or pre-calibrated geometric boundary data is acquired, and a geometric boundary mask is constructed based on the acquired platform CAD model or pre-calibrated geometric boundary data. The geometric boundary data includes: guardrail coordinates, ground area range, and precise coordinate information of dangerous areas such as tracks and pits. More specifically, the geometric coordinate data is converted into a pixel coordinate system corresponding to the feature map to generate a geometric boundary mask, clearly marking the pixel range of safe and dangerous areas. A landing point rationality subnetwork is introduced. The fused spatiotemporal feature map is used as input. The feature information related to the landing point is extracted through convolution and recurrent neural networks to predict the possible landing point coordinates of the jump behavior. The predicted landing point coordinates are matched with the geometric boundary mask to determine whether the landing point is located within a preset danger zone. If it is, the landing point is marked as unreasonable; otherwise, it is considered reasonable. A one-dimensional visual feature vector is obtained from the comprehensive spatiotemporal feature map through global average pooling. Simultaneously, the landing point reasonableness judgment result is converted into numerical features. The two are concatenated and input into a fully connected layer. The fully connected layer calculates the confidence score of the jump behavior based on the feature vector. Combining this with whether the landing point is in a danger zone and the risk level of the danger zone, a corresponding danger level (e.g., low, medium, high) is generated.

[0033] This embodiment significantly improves the recognition accuracy, reduces operation and maintenance costs, reduces reliance on manual inspections, achieves 24 / 7 automatic early warning, and improves the level of safety production management.

[0034] Please see Figure 2 The second embodiment of the platform jump abnormal behavior identification method in this invention includes: 201. Obtain continuous video frames, and filter candidate segments containing vertical displacement actions based on the obtained continuous video frames. Perform adaptive frame sampling on the candidate segments to generate input tensors. 202. Use an optimized video Swing transformer to extract features from the input tensor to obtain a multi-scale spatiotemporal feature map; 203. The obtained multi-scale spatiotemporal feature maps are fused to obtain a comprehensive spatiotemporal feature map; the comprehensive spatiotemporal feature map is used to predict the landing coordinates of the jump behavior through the landing point rationality network; the landing coordinates are matched with the geometric boundary mask map to determine whether the current landing point is located in the preset danger zone, thereby identifying abnormal jump behavior; 204. When the confidence level exceeds the threshold and the danger level is medium or high, a real-time alarm signal is triggered, and the time, location and video clip of the behavior are output. In this embodiment, a preset confidence threshold and danger level triggering conditions for jumping behavior are established, and the confidence and danger levels of jumping behavior are received in real time. The confidence level is compared with the preset threshold, and it is determined whether the danger level reaches the medium or high standard. Only when both conditions are met simultaneously is the subsequent alarm process triggered. When the alarm triggering conditions are met, a standardized alarm signal is immediately generated, which includes information such as alarm type and urgency. The alarm signal is transmitted to the alarm terminal in the monitoring center through a preset communication protocol, triggering on-site alarm devices such as sound and light, and simultaneously sending alarm notifications to the mobile terminals of relevant management personnel. The precise time and location of the abnormal behavior are collected simultaneously. A complete video segment containing the abnormal behavior is extracted from the original video stream, with the segment starting 5 seconds before the behavior occurs and ending 10 seconds after the behavior occurs, ensuring complete recording of the behavior process. The occurrence time, location, danger level, confidence score, and extracted video segment of the abnormal behavior are integrated to generate a standardized abnormal behavior report.

[0035] The hazard level output in this embodiment can be directly connected to audible and visual alarms, broadcasting systems, or robotic inspection platforms to form a closed-loop response mechanism.

[0036] The platform jump abnormal behavior recognition method in the embodiments of the present invention has been described above. The platform jump abnormal behavior recognition device in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 3 One embodiment of the platform jump abnormal behavior recognition device in this invention includes: The input tensor generation module 301 is used to acquire continuous video frames, filter candidate segments containing vertical displacement actions based on the acquired continuous video frames, and perform adaptive frame sampling on the candidate segments to generate input tensors. In this embodiment, the input tensor generation module 301 includes: The continuous video stream acquisition submodule 3011 is used to acquire continuous video streams through a surveillance camera. The format conversion submodule 3012 is used to convert each frame of video image in the acquired continuous video stream into a standard digital image format that meets preset requirements. Each frame of image after conversion contains the complete platform scene area. The image extraction submodule 3013 is used to obtain the inter-frame difference variance value for the converted video frames using the inter-frame difference variance method; and compare the inter-frame difference variance value with a set threshold. When the inter-frame difference variance value exceeds the set threshold, it is determined that there is a vertical displacement action between the two consecutive frames; repeatedly trigger the execution to obtain the frame sequence containing the vertical displacement action, and mark the frame sequence containing the vertical displacement action as a candidate segment; and extract T frames of images from the candidate segments through an adaptive sampling method. The step of obtaining the inter-frame difference variance value using the inter-frame difference variance method for the converted video frames includes: selecting two adjacent video images for the converted video frames and converting them into grayscale images; calculating the difference in grayscale values ​​of corresponding pixels in the two grayscale images; and summing the squared differences to obtain the inter-frame difference variance value.

[0037] The size adjustment submodule 3014 is used to adjust the size of each frame in the extracted T-frame images to H×W, while retaining its RGB three-channel information, and finally combine them to generate an input tensor of T×H×W×3 dimensions.

[0038] The multi-scale spatiotemporal feature map acquisition module 302 is used to extract features from the input tensor using an optimized video Swin transformer to obtain a multi-scale spatiotemporal feature map. In this embodiment, the multi-scale spatiotemporal feature map acquisition module 302 includes: The 3D window feature block generation submodule 3021 is used to construct a 3D sliding window in the spatiotemporal dimension of the input tensor according to a preset window size. During the sliding process, the window moves in the time and space dimensions according to a preset step size, dividing the entire input tensor into multiple non-overlapping or partially overlapping 3D window feature blocks. The spatiotemporal attention bias value generation submodule 3022 is used to generate a spatiotemporal jump trajectory template based on a physical parabolic model. Based on the jump trajectory template, for each 3D window feature block, the similarity between each spatiotemporal position within the block and the corresponding position on the trajectory template is calculated, and a spatiotemporal attention bias value is generated based on the similarity. The jump trajectory template is a complete spatial coordinate change curve calculated from the take-off point to the highest point of the flight and then to the landing point by combining the physical parabolic model with the calculation of the complete spatial coordinate change curve from the take-off point to the highest point of the flight and then to the landing point, including the gravitational acceleration and the preset initial jump velocity, and mapped to the pixel coordinate system of the image to form a spatiotemporal jump trajectory template.

[0039] The enhanced self-attention weight submodule 3023 is used to calculate the original self-attention weights for all feature vectors within each 3D window feature block, and then superimpose the attention bias value onto the original self-attention weights to obtain the enhanced self-attention weights. The local spatiotemporal feature generation submodule 3024 is used to obtain the local spatiotemporal features of the focus jump-takeoff-landing point by weighted summing of the feature vectors in the 3D window feature block based on the enhanced self-attention weights. The obtained local spatiotemporal features focusing on takeoff-flight-landing point are processed through layer normalization and fully connected layers to obtain enhanced local spatiotemporal features; The multi-scale spatiotemporal feature map generation submodule 3025 is used to sample the enhanced local spatiotemporal features sequentially using an asymmetric spatiotemporal downsampling strategy and a symmetric downsampling strategy to obtain a multi-scale spatiotemporal feature map; wherein, the multi-scale spatiotemporal feature map includes a high-level feature map and a low-level feature map.

[0040] The jump abnormal behavior recognition module 303 is used to fuse the obtained multi-scale spatiotemporal feature maps to obtain a comprehensive spatiotemporal feature map; the comprehensive spatiotemporal feature map predicts the landing coordinates of the jump behavior through the landing point rationality network; the landing coordinates are matched with the geometric boundary mask map to determine whether the current landing point is located in the preset danger area, thereby recognizing the jump abnormal behavior. In this embodiment, the jumping abnormal behavior recognition module 303 includes: The comprehensive spatiotemporal feature map generation submodule 3031 is used to upsample the high-level feature maps in the multi-scale spatiotemporal feature map to the same size as the low-level feature maps; and to add or concatenate the feature maps of the corresponding levels element by element, and integrate them through a convolutional layer to obtain the fused multi-scale information comprehensive spatiotemporal feature map.

[0041] The geometric boundary mask construction submodule 3032 is used to obtain the platform's CAD model or pre-calibrated geometric boundary data, and construct a geometric boundary mask based on the obtained platform CAD model or pre-calibrated geometric boundary data; The landing point coordinate prediction submodule 3033 is used to construct a landing point rationality network based on convolutional and recurrent neural networks; the comprehensive spatiotemporal feature map is input into the landing point rationality network to predict the landing point coordinates of the jump behavior; The anomaly recognition submodule 3034 is used to obtain a one-dimensional visual feature vector from the comprehensive spatiotemporal feature map through global average pooling, and at the same time convert the landing point rationality judgment result into numerical features. The two are concatenated and input into the fully connected layer. The fully connected layer calculates the confidence score of the jump behavior based on the visual feature vector, and generates the corresponding danger level by combining whether the landing point is in the danger zone and the risk level of the danger zone.

[0042] The early warning module 304 is used to trigger a real-time alarm signal and output the time, location and video clip of the behavior when the confidence level exceeds the threshold and the danger level is medium or high. In this embodiment, the early warning module 304 includes: The system presets a confidence threshold and hazard level trigger conditions for jump behavior, receiving the confidence level and hazard level of the jump behavior in real time. It compares the confidence level with the preset threshold and determines whether the hazard level reaches the medium or high standard. Only when both conditions are met simultaneously is the subsequent alarm process triggered. When the alarm trigger conditions are met, a standardized alarm signal is immediately generated, containing information such as alarm type and urgency. The alarm signal is transmitted to the alarm terminal in the monitoring center via a preset communication protocol, triggering on-site alarm devices such as sound and lights, and simultaneously sending alarm notifications to the mobile terminals of relevant management personnel. The system also synchronously collects the precise time and location of the abnormal behavior. A complete video segment containing the abnormal behavior is extracted from the original video stream, with the segment starting 5 seconds before the behavior occurs and ending 10 seconds after, ensuring a complete record of the behavior. The time, location, hazard level, confidence score, and extracted video segment of the abnormal behavior are integrated to generate a standardized abnormal behavior report.

[0043] above Figure 3 The platform jump abnormal behavior identification device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The electronic device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0044] Figure 4This is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present invention. The electronic device 700 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 710 (e.g., one or more processors) and a memory 720, and one or more storage media 730 (e.g., one or more mass storage devices) for storing application programs 733 or data 732. The memory 720 and storage media 730 can be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the electronic device 700. Furthermore, the processor 710 may be configured to communicate with the storage media 730 and execute the series of instruction operations in the storage media 730 on the electronic device 700.

[0045] Electronic device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input / output interfaces 750, and / or one or more operating systems 731, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The illustrated electronic device structure does not constitute a limitation on electronic devices and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0046] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the platform jump abnormal behavior identification method.

[0047] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0048] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0049] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying abnormal platform jumping behavior, characterized in that, include: Acquire continuous video frames, filter candidate segments containing vertical displacement actions based on the acquired continuous video frames, and generate input tensors by adaptive frame sampling of the candidate segments; A multi-scale spatiotemporal feature map is obtained by extracting features from the input tensor using an optimized video Swing transformer. The obtained multi-scale spatiotemporal feature maps are fused to obtain a comprehensive spatiotemporal feature map. The comprehensive spatiotemporal feature map is used to predict the landing coordinates of the jump behavior through the landing point rationality network. The landing coordinates are matched with the geometric boundary mask map to determine whether the current landing point is located in the preset danger zone, thereby identifying abnormal jump behavior.

2. The platform jump abnormal behavior identification method according to claim 1, characterized in that, The process of acquiring continuous video frames, filtering candidate segments containing vertical displacement actions based on the acquired continuous video frames, and generating input tensors by adaptive frame sampling of the candidate segments includes: Acquire continuous video streams through surveillance cameras; Each frame of video image in the acquired continuous video stream is converted into a standard digital image format that meets preset requirements. Each converted frame contains the complete platform scene area. The inter-frame difference variance method is used to obtain the inter-frame difference variance value for the converted video frames; the inter-frame difference variance value is compared with a set threshold. When the inter-frame difference variance value exceeds the set threshold, it is determined that there is a vertical displacement action between the two consecutive frames; the process is repeatedly triggered to obtain the frame sequence containing the vertical displacement action, and the frame sequence containing the vertical displacement action is marked as a candidate segment; T-frame images are extracted from the candidate segments through adaptive sampling. The size of each frame in the extracted T-frame images is adjusted to H×W, and its RGB three-channel information is preserved. Finally, they are combined to generate an input tensor of T×H×W×3 dimensions.

3. The platform jump abnormal behavior identification method according to claim 2, characterized in that, The method of obtaining the inter-frame difference variance value for the converted video frames using the inter-frame difference variance method includes: selecting two adjacent video images for the converted video frames and converting them into grayscale images; calculating the difference in grayscale values ​​of corresponding pixels in the two grayscale images; and summing the squared differences to obtain the inter-frame difference variance value.

4. The platform jump abnormal behavior identification method according to claim 1, characterized in that, The step of using an optimized video Swin transformer to extract features from the input tensor to obtain a multi-scale spatiotemporal feature map includes: According to the preset window size, a 3D sliding window is constructed in the spatiotemporal dimension of the input tensor; during the sliding process, the window moves in the time and space dimensions according to the preset step size, dividing the entire input tensor into multiple non-overlapping or partially overlapping 3D window feature blocks. A spatiotemporal jump trajectory template is generated based on a physical parabolic model. Based on the jump trajectory template, for each 3D window feature block, the similarity between each spatiotemporal position within the block and the corresponding position on the trajectory template is calculated, and a spatiotemporal attention bias value is generated based on the similarity. Calculate the original self-attention weights for all feature vectors within each 3D window feature block, and then superimpose the attention bias values ​​onto the original self-attention weights to obtain the enhanced self-attention weights. Based on the enhanced self-attention weights, the feature vectors within the 3D window feature block are weighted and summed to obtain the local spatiotemporal features of the focus jump-takeoff-landing point. The obtained local spatiotemporal features focusing on takeoff-flight-landing point are processed through layer normalization and fully connected layers to obtain enhanced local spatiotemporal features; The enhanced local spatiotemporal features are sampled sequentially using an asymmetric spatiotemporal downsampling strategy and a symmetric downsampling strategy to obtain a multi-scale spatiotemporal feature map; the multi-scale spatiotemporal feature map includes a high-level feature map and a low-level feature map.

5. The platform jump abnormal behavior identification method according to claim 4, characterized in that, The method of generating a spatiotemporal jump trajectory template based on a physical parabolic model includes: using the physical parabolic model to calculate the complete spatial coordinate change curve from the take-off point to the highest point of the flight and then to the landing point, including gravitational acceleration and a preset initial jump velocity, and mapping it to the pixel coordinate system of the image to form a spatiotemporal jump trajectory template.

6. The platform jump abnormal behavior identification method according to claim 4, characterized in that, The step of fusing the obtained multi-scale spatiotemporal feature maps to obtain a comprehensive spatiotemporal feature map includes: The high-level feature maps in the multi-scale spatiotemporal feature map are upsampled to the same size as the low-level feature maps; and the feature maps of the corresponding levels are added or concatenated element by element, and integrated through a convolutional layer to obtain a fused spatiotemporal feature map of multi-scale information.

7. The platform jump abnormal behavior identification method according to claim 1, characterized in that, The integrated spatiotemporal feature map predicts the landing coordinates of the jump behavior through a landing point rationality network; the landing coordinates are matched with a geometric boundary mask map to determine whether the current landing point is located within a preset danger zone, thereby identifying abnormal jump behavior, including: Obtain the platform's CAD model or pre-calibrated geometric boundary data, and construct a geometric boundary mask based on the obtained platform CAD model or pre-calibrated geometric boundary data; Constructing a landing point rationality network based on convolutional and recurrent neural networks; The comprehensive spatiotemporal feature map is input into the landing point rationality network to predict the landing point coordinates of the jump behavior; The spatiotemporal feature map is processed by global average pooling to obtain a one-dimensional visual feature vector. At the same time, the landing point rationality judgment result is converted into numerical features. The two are concatenated and input into a fully connected layer. The fully connected layer calculates the confidence score of the jump behavior based on the visual feature vector. Combined with whether the landing point is in the danger zone and the risk level of the danger zone, the corresponding danger level is generated.

8. A platform jumping abnormal behavior recognition device, characterized in that, include: The input tensor generation module is used to acquire continuous video frames, filter candidate segments containing vertical displacement actions based on the acquired continuous video frames, and perform adaptive frame sampling on the candidate segments to generate input tensors. The multi-scale spatiotemporal feature map acquisition module is used to extract features from the input tensor using an optimized video Swin transformer to obtain a multi-scale spatiotemporal feature map. The jump abnormal behavior recognition module is used to fuse the obtained multi-scale spatiotemporal feature maps to obtain a comprehensive spatiotemporal feature map; the comprehensive spatiotemporal feature map predicts the landing coordinates of the jump behavior through the landing point rationality network; the landing coordinates are matched with the geometric boundary mask map to determine whether the current landing point is located within the preset danger area, thereby identifying the jump abnormal behavior.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the platform jump abnormal behavior recognition method according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the platform jump abnormal behavior recognition method according to any one of claims 1 to 7.

Images