Diver behavior anomaly detection method and system based on space-time attention mechanism

Abstract

This invention provides a method and system for detecting abnormal diver behavior based on a spatiotemporal attention mechanism. The method includes: constructing an underwater local optical distortion vector field by tracking the displacement of suspended particles in a video. This vector field is then used in a feature extraction network to perform regularization correction on the image and guide the receptive field shift to extract limb spatial features. When a limb is obscured by a bubble, cross-frame cross-correlation calculation is performed by analyzing the deformation of the bubble boundary and the limb motion velocity vector field of the last frame before obscuration to infer the implicit motion trajectory of the obscured limb, and a compensating feature vector is constructed to generate a spatiotemporal feature matrix. Finally, the divergence value of the gradient field of this matrix is ​​calculated to quantify the degree of behavioral abruptness; when the divergence value breaks through the stable range, it is determined to be abnormal. This invention solves the technical problems of inaccurate behavioral feature extraction and interruption of temporal information caused by underwater environmental interference and bubble obscuration.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, specifically relating to a method and system for detecting abnormal diver behavior based on a spatiotemporal attention mechanism. Background Technology

[0002] Underwater diving operations, whether in marine engineering, underwater rescue, or scientific research, are considered high-risk specialized operations. Due to the complexity and unpredictability of the underwater environment, divers may face various emergencies such as equipment malfunction, physical limits, and entanglement with foreign objects. These situations often escalate into life-threatening accidents within a very short time. Therefore, real-time anomaly detection of divers' behavior through underwater video monitoring systems, enabling immediate detection of dangers and intervention, is crucial for ensuring the safety of divers. Existing computer vision-based behavior recognition technologies primarily use deep learning models to extract and classify spatiotemporal features of human posture sequences in video streams to determine their behavioral categories.

[0003] However, directly applying these technologies, successful in conventional terrestrial environments, to underwater diver monitoring presents two technical challenges arising from the unique underwater environment. First, underwater light propagation is affected by factors such as water flow and thermoclines, resulting in irregular refraction and distortion. This causes the diver's limb contours in video images to become distorted and drifted, severely interfering with the accuracy of key limb feature extraction. Second, the numerous dense and opaque bubbles produced by the diver's breathing frequently obscure key parts of their body, causing non-periodic gaps in video information in both time and space. For behavior recognition models that rely on the continuity of spatiotemporal features, this data interruption caused by bubble obstruction prevents the construction of a complete and coherent sequence of actions, leading to numerous errors or missed detections in the judgment of diver behavior. Summary of the Invention

[0004] This invention provides a method and system for detecting abnormal diver behavior based on a spatiotemporal attention mechanism to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for detecting abnormal diver behavior based on a spatiotemporal attention mechanism, the method comprising the following steps: A continuous video stream of the underwater working environment of divers is collected, and the dynamic displacement of suspended particles in the continuous video stream is tracked by high-pass filtering and optical flow method. A two-dimensional vector field of underwater local optical distortion is constructed based on the dynamic displacement. In the pre-defined feature extraction network with spatial attention mechanism, the inverse of the magnitude of the deformation vector in the two-dimensional vector field is used as a regularization term. At the same time, the direction of the deformation vector in the two-dimensional vector field is used to guide the receptive field of the feature extraction network to shift, thereby extracting the spatial features of the diver's limbs from the continuous video stream. When the spatial features of the limbs are detected to be obscured by bubbles, the normal deformation and tangential movement of the bubble boundary caused by the local water flow displacement are extracted. Extract the limb motion velocity vector field of the last frame before being occluded by the bubble, perform cross-frame cross-correlation calculation between the limb motion velocity vector field and the normal deformation and tangential movement, invert to generate implicit motion trajectory equation, and use the implicit motion trajectory equation to construct a compensation feature vector and input it into the preset time attention mechanism network to generate an aligned spatiotemporal feature matrix. The spatiotemporal feature matrix is ​​mapped to a spatiotemporal action topology sequence with a time series distribution, and the local behavioral mutation divergence of the spatiotemporal action topology sequence is obtained by calculating the divergence value of the gradient field of the spatiotemporal feature matrix between adjacent time frames. When the local behavioral abrupt change divergence is detected to exceed the preset stable range within a preset time window, it is determined that the target diver has engaged in abnormal behavior and an alarm signal is output.

[0006] Optionally, the step of acquiring a continuous video stream of the diver's underwater working environment, tracking the dynamic displacement of suspended particles in the continuous video stream using high-pass filtering and optical flow, and constructing a two-dimensional vector field of underwater local optical distortion based on the dynamic displacement includes the following steps: Acquire continuous video streams of the underwater working environment of divers and convert the continuous video streams into a sequence of grayscale images; High-pass filtering algorithm is used to filter out the background of grayscale image sequence and separate the underwater suspended particle set; The optical flow method is used to track the two-dimensional displacement coordinates of each suspended particle in an array of suspended particles in adjacent grayscale image sequences; The pixel space corresponding to the grayscale image sequence is divided into multiple local grid regions; The average displacement coordinates of suspended particles in each local grid region are calculated based on two-dimensional displacement coordinates, and a local grid deformation vector is generated. The local mesh deformation vectors of all local mesh regions are spliced ​​and combined to form a two-dimensional vector field of underwater local optical distortion.

[0007] Optionally, the step of using optical flow to track the two-dimensional displacement coordinates of each suspended particle in an adjacent grayscale image sequence includes the following steps: Assign a unique tracking identifier to each suspended particle in the set of suspended particles in the grayscale image sequence of the current frame; An optical flow evaluation model based on an image pyramid is constructed by inputting the current frame grayscale image sequence containing tracking identifiers and the adjacent next frame grayscale image sequence into the optical flow evaluation model. The initial coarse displacement of each suspended particle is calculated in the top-level image of the optical flow evaluation model; The initial coarse displacement is projected as a priori parameter into the underlying image of the optical flow evaluation model; By combining nonlocal spatial smoothing constraints, the initial coarse displacement in the underlying image is iteratively optimized to eliminate the interference of underwater high-frequency noise on the tracking trajectory and obtain the target displacement vector of each suspended particle. The two-dimensional displacement coordinates of each suspended particle in adjacent grayscale image sequences are determined based on the target displacement vector.

[0008] Optionally, the iterative optimization of the initial coarse displacement in the underlying high-resolution image by combining nonlocal spatial smoothing constraints to eliminate the interference of underwater high-frequency noise on the tracking trajectory, and obtaining the target displacement vector of each suspended particle, includes the following steps: A non-local neighborhood window is constructed with each suspended particle in the underlying image as the center. Calculate the gray-level similarity weight and spatial distance weight of adjacent pixels within a non-local neighborhood window; The gray-level similarity weight and spatial distance weight are combined to generate a nonlocal spatial smoothing constraint term. A global energy functional optimization objective is constructed by combining nonlocal spatial smoothing constraints with optical flow data terms. The global energy functional optimization objective is solved iteratively using the conjugate gradient descent method. When the convergence condition is met, the target displacement vector of each suspended particle is output.

[0009] Optionally, in the preset feature extraction network with spatial attention mechanism, the reciprocal of the magnitude of the deformation vector in the two-dimensional vector field is used as a regularization term, and the direction of the deformation vector in the two-dimensional vector field is used to guide the receptive field of the feature extraction network to shift, thereby extracting the spatial features of the diver's limbs from the continuous video stream, includes the following steps: Continuous video frames are input into a pre-defined feature extraction network with a spatial attention mechanism to generate an initial feature map; Extract the magnitude and direction of the deformation vector for each local grid region from the two-dimensional vector field; Calculate the reciprocal of the magnitude of the deformation vector and map the reciprocal of the magnitude of the deformation vector to a regularization penalty mask; The regularization penalty mask is multiplied element-wise with the initial attention weight map of the feature extraction network to suppress the attention weights in high distortion regions. The deformation vector direction is converted into a two-dimensional offset field, and the two-dimensional offset field is injected into the deformable convolutional layer of the feature extraction network. Deformable convolutional layers are used, and the receptive field of the convolutional kernel is guided to shift in the opposite direction of the water flow distortion according to a two-dimensional offset field. Based on the offset receptive field and the suppressed attention weight, the initial feature map is resampled and fused to extract the spatial features of the diver's limbs after eliminating optical artifacts.

[0010] Optionally, when the spatial features of a limb are detected to be obscured by a bubble, extracting the normal deformation and tangential movement of the bubble boundary caused by local water flow displacement includes the following steps: Real-time monitoring of the brightness of pixel regions containing spatial features of limbs, and calculation of local brightness gradients between adjacent video frames in a continuous video stream; When the local brightness gradient exceeds the preset bubble reflection threshold and the limb spatial features are lost, it is determined that the limb spatial features are occluded by bubbles. An edge detection algorithm is used to scan the contour of the feature-lost region and lock the initial set of pixel coordinates of the bubble boundary; Tracing the topological deformation trajectory of the initial set of pixel coordinates in a sequence of consecutively occluded video frames in a continuous video stream; The normal deformation magnitude of the bubble boundary in the normal direction and the displacement in the tangential direction are quantified based on the topological deformation trajectory, so as to obtain the normal deformation and tangential displacement of the bubble boundary caused by local water flow.

[0011] Optionally, tracing the topological deformation trajectory of the initial set of pixel coordinates in a sequence of continuously occluded video frames in a continuous video stream includes the following steps: Assign an independent Markov tracking node to each boundary pixel in the initial set of pixel coordinates; The state transition probability matrix of the Markov tracking node is defined based on the time interval between adjacent occluded video frames in a continuous video stream. Extract the texture gradient features around the boundary pixels as observation variables and input them into the preset hidden Markov model; By combining the state transition probability matrix with the observed variables, the Viterbi algorithm is used to decode the optimal hidden state sequence of each Markov tracking node; Identify discontinuous jump nodes in the optimal hidden state sequence caused by bubble fusion or rupture; The cubic spline interpolation algorithm is used to smooth out discontinuous jump nodes and generate continuous node displacement curves. The node displacement curves of all boundary pixels are collected to form the topological deformation trajectory of the initial pixel coordinate set.

[0012] Optionally, the step of extracting the limb motion velocity vector field of the last frame before being occluded by the bubble, performing cross-frame cross-correlation calculations on the limb motion velocity vector field with the normal deformation and tangential displacement, generating an implicit motion trajectory equation, and constructing a compensation feature vector using the implicit motion trajectory equation and inputting it into a preset temporal attention mechanism network to generate an aligned spatiotemporal feature matrix includes the following steps: The spatial features of the limbs within the time window before being occluded by the bubble are cached, and the limb motion velocity vector field of the last frame before being occluded by the bubble is calculated and extracted. The limb motion velocity vector field, normal deformation, and tangential displacement are mapped to a unified spatiotemporal reference frame. Construct a cross-frame cross-correlation function in a spatiotemporal reference frame and determine the peak point of the response of the cross-frame cross-correlation function; Based on the response peak point, the three-dimensional motion coordinates of the limbs hidden behind the bubble are deduced in reverse, and the implicit motion trajectory equation is generated by fitting the three-dimensional motion coordinates of the limbs. Discretize and sample the implicit motion trajectory equation to construct a compensation feature vector; After replacing the missing features in the limb spatial features with the compensation feature vector, the input is fed into the preset temporal attention mechanism network to generate the aligned spatiotemporal feature matrix.

[0013] In a second aspect, the present invention also provides a diver behavior anomaly detection system based on a spatiotemporal attention mechanism, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the diver behavior anomaly detection method based on a spatiotemporal attention mechanism as described in any one of the first aspects.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing instructions, characterized in that, when executed by a processor, the instructions cause the processor to be configured to perform the diver behavior anomaly detection method based on a spatiotemporal attention mechanism according to any one of the first aspects.

[0015] The beneficial effects of this invention are: This invention constructs and utilizes a two-dimensional vector field of underwater local optical distortion. During the feature extraction stage, it not only regularizes and corrects image deformation caused by water flow disturbance but also uses the direction of the deformation vector to guide the receptive field of the feature extraction network to dynamically shift, solving the problem of inaccurate positioning of diver limb key points caused by non-uniform water flow. Furthermore, by analyzing the normal and tangential deformation of the bubble boundary and performing cross-frame cross-correlation calculations with the limb motion velocity vector field of the last frame before occlusion, an implicit motion trajectory equation describing the movement trend of the occluded limb is derived. This transforms the occlusion, which would otherwise cause information loss, into an indirect source of information for inferring the motion state behind it, ensuring the continuity and integrity of temporal features. Finally, by calculating the divergence value of the gradient field of the spatiotemporal feature matrix, the degree of abrupt change in behavior over time is directly quantified. If abrupt change occurs, an abnormal behavior is determined. This invention solves the technical challenges of inaccurate behavioral feature extraction and interruption of temporal information caused by underwater environmental interference and bubble occlusion. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for detecting abnormal diver behavior based on a spatiotemporal attention mechanism in one embodiment of this application.

[0017] Figure 2 This is a schematic diagram of the network topology of the feature extraction network in one embodiment of this application.

[0018] Figure 3 This is a schematic diagram of the network topology of the time attention mechanism network in one embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] Figure 1 This is a flowchart illustrating a diver behavior anomaly detection method based on a spatiotemporal attention mechanism in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps. For example Figure 1 As shown, the method for detecting abnormal diver behavior based on a spatiotemporal attention mechanism disclosed in this invention specifically includes the following steps: S101. Acquire a continuous video stream of the underwater working environment of divers, and track the dynamic displacement of suspended particles in the continuous video stream through high-pass filtering and optical flow method, and construct a two-dimensional vector field of underwater local optical distortion based on the dynamic displacement.

[0022] Optical distortion in underwater working environments primarily stems from changes in refractive index caused by uneven water density, resulting in localized distortion of video images. Tracking the motion of suspended particles can indirectly reflect the optical disturbance characteristics of the water. First, a continuous video stream of the diver's working area is acquired. The video acquisition equipment must be equipped with a waterproof housing and have a sampling rate of at least 30 frames per second to ensure motion continuity. The acquired color video frames are converted into grayscale image sequences. A high-pass filter is applied to the grayscale image sequence, with the cutoff frequency set between 0.1 and 0.3 normalized frequencies to effectively filter out low-frequency background components while retaining high-frequency edge information of the suspended particles. The Lucas-Kanade optical flow algorithm is used to track the displacement of suspended particles between adjacent frames. This algorithm assumes consistent pixel motion and constant brightness within local regions. The image space is divided into a 16×16 pixel local grid. Within each grid, a weighted average of the displacements of all suspended particles is calculated, with the weight proportional to the gradient intensity of the particles. Each grid generates a deformation vector. The deformation vectors of all grids are combined to form a two-dimensional vector field, which comprehensively describes the spatial distribution characteristics of local underwater optical distortion.

[0023] S102. In the preset feature extraction network with spatial attention mechanism, the inverse of the magnitude of the deformation vector in the two-dimensional vector field is used as a regularization term. At the same time, the direction of the deformation vector in the two-dimensional vector field is used to guide the receptive field of the feature extraction network to shift, thereby extracting the spatial features of the diver's limbs from the continuous video stream.

[0024] The spatial attention mechanism highlights important regions by dynamically adjusting the weights at different locations in the feature map. In underwater scenes, it requires adaptive adjustment to address optical distortion. Continuous video frames are input into a feature extraction network based on a ResNet-50 improvement, which inserts a spatial attention module between the third and fourth convolutional blocks. The magnitude and orientation angle of the deformation vector are extracted from the two-dimensional vector field. The reciprocal of the magnitude is calculated, which approaches zero in severely distorted regions. This reciprocal is upsampled to the same size as the feature map to form a regularization mask, which is then multiplied element-wise with the attention weight map. The deformation vector orientation is converted into a two-dimensional offset field. In the deformable convolutional layer, the sampling position of the standard convolutional kernel is adjusted so that the receptive field can compensate for sampling in the opposite direction of water flow distortion. The feature map after offset sampling and weight suppression is resampled using bilinear interpolation, ultimately extracting the geometrically corrected spatial features of the diver's limbs. These features are highly robust to optical artifacts.

[0025] S103. When the spatial features of the limb are detected to be obscured by bubbles, the normal deformation and tangential movement of the bubble boundary caused by the local water flow displacement are extracted.

[0026] Bubble occlusion is the most common interference factor in underwater vision. Strong reflections from the bubble surface can cause local image saturation and loss of limb features. The brightness changes of the pixel region containing limb features are monitored in real time, and the local brightness gradient between adjacent frames is calculated. When the gradient exceeds a preset threshold of 80 and the confidence of the limb feature suddenly drops below 0.3, bubble occlusion is determined. The Canny edge detection algorithm is used to scan the feature-lost region, with dual thresholds set to 50 and 150, to extract the initial set of pixel coordinates of the bubble boundary. Bubbles deform and displace under the influence of water flow; boundary tracking uses a matching method based on shape context descriptors. In consecutive occluded frames, a shape context histogram is calculated for each boundary point, containing 12 angle bins and 5 radial distance bins. Correspondence is established by minimizing the shape context distance between boundary points in adjacent frames, using a chi-square test formula. The deformation of the boundary in the normal direction is obtained by calculating the boundary curvature change. The tangential movement is calculated through the boundary centroid displacement, where the centroid coordinates are the arithmetic mean of all boundary point coordinates, and the movement is the Euclidean distance between the centroids of adjacent frames.

[0027] S104. Extract the limb motion velocity vector field of the last frame before being occluded by the bubble, perform cross-frame cross-correlation calculation on the limb motion velocity vector field with the normal deformation and tangential movement, invert to generate implicit motion trajectory equation, and use the implicit motion trajectory equation to construct a compensation feature vector and input it into the preset time attention mechanism network to generate an aligned spatiotemporal feature matrix.

[0028] The movement trajectory of the limbs behind the bubble cannot be directly observed and needs to be inferred through physical correlation. The limb feature sequence within the time window before occlusion is extracted from the cache, and the motion velocity vector field of the last visible frame is calculated using the temporal difference method. The velocity vector field, bubble normal deformation, and tangential movement are uniformly mapped to the world coordinate system; this mapping requires camera intrinsic matrix and underwater refractive index correction. A cross-frame cross-correlation function is constructed, and the temporal coupling relationship between limb movement and bubble deformation is determined by finding the peak value of the cross-correlation function. Based on Stokes' drag law in fluid mechanics, there is a linear relationship between the force on the bubble and its surface deformation, and this force also acts on the surrounding fluid and the diver's limbs. Using this mechanical coupling relationship, the three-dimensional motion coordinate sequence of the limbs is inverted using the least squares method. Cubic spline fitting is performed on the coordinate sequence to generate an implicit motion trajectory equation. The trajectory equation is sampled at equal intervals, and kinematic features are extracted to construct a compensation feature vector, whose dimension is consistent with the normal limb features. The compensation vector is input into a temporal attention network, which uses a multi-head self-attention mechanism to calculate the temporal dependencies between features, ultimately generating a time-aligned spatiotemporal feature matrix.

[0029] S105. Map the spatiotemporal feature matrix to a spatiotemporal action topology sequence with a time series distribution, and obtain the local behavioral mutation divergence of the spatiotemporal action topology sequence by calculating the divergence value of the gradient field of the spatiotemporal feature matrix between adjacent time frames.

[0030] The spatiotemporal feature matrix contains information about the diver's movement patterns in time and space, which needs to be converted into quantifiable behavioral descriptors. A graph convolutional network is used to map the feature matrix into a spatiotemporal action topology sequence. The topology is based on the human skeleton, with nodes representing key joints and edges representing limb connections. Each time frame corresponds to a topological snapshot, and the sequence length is typically set to 64 frames to cover the complete action cycle. To detect abrupt behavioral changes, the gradient field of the feature matrix between adjacent frames is calculated. The gradient calculation uses an extended form of the Sobel operator in the time dimension. The divergence of the gradient field is calculated; the divergence value reflects the degree of divergence or convergence of features in local regions.

[0031] S106. When the local behavioral change divergence is detected to exceed the preset stable range within a preset time window, it is determined that the target diver has engaged in abnormal behavior and an alarm signal is output.

[0032] The divergence value of normal behavior exhibits stable fluctuations within the time window, with a statistical distribution approximating a Gaussian distribution. The stable interval is defined as the mean plus or minus two standard deviations, covering approximately 95% of normal samples. When the divergence value continuously exceeds this interval for more than three frames within the preset time window, abnormal behavior is identified. Abnormal behaviors include dangerous states such as sudden falls, violent struggles, and loss of consciousness, immediately triggering audible and visual alarms and recording video footage of the abnormal moment for subsequent analysis.

[0033] In one implementation, a continuous video stream of the diver's underwater working environment is acquired, and the dynamic displacement of suspended particles in the continuous video stream is tracked using high-pass filtering and optical flow. The construction of a two-dimensional vector field of local underwater optical distortion based on the dynamic displacement includes the following steps: Acquire continuous video streams of the underwater working environment of divers and convert the continuous video streams into a sequence of grayscale images; High-pass filtering algorithm is used to filter out the background of grayscale image sequence and separate the underwater suspended particle set; The optical flow method is used to track the two-dimensional displacement coordinates of each suspended particle in an array of suspended particles in adjacent grayscale image sequences; The pixel space corresponding to the grayscale image sequence is divided into multiple local grid regions; The average displacement coordinates of suspended particles in each local grid region are calculated based on two-dimensional displacement coordinates, and a local grid deformation vector is generated. The local mesh deformation vectors of all local mesh regions are spliced ​​and combined to form a two-dimensional vector field of underwater local optical distortion.

[0034] In this embodiment, the underwater video acquisition device is typically equipped with a high-definition camera and a waterproof housing. The camera resolution is recommended to be no less than 1920×1080 pixels to ensure sufficient detail capture. The video acquisition frame rate is set to 25 to 30 frames per second, which ensures motion continuity without excessive data storage burden. The camera must be kept relatively stable during acquisition, which can be achieved using a tripod or a handheld stabilizer by the diver. The acquired raw video stream is in RGB three-channel color format, with each pixel containing red, green, and blue color components, ranging from 0 to 255. Since subsequent optical flow calculations and particle tracking primarily rely on brightness information rather than color information, the color image needs to be converted to a single-channel grayscale image to reduce computational complexity. The grayscale conversion uses a weighted average method, which considers the differences in human eye sensitivity to different colors. The conversion formula is as follows: ,in Representing coordinates grayscale value at that location , , These represent the red, green, and blue channel values ​​for that location, respectively. The weighting coefficients of 0.299, 0.587, and 0.114 are derived from the characteristics of human vision; the human eye is most sensitive to green and least sensitive to blue. The converted grayscale image retains the spatial resolution of the original image; each frame remains a 1920×1080 pixel matrix, but the data volume is reduced to one-third of the original. After a continuous video stream is converted into a grayscale image sequence, each frame in the sequence is numbered chronologically, forming an ordered image set. This sequence provides the basic data input for all subsequent processing steps.

[0035] Suspended particles in the underwater environment include plankton, sediment, and organic debris. These particles are typically between 0.1 and 5 millimeters in size and appear as high-frequency details in images. The background mainly consists of large-scale objects such as divers' bodies, equipment, and seabed topography, corresponding to low-frequency components in the frequency domain. A high-pass filter can suppress low-frequency signals while preserving high-frequency signals, thus separating the background from the particles. Implementing high-pass filtering in the frequency domain requires first performing a two-dimensional Fourier transform on the grayscale image, converting the spatial domain signal into a frequency domain representation. The transfer function of an ideal high-pass filter is... ,in Representing frequency point Distance to the center of the frequency domain This is the cutoff frequency. The cutoff frequency is determined based on the particle size and is typically set to 0.15 to 0.25 times the frequency of the image diagonal. Ideal high-pass filters produce ringing effects; in practical applications, Butterworth high-pass filters are used, with the transfer function modified as follows: ,in The filter order is typically between 2 and 4. The filtered frequency domain image is then converted back to the spatial domain using an inverse Fourier transform, resulting in an image containing only high-frequency components. This image is then binarized, with a threshold set to 1.5 times the mean grayscale value to further separate particles from noise. Connected component analysis identifies independent particle regions; regions smaller than 5 pixels or larger than 100 pixels are considered noise or non-particle objects and are removed, ultimately yielding a collection of underwater suspended particles.

[0036] Optical flow estimates pixel motion based on the assumptions of constant brightness and spatial smoothness. The constant brightness assumption assumes that the brightness value of the same object remains constant between adjacent frames. For each particle in the set of suspended particles, in the current frame... The centroid coordinates of the particle are used as the starting point for tracking. The Lucas-Kanade optical flow algorithm assumes that all pixels in the local neighborhood have the same motion vector and solves for the motion parameters using the least squares method. A 5×5 pixel window is selected around the particle's centroid, and each pixel within the window satisfies the optical flow constraint equation. ,in and The images are respectively in and Spatial gradient of direction, For time gradient, and Let be the horizontal and vertical motion velocities to be determined. The spatial gradient is calculated using the Sobel operator, and the temporal gradient is the gray-level difference between corresponding pixels in adjacent frames. The constraint equations for all pixels within the window constitute an overdetermined system of equations, which are solved using the least squares method to obtain the motion vector. The pyramid optical flow method is used to handle large displacements by constructing the image as a multi-layered pyramid, with each layer having half the resolution of the layer above. Motion is estimated starting from the top layer with coarse resolution, and the results are passed down layer by layer and refined to the bottom layer. For the... Individual particles, in adjacent frames and The displacement vector is obtained by tracing between them. Particles in The new coordinates in are The coordinates are recorded as two-dimensional displacement coordinates and stored in the tracking database.

[0037] Image space partitioning employs a uniform grid strategy, with grid size determined based on image resolution and particle density. For a 1920×1080 resolution image, it is typically divided into 120×68 grids, each grid being 16×16 pixels in size. Grids that are too large will result in insufficient spatial resolution, failing to capture local distortion details; grids that are too small will lead to statistical instability due to insufficient particle count. Each grid region is uniquely identified by its top-left corner coordinates and width and height parameters. Each grid is denoted as Iterate through all tracked particle displacement coordinates and determine which grid region each particle's initial position belongs to. For particles that fall into the grid... Extract the corresponding set of displacement vectors for all particles within the array. ,in This represents the number of particles within the grid. The weighted average of the displacement vectors is calculated as the overall deformation vector of the grid. The weights are proportional to the gradient strength of the particles; particles with higher gradient strength have sharper edge features and higher tracking accuracy. (Grid) The deformation vector is calculated as follows ,in For the first The weight of each particle is defined as follows: This refers to the gradient magnitude. The deformation vector contains horizontal and vertical components, representing the average displacement of the grid in the two directions, respectively. When the number of particles in a grid is less than 3, the grid data is considered unreliable, and bilinear interpolation of the deformation vectors of adjacent grids is used to fill in the gaps. The deformation vectors of all grids are arranged in spatial order to form a matrix structure, which completely describes the local deformation distribution of the entire image space.

[0038] The calculation of local mesh deformation vectors provides independent displacement information for each mesh, but these discrete vectors need to be organized into a continuous field structure to fully describe the optical distortion characteristics. The meshes are numbered from left to right and top to bottom, forming a one-dimensional index sequence. The deformation vector of each mesh is then... Stored as the corresponding elements of a vector field matrix, the matrix dimension is ,in and These represent the number of rows and columns of the grid, respectively, with the third dimension, 2, representing the horizontal and vertical components. The continuity of the vector field is achieved through bicubic interpolation, an interpolation function capable of estimating the deformation vector value at any non-grid point. For the query point... First, the four adjacent grid cells containing the given point are identified, and then interpolation is performed based on distance weights. The vector field also needs to be smoothed to eliminate abrupt changes caused by noise. A Gaussian filter is used to convolve the two components of the vector field, with the filter standard deviation set to 1.5 grid cells. The smoothed vector field exhibits continuous spatial variation characteristics, with no drastic jumps in the direction and magnitude of deformation vectors at adjacent positions. The vector field is visualized using an arrow diagram or color-coded diagram, where the arrow length represents the deformation amplitude and the direction represents the displacement direction. This two-dimensional vector field fully characterizes the spatial distribution pattern of underwater optical distortion, with severely distorted regions corresponding to larger vector magnitudes, and the distortion direction represented by the vector direction. The vector field serves as an important intermediate result, passed to the subsequent feature extraction module to guide the neural network in adaptive geometric correction and attention adjustment.

[0039] The process of concatenating and combining vector fields is essentially a process of reconstructing discretely sampled deformation information into a globally continuous representation. The deformation vectors of all local grid regions have been organized into a matrix structure according to their spatial positions, but this matrix is ​​still a discrete grid-level representation. To support subsequent continuous convolution operations in the neural network, the discrete vector field needs to be upsampled to the same resolution as the original image. Upsampling uses bilinear interpolation. For each pixel position in the original image, weighted interpolation is performed based on the deformation vectors of the grid to which that position belongs and the adjacent grids. The interpolation weight is inversely proportional to the distance from the pixel to the grid center; the closer the distance, the greater the weight. The upsampled vector field has a resolution of 1920×1080, with each pixel position corresponding to a deformation vector. The two components of the vector field are stored as independent channels, forming a dual-channel deformation map. To enhance the spatial consistency of the vector field, edge-preserving filtering is applied again to the upsampled result. This filter smooths the internal regions while maintaining clear distortion boundaries. The final two-dimensional vector field is obtained. It has the same spatial resolution as the input image, providing accurate local distortion information for each pixel. The magnitude distribution of the vector field reflects the spatial variation of distortion intensity, with regions of large magnitude corresponding to locations with large water density gradients. Differential features of the vector field, such as divergence and curl, further reveal the physical nature of the distortion; divergence represents the degree of divergence or convergence of the deformation, and curl represents rotational deformation.

[0040] In one embodiment, tracking the two-dimensional displacement coordinates of each suspended particle in an array of suspended particles using optical flow in adjacent grayscale image sequences includes the following steps: Assign a unique tracking identifier to each suspended particle in the set of suspended particles in the grayscale image sequence of the current frame; An optical flow evaluation model based on an image pyramid is constructed by inputting the current frame grayscale image sequence containing tracking identifiers and the adjacent next frame grayscale image sequence into the optical flow evaluation model. The initial coarse displacement of each suspended particle is calculated in the top-level image of the optical flow evaluation model; The initial coarse displacement is projected as a priori parameter into the underlying image of the optical flow evaluation model; By combining nonlocal spatial smoothing constraints, the initial coarse displacement in the underlying image is iteratively optimized to eliminate the interference of underwater high-frequency noise on the tracking trajectory and obtain the target displacement vector of each suspended particle. The two-dimensional displacement coordinates of each suspended particle in adjacent grayscale image sequences are determined based on the target displacement vector.

[0041] In this embodiment, in the grayscale image of the current frame, the set of suspended particles has been separated through the aforementioned high-pass filtering and binarization processing, with each particle representing a connected pixel region. A connected component labeling algorithm is used to assign an initial identifier to each particle region. The labeling process starts from the top left corner of the image and proceeds in raster scan order. When an unlabeled foreground pixel is encountered, a new identifier is created and propagated to the entire connected region. Identifiers are assigned integer numbers, starting from 1 and incrementing, avoiding the use of 0 to distinguish the background. Each particle's identifier is bound to its geometric features, which include descriptors such as centroid coordinates, area, major axis direction, and compactness. The centroid coordinates are calculated by the arithmetic mean of all pixel coordinates within the region, the area is the number of pixels, and compactness is defined as the ratio of the square of the perimeter to the area. Identification information is stored in a hash table data structure, using the identifier number as the key and the feature vector as the value, supporting fast querying and updating.

[0042] An image pyramid is a multi-scale representation structure that constructs a coarse-to-fine image hierarchy through layer-by-layer downsampling, effectively handling large-scale motion and improving computational efficiency. The pyramid construction starts with the original grayscale image as layer 0 (the bottom layer), halving the resolution with each subsequent layer until the top layer's image size is reduced to approximately 64×64 pixels. Downsampling uses Gaussian filtering followed by interval sampling, with the Gaussian kernel standard deviation set to 0.8 to balance blur and aliasing. For a resolution of [missing information - likely a specific resolution], [missing information - likely a specific resolution], [missing information - likely a specific resolution]. The Layer image, the previous layer has a resolution of The number of pyramid layers is determined based on the image size and the expected maximum displacement, typically ranging from 4 to 6 layers. Separate image pyramids are constructed for the current frame and the next frame, with both pyramids having the same number of layers and size. Before pyramid construction, the current frame, containing tracking markers, encodes the marker information into an additional label map. This label map has the same size as the grayscale image, with each pixel value corresponding to the marker number of its respective particle. The label map also undergoes pyramid downsampling, but uses nearest-neighbor interpolation instead of Gaussian filtering to preserve the discreteness of the markers. At each layer of the pyramid, the label map indicates the spatial extent of the particle region, allowing optical flow calculations to focus on the particle location while ignoring the background region. The optical flow evaluation model is based on the classic Lucas-Kanade framework but extended to an iterative pyramid version. The model input includes three data structures: the current frame pyramid, the next frame pyramid, and the label pyramid. Starting from the top layer of the pyramid, an initial optical flow estimate is calculated for each marker particle, and the estimate is then passed to the next layer as an initial guess for refinement at higher resolutions. This coarse-to-fine strategy avoids optical flow calculations getting trapped in local optima and is particularly suitable for the large displacements that particles may produce in underwater environments. The pyramid structure also reduces computational complexity, with the top-level image having only one-sixteenth the number of pixels as the bottom-level image, significantly reducing the computational cost of iterative optimization.

[0043] The top-level image has the lowest resolution but the largest coverage, making it suitable for capturing the overall motion trend of particles without being affected by detailed noise. In the top-level image, each labeled particle typically occupies only a few pixels, and the centroid coordinates of the particles are determined through the label map. For particles labeled as... The particles are located in the top-level image of the current frame to determine their centroids. Select around this location A search window for pixels. Template matching is performed in the corresponding region of the top-level image in the next frame, with the template being the area surrounding the particles in the current frame. Pixel blocks. Matching uses normalized cross-correlation coefficients as a similarity measure, calculated using the following formula: ,in For template image blocks, For the search area, and These are the mean, This represents the displacement offset. The position with the highest correlation coefficient within the search window is the matching position of the particle in the next frame, and the displacement vector is... This displacement vector represents the coarse motion of the particle at the scale of the top-level image. Since the top-level image has undergone multiple downsampling steps, the actual physical displacement needs to be multiplied by a scaling factor. For a 4-layer pyramid, the top layer corresponds to the 3rd layer, and the scaling factor is... The initial coarse displacement is calculated as follows: While the coarse displacement has limited precision, it provides a reliable direction and approximate magnitude of motion, avoiding the problem of an excessively large search space during the finer-level refinement. For all identified particles, the coarse displacement is calculated in batches and stored as a displacement vector array, with the array index corresponding to the particle identifier.

[0044] The core mechanism of the pyramid optical flow method is the projection of coarse displacement from the top to the bottom. This projection process needs to consider the scale transformation caused by changes in resolution. From the... Layer to the first During layer projection, each component of the displacement vector needs to be multiplied by 2 because the spatial resolution of the next layer doubles. Projection not only transmits the displacement value but also the confidence information of the displacement. The confidence level is determined based on the correlation coefficient of the top-level matching; a higher correlation coefficient indicates a higher confidence level. In the bottom-level image, the coarse displacement of the projection serves as the initial value for optical flow optimization, defining the center position of the search neighborhood. For particles... Assuming in the first Displacement estimates have been obtained for the layer. Projected to the The initial displacement during layering is set to In the first In the layer image, the coordinates of the centroid of the particles are also doubled. Centered on the initial displacement, in A local search region is established within a pixel area, much smaller than the full-image search, significantly reducing computational cost. The projected displacement is also used for image registration, shifting the next frame image in the opposite direction according to the displacement vector, so that the positions of particles in the two frames are roughly aligned. The residual between the registered image pairs is small, which is beneficial for subsequent fine-tuning. The projection process is performed layer by layer, from the top layer all the way down to the bottom layer, i.e., the original resolution image. With each layer down, the displacement accuracy theoretically doubles, but noise and detail interference also increase.

[0045] High-frequency noise in underwater environments primarily originates from light scattering, sensor thermal noise, and water flow disturbances. These noises can cause random fluctuations in optical flow estimation. Traditional optical flow methods employ local smoothing constraints, assuming that adjacent pixels have similar motion vectors. However, local constraints have limited ability to suppress strong noise. Non-local spatial smoothing constraints, drawing inspiration from the non-local mean concept in image denoising, assume that similar image patches should have similar motions, even if these patches are spatially far apart. For particles... Extract the surrounding area from the location in the underlying image. Searching for the neighborhood of a pixel, where each pixel in the neighborhood... Corresponding to one Image blocks Calculate the central particle image patch. The similarity with neighboring image patches is measured using a Gaussian weighted sum of the gray-level differences between patches. The similarity weight is defined as follows: ,in As a smoothing parameter, it is typically set to 1.5 times the noise standard deviation. Spatial distance weighting is... ,in Indicates position coordinates, This is the spatial scale parameter. The overall weight is the product of the similarity weight and the spatial weight. The nonlocal smoothing constraint term is represented as the difference between the particle displacement and the neighborhood weighted average displacement, and the constraint energy is... A total energy function is constructed by combining smoothing constraints with optical flow data terms, where the data term measures the brightness consistency of the image patch after displacement. The energy function is iteratively minimized using the conjugate gradient method, with the displacement vector updated in each iteration to decrease the energy. The rate of energy change is monitored during the iteration process, and the process terminates when the energy change is less than a threshold of 0.01% for two consecutive iterations or when the number of iterations exceeds 50. The converged displacement vector is the target displacement vector. This vector combines data fidelity and spatial smoothness, effectively suppressing noise interference while preserving true motion information.

[0046] The target displacement vector represents the particle's motion from the current frame to the next frame, containing both horizontal and vertical components. For those identified as... The centroid coordinates of the particles in the current frame are: The target displacement vector is The two-dimensional displacement coordinates of the particles in the next frame are directly calculated through coordinate transformation, using the following formula: The calculated coordinates may be non-integer values ​​because optical flow estimation achieves sub-pixel accuracy, and non-integer coordinates are used to locate the corresponding grayscale values ​​in the image through bilinear interpolation. Two-dimensional displacement coordinates not only record the new position of the particle but also implicitly contain information about its velocity and direction. These displacement coordinates are stored in the tracking database and associated with particle identifiers to form a temporal trajectory chain. For some particles, occlusion or leaving the field of view may prevent a match from being found in the next frame; the displacement coordinates of these particles are marked as invalid, and the corresponding identifiers are temporarily suspended in the tracking database. Suspended identifiers are retained for a certain time window. If the particle reappears in a subsequent frame and the feature match is successful, the identifier is restored and the trajectory is completed; if it does not reappear within the timeout period, the identifier is deleted to release resources.

[0047] In one implementation, the initial coarse displacement in the underlying high-resolution image is iteratively optimized by incorporating nonlocal spatial smoothing constraints to eliminate the interference of underwater high-frequency noise on the tracking trajectory, and the target displacement vector of each suspended particle is obtained by the following steps: A non-local neighborhood window is constructed with each suspended particle in the underlying image as the center. Calculate the gray-level similarity weight and spatial distance weight of adjacent pixels within a non-local neighborhood window; The gray-level similarity weight and spatial distance weight are combined to generate a nonlocal spatial smoothing constraint term. A global energy functional optimization objective is constructed by combining nonlocal spatial smoothing constraints with optical flow data terms. The global energy functional optimization objective is solved iteratively using the conjugate gradient descent method. When the convergence condition is met, the target displacement vector of each suspended particle is output.

[0048] In this embodiment, in the underlying image, i.e., the original resolution grayscale image, each suspended particle has already obtained its centroid coordinates and tracking identifier through the aforementioned steps. A square neighborhood window is constructed, centered on the particle's centroid, and its side length is adaptively determined based on the particle size and image resolution. For typical underwater suspended particles, their diameter occupies 3 to 8 pixels in the image, and the neighborhood window side length is set to 5 to 7 times the particle diameter, typically 21 to 49 pixels. A window that is too small limits the effectiveness of nonlocal search, while a window that is too large increases the computational burden and may introduce irrelevant regions. Window boundaries require boundary checks. When a particle approaches the image edge, the window may extend beyond the image range. In this case, a mirror fill or constant fill strategy is used to expand the pixel values ​​outside the boundary. Mirror fill mirrors the boundary pixels along the boundary, maintaining image continuity; constant fill uses the image mean to fill the region outside the boundary. The number of pixels contained within the window is the square of the side length; for example, a 21×21 window contains 441 pixels. Each window pixel records not only its grayscale value but also its spatial offset coordinates relative to the particle's centroid. After the window is constructed, local image patches surrounding each pixel within the window need to be extracted. These patches are typically 5×5 or 7×7 pixels in size and are used for subsequent texture similarity calculations. Image patch extraction employs a sliding window approach, extracting a fixed-size image patch centered on each pixel within the neighborhood window. All image patches are organized into a three-dimensional array structure, with the first and second dimensions corresponding to the spatial positions of the neighborhood windows, and the third dimension being the flattened vector of the image patch.

[0049] Gray-level similarity weights measure the similarity in texture patterns between different pixels within a neighborhood window and the central particle. Pixels with high similarity should have similar motion characteristics. A reference image patch extracted from the central particle's location is denoted as a vector containing 25 pixel values ​​and compared with the image patch at each candidate pixel location within the neighborhood window. The difference between image patches is measured using Euclidean distance; the smaller the distance, the more similar the textures. Since the original Euclidean distance is sensitive to illumination changes, the image patches are normalized before calculation by subtracting the mean and dividing by the standard deviation, ensuring the comparison is based on relative gray-level patterns rather than absolute brightness. The normalized distance is converted into similarity weights, which are then soft-mapped using a Gaussian kernel function. The weight formula is as follows: ,in Represents the central particle image patch. Indicates position Image patch For smoothing parameters. Typically, the standard deviation of the image noise is set to 1.5 to 2 times. The standard deviation of underwater image noise can be estimated using the gray-level variance of a uniform region. Spatial distance weights are based on the geometric distance between a pixel and the central particle; pixels closer to the central particle have a greater influence on its motion. The spatial distance uses a Euclidean metric, and the weights also use a Gaussian kernel function, as shown in the formula: ,in Indicates the coordinates of the central particle's position. Represents pixels Location coordinates, This is a spatial scale parameter, typically set to one-third of the neighborhood window radius. Spatial weights ensure that even distant pixels with high texture similarity will not have an excessive impact on the central particle, maintaining the principle of spatial locality.

[0050] Gray-level similarity weights and spatial distance weights capture texture consistency and spatial proximity, respectively. Their fusion leverages multi-dimensional information to construct more robust smoothing constraints. The fusion employs element-wise multiplication; for each pixel position within the neighborhood window, the comprehensive weight is calculated as follows: The multiplication operation achieves the effect of logical AND, ensuring that only pixels that simultaneously satisfy both texture similarity and spatial proximity receive a higher overall weight. The overall weights need to be normalized so that the sum of the weights of all pixels within the window is 1, guaranteeing the probabilistic interpretation of the weights—that is, the proportion of each pixel's contribution to the central particle's motion. Based on the normalized weights, a nonlocal spatial smoothing constraint term is constructed. This constraint term expresses the prior assumption that the central particle's motion vector should approximate the weighted average motion vector within its neighborhood. The energy form of the constraint term is the square norm of the difference between the central particle's displacement and the neighborhood's weighted average displacement, mathematically expressed as... ,in For the central particle The displacement vector, For position The displacement vector of the pixel at that location. This constraint term penalizes spatial discontinuities in the displacement vector; similar pixels with larger weights contribute more to the constraint strength. Compared to traditional local smoothing constraints, nonlocal constraints can find similar structures across boundaries and occluded regions, and have a stronger ability to suppress noise and outliers. In underwater scenes, suspended particles may be affected by water flow from different directions. Nonlocal constraints, by finding groups of particles with similar motion patterns, can effectively distinguish between real motion and noise disturbances, improving the consistency and accuracy of optical flow estimation.

[0051] The global energy functional optimization objective comprehensively considers both data fidelity and spatial smoothness, solving for the optimal motion vector field by minimizing the energy function. The optical flow data term measures whether the motion vector can maintain consistent image brightness between adjacent frames, and is constructed based on the fundamental optical flow constraint equations. For particles... The energy of the data item is defined as the sum of squares of the brightness residuals after motion compensation, and its expression is: ,in microparticles Covered pixel area For the grayscale image of the next frame, and Let be the motion vector components of the particle. The data term requires that the motion vectors accurately align between two image frames; a smaller residual indicates better alignment. The data term is linearly combined with a non-local spatial smoothness constraint term to construct a global energy functional. ,in To smooth the weighting coefficients, a balance is maintained between data fidelity and smoothness. Coefficients The choice of depends on the image noise level; higher noise requires stronger smoothing constraints, typically ranging from 0.5 to 5. The energy functional is a multivariate function of all particle motion vectors, and the goal is to find the vector field configuration that minimizes energy. The energy function is nonlinear because image brightness requires interpolation at non-integer coordinates, introducing nonlinearity. The nonlinear terms are linearized using Taylor expansion, and a first-order approximation is made near the current motion vector estimate, transforming the problem into a linear least squares problem. The linearized energy function is quadratic, which can be represented in matrix form for easier numerical solution.

[0052] Global energy functional minimization is equivalent to solving a system of normal equations, where the coefficient matrix is ​​a symmetric positive definite matrix, satisfying the application conditions of the conjugate gradient method. The algorithm starts with initial motion vector estimation, typically set as a coarse displacement projected from the upper layer of the pyramid. An initial residual vector is calculated, representing the deviation direction between the current solution and the true solution. The initial search direction is set as the residual direction. During iteration, a one-dimensional line search is performed along the search direction to determine the optimal step size, calculated analytically. After updating the motion vector, a new residual vector is calculated. A conjugate coefficient is calculated based on the old and new residuals, measuring the relative magnitude of the residual change. When updating the search direction, the new residual and the previous search direction are combined; the conjugate direction ensures the orthogonality of the search path, avoiding repeated searches of already optimized directions and accelerating convergence. Convergence conditions include the residual norm being less than a threshold or the number of iterations reaching an upper limit. The residual norm threshold is typically set to 0.1% of the initial residual, and the maximum number of iterations is set to 50 to 100. When the residual norm... or number of iterations The iteration terminates at time, where This is the relative error threshold. This represents the maximum number of iterations. When the convergence condition is met, the current motion vector is the target displacement vector. Output the displacement results of all particles.

[0053] In one implementation, in a pre-defined feature extraction network with a spatial attention mechanism, the reciprocal of the magnitude of the deformation vector in the two-dimensional vector field is used as a regularization term. Simultaneously, the direction of the deformation vector in the two-dimensional vector field is used to guide the receptive field of the feature extraction network to shift. Extracting the spatial features of a diver's limbs from a continuous video stream includes the following steps: Continuous video frames are input into a pre-defined feature extraction network with a spatial attention mechanism to generate an initial feature map; Extract the magnitude and direction of the deformation vector for each local grid region from the two-dimensional vector field; Calculate the reciprocal of the magnitude of the deformation vector and map the reciprocal of the magnitude of the deformation vector to a regularization penalty mask; The regularization penalty mask is multiplied element-wise with the initial attention weight map of the feature extraction network to suppress the attention weights in high distortion regions. The deformation vector direction is converted into a two-dimensional offset field, and the two-dimensional offset field is injected into the deformable convolutional layer of the feature extraction network. Deformable convolutional layers are used, and the receptive field of the convolutional kernel is guided to shift in the opposite direction of the water flow distortion according to a two-dimensional offset field. Based on the offset receptive field and the suppressed attention weight, the initial feature map is resampled and fused to extract the spatial features of the diver's limbs after eliminating optical artifacts.

[0054] In this embodiment, refer to Figure 2 The feature extraction network employs a deep convolutional neural network architecture, using ResNet-50 as the backbone and embedding a spatial attention module between the third and fourth convolutional blocks. Continuous video frames are first preprocessed, with the resolution uniformly adjusted to 512×512 pixels and pixel values ​​normalized to between 0 and 1. Video frames are input into the network in RGB three-channel format. The first convolutional layer uses a 7×7 kernel to extract low-level texture features, with a stride of 2 for initial downsampling. After a max-pooling layer, the feature map size is reduced to 128×128, and the number of channels is expanded to 64. Subsequently, deep semantic features are extracted progressively through four residual block groups. Each residual block contains multiple residual units, and residual connections allow for direct gradient propagation, avoiding degradation issues. A spatial attention module is inserted at the output of the third residual block. This module first performs global average pooling and global max pooling on the feature map, obtaining two one-dimensional channel descriptors. The two descriptors are concatenated and then used to generate channel attention weights through a shared multilayer perceptron. These weights are multiplied by the original feature map to achieve channel-level feature recalibration. The channel-weighted feature maps are then subjected to average pooling and max pooling along the channel dimension to obtain two spatial attention maps. These two spatial attention maps are concatenated and passed through a 7×7 convolutional layer to generate a spatial attention weight map. The weight map has the same spatial dimension as the feature map, and the weight value at each location is between 0 and 1. The spatial attention weight map is element-wise multiplied with the feature map to achieve adaptive weighting at spatial locations, highlighting important regions and suppressing irrelevant background. The attention-modulated feature map is then passed through a fourth residual block, ultimately outputting an initial feature map of size 32×32 with 2048 channels.

[0055] A two-dimensional vector field typically has a resolution of 120×68, meaning it contains 8160 grid cells, each covering a 16×16 pixel area of ​​the original image. Extracting deformation information from the vector field requires traversing all grid cells and calculating the geometric properties of the deformation vector for each cell. The magnitude of the deformation vector represents the distortion intensity of that grid region, and its calculation formula is... ,in and They are grids The deformation vector comprises the horizontal and vertical components. The modulus is measured in pixels, typically ranging from 0 to 10 pixels; a larger modulus indicates more severe optical distortion. The direction of the deformation vector is calculated using the arctangent function, as shown in the formula: The angle value, ranging from -π to π, represents the dominant direction of deformation. The calculation of the orientation angle requires handling the special case where the denominator is zero. When the horizontal component is close to zero, the angle is set to π / 2 or -π / 2 based on the sign of the vertical component. The extracted modulus and orientation information are stored as two independent matrices, with matrix sizes consistent with the grid resolution. To match the feature map size of the feature extraction network, the modulus and orientation matrices need to be upsampled or downsampled. Bilinear interpolation is used to interpolate the grid-level deformation information to the spatial resolution of the feature map. For a 32×32 feature map, each feature location corresponds to a 16×16 pixel region in the original image.

[0056] The reciprocal of the deformation vector's magnitude reflects the reliability of the distorted region. Regions with large magnitudes correspond to severe distortions, resulting in low reliability of feature extraction, and the attention weight should be reduced accordingly. The calculation of the reciprocal of the magnitude requires adding a smoothing factor to prevent division by zero errors. The formula is as follows: ,in This is a smoothing factor. The reciprocal value ranges from 0.1 to 100. When the distortion mode length is close to zero, the reciprocal approaches 100, and when the distortion mode length is 10 pixels, the reciprocal is approximately 0.1. The reciprocal value needs to be normalized to fit the numerical range of the attention weights. A non-linear mapping is performed using the Sigmoid function, and the mapping formula is as follows: ,in For steepness parameter, The center offset parameter is used. The mapped values ​​range from 0 to 1, exhibiting an S-shaped curve distribution. Mapped values ​​are close to 1 in low-distortion regions and close to 0 in high-distortion regions. These mapped values ​​constitute a regularization penalty mask, with a size identical to the feature map's spatial dimension (32×32). Each element of the mask corresponds to a spatial location in the feature map, and the value represents the confidence weight of the feature at that location. The regularization penalty mask is essentially a soft constraint; it doesn't hard-remove distorted regions but rather reduces their weights, preserving a certain degree of information flow. The mask generation process converts the distortion intensity in physical space into weight coefficients in the feature space, enabling cross-layer transfer from low-level visual information to high-level semantic features. The mask can also undergo spatial smoothing by convolving it with a 3×3 Gaussian filter to eliminate isolated outliers and make the weight distribution more continuous.

[0057] The spatial attention module of the feature extraction network has generated an initial attention weight map. This weight map is adaptively calculated based on the content of the feature map, reflecting the semantic importance of different spatial locations. The initial attention weight map is 32×32 pixels, with a weight value between 0 and 1 for each location. Locations with high weights correspond to foreground targets such as divers' limbs, while locations with low weights correspond to background regions. A regularized penalty mask provides a geometric reliability assessment based on optical distortion; the combination of these two methods comprehensively considers both semantic importance and geometric reliability. Element-wise multiplication achieves the fusion of the two weight maps, calculated using the following formula: ,in For the initial attention weight map at position The value, This represents the corresponding value of the regularization penalty mask. The multiplication operation has the effect of logical AND, and only positions that simultaneously satisfy semantic importance and geometric reliability can maintain a high weight. For high-distortion regions, even if the semantic importance is high, the weights after multiplication will be significantly suppressed because the mask value is close to zero. (Suppressed attention weight map) Renormalization is performed so that the weights at all positions sum to 1, preserving the probability distribution characteristics. Normalization is achieved by dividing by the sum of the weights, ensuring that weight adjustments do not alter the overall energy of the feature map. Suppression effectively reduces the interference of distorted regions on feature extraction, allowing the network to focus more on clear and reliable image regions. In underwater scenes, local distortions caused by water flow disturbances are often concentrated in specific areas; weight suppression can prevent erroneous features from these areas from contaminating the overall representation. The suppressed weight map serves as a new attention mask, which is element-wise multiplied with the feature map to achieve adaptive feature recalibration.

[0058] The deformation vector direction represents the dominant direction of optical distortion, reflecting the image warping trend caused by water flow disturbance. Converting the direction information into a two-dimensional offset field requires coordinate decomposition, which includes two channels: a horizontal offset component and a vertical offset component. For the mesh... Deformation direction angle The horizontal offset is calculated as follows Vertical offset is calculated as ,in This represents the offset amplitude coefficient; the negative sign indicates an offset in the opposite direction of distortion. The offset amplitude coefficient is adaptively adjusted based on the distortion intensity. Strongly distorted regions require a larger offset for compensation, and the coefficient can be set to [value missing]. The offset is proportional to the degree of distortion but does not exceed 5 pixels. The offset field is upsampled to a feature map resolution of 32×32, with each feature location corresponding to a two-dimensional offset vector. The offset field is injected into a deformable convolutional layer, which is an extension of standard convolution, allowing the sampling position of the convolution kernel to be dynamically adjusted according to the input. Standard convolution samples on a regular grid, while deformable convolution samples at the offset position, with the sampling position being... ,in This represents the standard grid position. The offset field injection is achieved by adding the offset vector to the offset parameters of the deformable convolutional layer, allowing the receptive field of the convolutional kernel to adaptively adjust to follow the distortion direction. Deformable convolutional layers are typically placed deep within feature extraction networks to perform geometric correction on high-level semantic features. The introduction of the offset field enables the convolutional operation to possess geometric transformation capabilities, compensating for image-level distortions at the feature level and achieving implicit geometric alignment.

[0059] Deformable convolutional layers adjust the sampling positions of the convolutional kernels based on the injected two-dimensional offset field, achieving dynamic shifting of the receptive field. Standard convolutional kernels sample on the feature map using a fixed rectangular grid; for example, a 3×3 convolutional kernel samples at 9 locations. Deformable convolutions add an offset at each sampling location, allowing the sampling points to deviate from the regular grid. For feature map locations… The first convolution kernel The original location of each sampling point is ,in For example, the 9 relative offsets of a 3×3 convolution kernel are { , ,...,(1,1)}。 After adding dynamic offset, the actual sampling position becomes Offset The data is read from a 2D offset field. The offset direction is opposite to the water flow distortion direction, allowing the convolutional kernel to trace back to the original position before distortion for sampling, achieving a geometric correction effect. Since the sampled position after offset is usually a non-integer coordinate, feature values ​​need to be obtained through bilinear interpolation. Bilinear interpolation is calculated by weighting the feature values ​​of four integer positions around the sampled point, with the weights inversely proportional to the distance. The output feature map of deformable convolution combines offset sampling and convolution operations, preserving the translation invariance of convolution while gaining the flexibility of geometric transformation. The offset of the receptive field enables the network to extract geometrically consistent features in distorted images, avoiding feature misalignment and blurring caused by distortion. In underwater scenes, the local distortion caused by water flow disturbance is effectively compensated by the offset field, and the convolutional kernel can aggregate information along the correct geometric path.

[0060] After geometric correction through deformable convolution and suppression adjustment of attention weights, the feature maps already possess strong anti-distortion capabilities, but further resampling and fusion operations are still needed to generate the final limb spatial features. The resampling process aligns the initial feature map based on the offset receptive field position using spatial transformation. For each position in the feature map, the source position is calculated according to the corresponding offset vector, and feature values ​​are extracted from the initial feature map using bilinear interpolation. Resampling achieves geometric alignment of features, ensuring spatial consistency across different positions. The suppressed attention weight map serves as the weight coefficient for feature fusion, performing weighted fusion with the resampling feature map. The fusion operation employs element-wise multiplication followed by summation, with high-weight positions contributing more to features and low-weight positions contributing less. Fusion also includes the integration of multi-scale features, concatenating feature maps from different convolutional layers after upsampling alignment to form a comprehensive feature representation containing multi-level information. After feature concatenation, channel dimensionality reduction is performed using 1×1 convolution, compressing high-dimensional features into a fixed-dimensional limb spatial feature vector. The dimension of the limb spatial feature vector is typically set to 512 or 1024, which preserves rich semantic information while controlling computational complexity. The extracted limb spatial features have eliminated the influence of optical artifacts and clearly represent the diver's limb position, posture, and motion state. The robustness of the features is ensured through the synergistic effect of geometric correction, attention modulation, and multi-scale fusion, enabling stable extraction even in complex underwater environments.

[0061] In one embodiment, mapping the spatiotemporal feature matrix to a spatiotemporal action topology sequence with a time-series distribution, and obtaining the local behavioral abrupt divergence divergence of the spatiotemporal action topology sequence by calculating the divergence value of the gradient field of the spatiotemporal feature matrix between adjacent time frames, includes the following steps: The spatiotemporal feature matrix continuously output by the temporal attention mechanism network is expanded into a multidimensional tensor according to the timestamp order; The nodes after the multidimensional tensor is expanded are defined as action topology nodes, and a spatiotemporal action topology sequence with time series distribution is constructed. The Euclidean distance between action topology nodes in the multidimensional tensor space between adjacent time frames is defined as the node spacing. By taking the first-order partial derivative of the node spacing on the time axis, the gradient field of the spatiotemporal feature matrix between adjacent time frames is obtained. The nonlinear divergence value is solved by applying the nonlinear Laplace operator to the gradient field of the spatiotemporal characteristic matrix; By using the nonlinear divergence value as the energy representation of topological dislocations generated by the spatiotemporal action topological sequence, the local behavioral mutation divergence representing the action evolution state is obtained.

[0062] In this embodiment, the temporal attention mechanism network performs temporal modeling on the input feature sequence, capturing the dependencies between features at different times through a self-attention mechanism. The spatiotemporal feature matrices output by the network are arranged in the order of the timestamps of the video frames, forming a temporal feature set. Each spatiotemporal feature matrix corresponds to one video frame, and the matrix dimension is typically [missing information]. ,in and These are the space height and width, respectively. denoted as the number of feature channels. For a typical diver behavior detection task, the spatial resolution is set to 8×8, and the number of channels is 512. Therefore, a single-frame feature matrix contains 32768 feature values. Multidimensional tensor expansion reshapes the three-dimensional feature matrix into a one-dimensional vector. The expansion order follows a row-major principle: first expand along the width direction, then along the height direction, and finally along the channel direction. The length of the expanded feature vector is... All feature values ​​are arranged in a fixed index order. For features containing Video segments of a certain time frame are expanded to generate... There are several one-dimensional feature vectors, which form a two-dimensional tensor. The first dimension is the time axis, and the second dimension is the feature dimension. The purpose of tensor expansion is to transform spatially distributed features into a unified vector representation, facilitating subsequent distance calculations and topological analysis. The expansion process preserves the complete information of the features without losing any values; it only changes the organization of the data. The expanded tensor can be viewed as a set of points in a high-dimensional space, where each point represents the diver's state at a given moment, and the coordinates of the point are determined by the components of the feature vectors.

[0063] Action topology nodes are abstract representations of a diver's motion state at a specific moment, with each node corresponding to a one-dimensional feature vector after expansion. Nodes not only contain feature values ​​but also carry timestamp information, identifying the node's position in the time series. For the... Each time frame, the action topology node is denoted as... The feature vector of the node is ,in This is the feature dimension. Nodes from all time frames are connected in chronological order to form a spatiotemporal action topology sequence. The topological sequence has a chain-like structure, with temporal dependencies between adjacent nodes; the state of the previous node influences the evolution of the next. The construction of the topological sequence borrows the concept of paths from graph theory, treating the time series as a directed graph, with nodes as vertices and the flow of time as directed edges. Unlike traditional graph structures, action topological sequences only include connections in the temporal dimension, neglecting lateral connections in the spatial dimension; therefore, they are linear topologies rather than mesh topologies. The temporal distribution characteristics of the sequence are reflected in the uniformity of node intervals; the time interval between adjacent nodes is constant, equal to the reciprocal of the video frame rate. For example, a video of 30 frames per second corresponds to approximately 33 milliseconds of node intervals. The spatiotemporal action topological sequence fully records the evolution of the diver's behavior; normal behavior corresponds to a smooth and continuous topological trajectory, while abnormal behavior corresponds to abrupt changes or breaks in the topological trajectory.

[0064] Node spacing is a geometric measure of the difference in a diver's state between adjacent time points, reflecting the speed and magnitude of behavioral evolution. In a multidimensional tensor space, each action topology node corresponds to a high-dimensional point, and the node's coordinates are determined by the components of its eigenvector. Adjacent nodes... and The Euclidean distance between them is calculated as follows: ,in Represents a node The eigenvector of the first Each component has a distance. Euclidean distance is the most commonly used distance metric, possessing rotation invariance and scale consistency, making it suitable for measuring the distance between points in high-dimensional space. The magnitude of the distance value directly reflects the drasticness of feature changes; a small distance indicates similar features between adjacent frames, corresponding to smooth motion; a large distance indicates significant feature differences, corresponding to rapid motion or abrupt pose changes. Node spacing sequence This constitutes a one-dimensional time signal, and the fluctuation pattern of the signal reveals the dynamic characteristics of the behavior. Normal behavior exhibits periodic or gradual changes in node spacing; for example, swimming movements correspond to periodic distance fluctuations, and slow movement corresponds to small-amplitude distance changes. Abnormal behavior causes sudden increases or decreases in node spacing; for example, a sudden fall corresponds to a sharp increase in distance, and loss of consciousness corresponds to a continuous decrease in distance. The computational complexity of node spacing is O(n log n). For a 512-dimensional feature vector, a single calculation requires only a few hundred floating-point operations, which meets the requirements for real-time processing.

[0065] The gradient field of the spatiotemporal feature matrix describes the rate of change of features over time and is a core tool for behavioral dynamics analysis. The rate of change of distance is obtained by taking the first-order partial derivative of the node spacing sequence on the time axis, which reflects the acceleration characteristics of behavioral evolution. The first-order partial derivative is calculated using numerical differentiation methods. For discrete-time series, the forward difference formula is: ,in The time interval is used. Forward difference calculation is simple but has limited accuracy; the central difference formula... It achieves higher accuracy by reducing truncation error through symmetric sampling. The gradient field contains not only the rate of change of distance but also the time derivatives of each component of the eigenvector. For the eigenvector... The gradient vector is calculated as follows: Each component of the gradient vector represents the rate of change of the corresponding feature channel. The magnitude of the gradient field... The gradient field represents the total rate of feature change, and its direction indicates the dominant mode of change. (Time series of the gradient field) The gradient field constitutes a high-dimensional vector field, and the streamlines of the vector field reflect the evolutionary trajectory of the behavioral state. The gradient field of normal behavior exhibits a smooth and continuous distribution, with the direction and amplitude of the gradient vector changing slowly. Abnormal behavior leads to singularities or discontinuities in the gradient field; for example, a sudden stop corresponds to a sharp decay of the gradient vector, and a sudden change in direction corresponds to a rotation of the gradient vector.

[0066] The Laplace operator is a second-order differential operator used to measure the local curvature and divergence of scalar or vector fields. The traditional Laplace operator is a linear operator, defined as the divergence of the gradient, and is applicable to linear systems. The nonlinear Laplace operator is a generalization of the linear operator, capable of handling complex behavioral patterns in nonlinear dynamical systems. For the gradient field of the spatiotemporal characteristic matrix... The result of the nonlinear Laplace operator is the nonlinear divergence value, calculated using the following formula: ,in It is a non-linear exponent, typically taking a value between 1.5 and 2. When It degenerates into the standard Laplace operator when This exhibits nonlinear characteristics. The nonlinear term... Gradient magnitudes are weighted to enable the operator to adapt to gradients of different magnitudes. Divergence is calculated in a discrete form, by taking the partial derivatives of each component of the gradient field and then summing them. For a one-dimensional time series, divergence simplifies to the time derivative, calculated as follows: The physical meaning of nonlinear divergence is the net outflow of the characteristic flow. Positive divergence indicates local divergence of the characteristic, negative divergence indicates local convergence, and zero divergence indicates that the characteristic remains conserved. In behavioral analysis, divergence reflects the stability of action patterns; stable actions correspond to small divergence, and unstable actions correspond to large divergence. Nonlinear divergence is more sensitive to anomalous behavior because nonlinear weighting amplifies the abrupt change effect of the gradient, making anomalous features more prominent in the divergence field.

[0067] The spatiotemporal action topological sequence is viewed as a dynamic lattice, with normal behavior corresponding to regularly arranged lattice nodes and anomalous behavior corresponding to dislocations or defects in the lattice. The nonlinear divergence value, as an energy representation of topological dislocations, quantifies the degree to which the topological structure deviates from its normal state. A larger energy representation indicates more severe topological dislocations and a higher probability of anomalous behavior. The local behavioral mutation divergence is defined as the statistical characteristic of the nonlinear divergence value within a time window, calculated using the following formula: ,in The time window width is typically set to 1 to 2 seconds, corresponding to the number of frames. Window averaging eliminates the influence of transient noise and extracts persistent behavioral change features. (Time series of local behavioral abrupt change divergence) This quantitative description of behavioral evolution states is used, with peak values ​​in the sequence corresponding to moments of behavioral abrupt changes and stationary segments corresponding to periods of normal behavior. The threshold for abrupt change divergence is determined using statistical learning methods. The divergence distribution is calculated for normal behavior samples, and the mean plus twice the standard deviation is used as the anomaly threshold. When the divergence value exceeds the threshold and persists for more than a preset number of frames, abnormal behavior is detected, and an alarm is triggered. Local behavioral abrupt change divergence integrates information from three levels: geometric distance, temporal gradient, and topological divergence, forming a complete analytical chain from low-level features to high-level semantics, providing a reliable quantitative indicator for detecting anomalies in diver behavior.

[0068] In one embodiment, when a limb spatial feature is detected to be obscured by a bubble, extracting the normal deformation and tangential displacement of the bubble boundary caused by local water flow displacement includes the following steps: Real-time monitoring of the brightness of pixel regions containing spatial features of limbs, and calculation of local brightness gradients between adjacent video frames in a continuous video stream; When the local brightness gradient exceeds the preset bubble reflection threshold and the limb spatial features are lost, it is determined that the limb spatial features are occluded by bubbles. An edge detection algorithm is used to scan the contour of the feature-lost region and lock the initial set of pixel coordinates of the bubble boundary; Tracing the topological deformation trajectory of the initial set of pixel coordinates in a sequence of consecutively occluded video frames in a continuous video stream; The normal deformation magnitude of the bubble boundary in the normal direction and the displacement in the tangential direction are quantified based on the topological deformation trajectory, so as to obtain the normal deformation and tangential displacement of the bubble boundary caused by local water flow.

[0069] In this embodiment, limb spatial features correspond to specific pixel regions in the image. These regions typically contain the diver's joints, limb contours, and local texture information. Real-time monitoring requires establishing a dynamic mapping relationship between pixel regions and features. The feature map output by the feature extraction network is used to back-locate the pixel coordinates of the original image. Each response position of the feature map corresponds to a receptive field region in the original image. The size of the receptive field is determined by the downsampling factor of the network. For example, in a network with a downsampling factor of 32, one pixel in the feature map corresponds to a 32×32 pixel block in the original image. For detected limb keypoints, a circular region with a radius of 20 pixels around the keypoint is extracted as the monitoring area. The brightness value within the monitoring area is obtained by averaging the grayscale values ​​of all pixels. The average brightness is calculated as follows: ,in Represents the set of pixels in the monitored area. The number of pixels. The local brightness gradient between adjacent video frames reflects the rate of change of brightness over time, and is calculated using the following formula: A positive brightness gradient indicates a brighter area, while a negative value indicates a darker area; the absolute value indicates the degree of change. Underwater bubbles form smooth spherical or ellipsoidal interfaces due to surface tension. These interfaces produce strong specular reflection and refraction of light, resulting in a significantly brighter bubble region than the surrounding water. When a bubble enters the monitoring area, the local brightness increases sharply within a short period, and a positive peak appears in the brightness gradient. The monitoring process uses a sliding window mechanism, calculating the brightness gradient immediately after receiving each new image frame, with a calculation latency controlled within 5 milliseconds. The brightness gradient sequence is stored in a circular buffer, which retains the gradient values ​​of the most recent 30 frames for subsequent time-series analysis and anomaly detection.

[0070] Setting the bubble reflection threshold requires comprehensive consideration of underwater lighting conditions and the physical characteristics of the bubbles. In typical underwater scenarios, the brightness value of the background water is between 50 and 100 (8-bit grayscale range), while the brightness value of the bubble reflection area can reach over 200, with a brightness gradient typically exceeding 80. A preset bubble reflection threshold of 80 is used, determined through statistical analysis of a large number of underwater bubble samples, ensuring detection rate while controlling false alarm rate. When a local brightness gradient exceeds the threshold, further verification is needed to determine if limb spatial features are lost. The loss of limb spatial features is assessed using feature confidence. The feature extraction network outputs a confidence score along with the feature vector, ranging from 0 to 1, with a score above 0.5 indicating reliable features. When the confidence score suddenly drops below 0.3, feature loss is considered. The cause of feature loss may be occlusion, blurring, or removal from the field of view, requiring differentiation based on brightness gradient information. When both conditions are met—brightness gradient exceeding the threshold and feature confidence score below 0.3—the limb spatial features are determined to be occluded by a bubble. The decision-making logic employs a logical AND operation, ensuring that both conditions are met simultaneously before triggering an occlusion event. Occlusion determination also incorporates a time persistence constraint, requiring an abnormal state to persist for more than 3 frames (approximately 100 milliseconds) before occlusion is confirmed, avoiding misjudgments caused by transient noise. Once an occlusion event is confirmed, bubble tracking and trajectory compensation modules are immediately activated, recording the timestamp of the occlusion start and the feature state of the last visible frame.

[0071] Edge detection algorithms are used to identify locations in an image where grayscale or color changes abruptly. Bubble boundaries correspond to transitional regions in the image where brightness decreases from high to low values. The Canny edge detection algorithm, due to its excellent edge localization accuracy and noise resistance, has become the preferred method for bubble boundary detection. The Canny algorithm first applies a Gaussian filter to the feature-loss regions, with a filter standard deviation set to 1.5, which smooths noise while preserving edge information. After filtering, the gradient magnitude and direction of the image are calculated. The gradient is calculated using the Sobel operator, with the horizontal and vertical gradients respectively... and The gradient magnitude is Non-maximum suppression is applied to the gradient magnitude by comparing the gradient values ​​of adjacent pixels along the gradient direction, retaining local maxima and suppressing non-maxima, thus refining the edges to a single pixel width. Dual threshold detection categorizes edge points into strong and weak edges. The threshold for strong edges is set to the 70th percentile of the gradient magnitude distribution, and the threshold for weak edges is set to the 30th percentile. Strong edge points are directly retained, while weak edge points are only retained if they are connected to strong edge points; isolated weak edge points are discarded as noise. Edge connectivity is achieved through connected component analysis, using the 8-connectivity criterion to search for adjacent edge points. The detected edge point set constitutes the initial pixel coordinate set of the bubble boundary, where each element is a two-dimensional coordinate. The initial coordinate set typically contains tens to hundreds of points, distributed in a closed or nearly closed curve shape. The coordinate set is then sorted, and an ordered linked list is created based on the spatial proximity of the points. The order of the linked list reflects the topological continuity of the boundary.

[0072] The topological deformation trajectory describes the geometric evolution of the bubble boundary over time, including multi-dimensional information such as positional movement, shape changes, and scale scaling. In a sequence of continuously occluded video frames, the bubble boundary is affected by water flow, buoyancy, and surface tension, resulting in continuous boundary morphology changes. Tracking the initial set of pixel coordinates requires re-performing edge detection in each frame to obtain the boundary coordinate set of the current frame. The correspondence between the current frame coordinate set and the initial coordinate set is established through shape matching, using an iterative nearest-point method. The iterative nearest-point method first performs coarse alignment on the two coordinate sets, eliminating overall translation through centroid alignment. The centroid is calculated as the arithmetic mean of all coordinate points. After alignment, the nearest distance from each initial point to the current frame point set is calculated, establishing a point-to-point correspondence. Rigid transformation parameters, including rotation angles and translation vectors, are calculated based on the point-to-point correspondence. These transformation parameters are solved by minimizing the sum of squared distances between point pairs. The positions of the initial point set are updated using the transformation parameters, and the nearest-point correspondence is recalculated. This iterative optimization continues until convergence. The converged point pairs determine the inter-frame tracking trajectories of the boundary points, and the position sequence of each initial point in subsequent frames constitutes the spatiotemporal trajectory of that point. The set of trajectories of all boundary points constitutes the topological deformation trajectory of the bubble boundary. The trajectory not only records the positional changes of the points but also implicitly contains information about the evolution of the local curvature and overall shape of the boundary. The extraction of the topological deformation trajectory also needs to handle the appearance and disappearance of boundary points. When the bubble merges or bursts, some boundary points will suddenly disappear or appear.

[0073] The normal deformation of the bubble boundary reflects the shape change of the boundary in the direction perpendicular to the tangent, while the tangential displacement reflects the positional drift of the boundary along the tangent. The normal direction is obtained by rotating the tangent direction of the boundary curve by 90 degrees for each boundary point. The tangent direction is estimated by connecting adjacent points, and the tangent vector is... The normalized unit tangent vector is The normal vector is obtained by rotating the tangent vector counterclockwise by 90 degrees, and is calculated as follows: subscript and Represents vector components. For boundary points in the topological deformation trajectory, calculate the displacement vector between adjacent frames. The projection of the displacement vector onto the normal direction is the normal deformation, calculated as follows: The projection is a scalar; a positive value indicates that the boundary bulges outward, and a negative value indicates that it concaves inward. The projection of the displacement vector onto the tangential direction is the tangential displacement, calculated as... Positive values ​​indicate movement along the positive tangent direction, while negative values ​​indicate movement in the opposite direction. The normal deformation and tangential movement at all boundary points are statistically analyzed, and the mean and standard deviation are calculated. The mean represents the overall deformation and movement trend, while the standard deviation represents local variability. The magnitude of the normal deformation is calculated by averaging the absolute values, reflecting the intensity of the bubble being displaced by the water flow. The directional consistency of the tangential movement is evaluated by the ratio of the magnitude of the vector sum to the scalar sum; a ratio close to 1 indicates that all points move in the same direction, while a ratio close to 0 indicates chaotic movement directions. The normal deformation and tangential movement serve as characteristic parameters of bubble dynamics and are used for subsequent limb movement trajectory inversion, inferring the active movement of the obscured limb from the passive movement of the bubble.

[0074] In one embodiment, tracing the topological deformation trajectory of an initial set of pixel coordinates in a sequence of consecutively occluded video frames of a continuous video stream includes the following steps: Assign an independent Markov tracking node to each boundary pixel in the initial set of pixel coordinates; The state transition probability matrix of the Markov tracking node is defined based on the time interval between adjacent occluded video frames in a continuous video stream. Extract the texture gradient features around the boundary pixels as observation variables and input them into the preset hidden Markov model; By combining the state transition probability matrix with the observed variables, the Viterbi algorithm is used to decode the optimal hidden state sequence of each Markov tracking node; Identify discontinuous jump nodes in the optimal hidden state sequence caused by bubble fusion or rupture; The cubic spline interpolation algorithm is used to smooth out discontinuous jump nodes and generate continuous node displacement curves. The node displacement curves of all boundary pixels are collected to form the topological deformation trajectory of the initial pixel coordinate set.

[0075] In this embodiment, the Markov tracking node is a probabilistic model of the state evolution of bubble boundary pixels in the time series, and each node corresponds to a tracking instance of a boundary pixel. The initial set of pixel coordinates is obtained through the aforementioned edge detection, and the set contains dozens to hundreds of boundary points, each with coordinates of... Each boundary pixel is assigned an independent Markov tracking node, with the node number corresponding one-to-one with the pixel index. Each node is denoted as The state space of a node is defined as the set of possible pixel locations. Considering the limited range of motion of the bubble boundary, the state space is restricted to a circular region with a radius of 50 pixels around the initial position. The state space is discretized into a grid with a grid spacing of 2 pixels, so each node has approximately 2000 states. The initial state of a node is set to the initial pixel coordinates, with an initial state probability of 1, and all other states having a probability of 0. The Markov property is assumed that the node at time t... The state depends only on time. The state of a node is independent of its state at earlier times, an assumption that simplifies the complexity of probabilistic modeling. The independence assumption ensures that state transitions between different nodes are independent. Although adjacent boundary points are spatially related in reality, the independence assumption allows the tracking algorithm to process all nodes in parallel, significantly improving computational efficiency. Each node maintains a state vector, the length of which is equal to the size of the state space, and each element of the vector represents the probability of the node being in the corresponding state.

[0076] The state transition probability matrix describes the probability that a Markov tracking node will transition from one state to another. The rows and columns of the matrix correspond to the initial state and the target state, respectively. Matrix elements Indicates from state Transition to state The probability of all states The sum of the transition probabilities at the start is 1, that is... The transition probability is defined based on the motion characteristics of the bubble boundary. The bubble is affected by buoyancy and water flow, and its motion speed is typically between 5 and 20 pixels per second. The time interval between adjacent occluded video frames is the reciprocal of the frame rate; for a video of 30 frames per second, the time interval is 33 milliseconds. Within a time interval, the displacement distance of the boundary pixels follows a Gaussian distribution. A mean of zero indicates no systematic drift, and the standard deviation is calculated based on the bubble's motion speed and the time interval. The standard deviation is set to... ,in The maximum motion speed is represented by pixels per second. Transition probabilities are modeled using a Gaussian kernel function, starting from the state... to state The transition probability is ,in For state and state Euclidean distance at corresponding positions A normalization constant is used to ensure the probability sum is 1. The Gaussian kernel function ensures a high probability of transitioning to neighboring states and a low probability of transitioning to distant states, consistent with the smooth motion characteristics of the bubble boundary. The transition probability matrix is ​​sparse because each state only has a significant transition probability with its neighboring states; the transition probability of distant states is close to zero and can be ignored. Sparsity is achieved by setting a cutoff radius; distances exceeding this radius are considered less than a certain threshold. The state transition probability is set to zero to reduce storage and computational overhead. The transition probability matrix is ​​shared by all Markov tracking nodes because it is assumed that all boundary pixels have the same motion statistics, and sharing the matrix avoids the redundancy of calculating the transition probability separately for each node.

[0077] Texture gradient features capture local image patterns around boundary pixels, used to distinguish different boundary locations and states. For each boundary pixel location, the surrounding texture gradient is extracted. An image patch of pixels contains 25 grayscale values. The gradient magnitude and direction of the image patch are calculated. The gradient is obtained using the Sobel operator; the horizontal gradient is... The vertical gradient is The gradient magnitude is The gradient direction is The gradient direction is quantized into 8 direction bins, each covering a 45-degree angle range. The gradient magnitude within each direction bin is summed to form an 8-dimensional direction histogram. The direction histogram is normalized to a unit vector to eliminate the influence of illumination intensity. The normalized histogram serves as the texture gradient feature vector. The feature vector has 8 dimensions, with each dimension corresponding to the response intensity of a gradient direction. Texture gradient features are sensitive to the local shape and texture pattern of the boundary. Boundaries at different locations have different feature distributions, and the differences in features are used to construct the observation model. The observation variable is defined as the similarity between the texture gradient features of the candidate state position in the current frame and the reference features of the initial frame. The similarity is measured using cosine distance and calculated as the inner product of the feature vectors. The observation variable is input into the observation model of the Hidden Markov Model, which describes the probability of observing a specific feature in a given hidden state. The observation probability is modeled using a Gaussian likelihood function, where the mean of the likelihood function is the reference feature and the standard deviation is the feature noise level. A high observation probability indicates that the features at the candidate state position match the reference features well, and the state is more likely to be the true state.

[0078] The Viterbi algorithm is a dynamic programming algorithm for finding the optimal hidden state sequence in a Hidden Markov Model (HMM). It finds the state path that maximizes the probability of the observation sequence through recursive calculation. The algorithm's input includes the state transition probability matrix, the observation probability sequence, and the initial state probability. The output is the optimal state at each time step. The algorithm maintains two dynamic programming tables: the first records the maximum path probability to each state, and the second records the predecessor states that achieve the maximum probability. In the initialization phase, the path probability of the first frame is set to the product of the initial state probability and the observation probability. In the recursive phase, for each time step... Each state Calculate all possible predecessor states The path probability is the product of the path probability and the transition probability. The predecessor state with the largest product is selected, and the path probability is updated to... ,in For a moment state The maximum path probability, The probability of observation is given. Simultaneously, the index of the preceding state that maximizes the path probability is recorded. The recursive process continues from the second frame to the last frame, with the calculation in each frame depending on the result of the previous frame. In the backtracking phase, starting from the last frame, the state with the highest path probability is selected as the optimal termination state. The process is then reversed according to the preceding state table to determine the optimal state at each time step, forming a complete sequence of optimal hidden states. The optimal state sequence corresponds to the most probable position of the boundary pixel at each time step, and the sequence comprehensively considers both motion smoothness and observation consistency constraints. The time complexity of the Viterbi algorithm is O(log n). ,in For time frames, Given the number of states, the sparsity of the transition probability matrix can reduce the complexity to [missing value]. ,in The number of valid predecessors for each state.

[0079] While the optimal hidden state sequence combines motion smoothness and observation consistency, discontinuous jumps may occur during topological events such as bubble merging or bursting. Bubble merging refers to two or more bubbles merging into a larger bubble; boundary pixels may suddenly disappear or jump to a new position at the moment of merging. Bubble bursting refers to the bubble breaking due to surface tension imbalance; boundary pixels lose tracking at the moment of bursting. Discontinuous jump nodes are characterized by abnormally large displacement distances between adjacent frames, exceeding the reasonable range of normal motion speed. Jump nodes are identified by calculating the displacement distance between adjacent states; the distance calculation is as follows: ,in For a moment The optimal position coordinates. Set the jump threshold to... The threshold value is five times the standard deviation of the normal displacement. When the displacement distance exceeds this threshold, it is considered a skipped node. Skip node identification also considers abrupt changes in observation probability. When the observation probability suddenly drops from a high value to near zero, it indicates that the boundary pixel has lost reliable observation support, possibly indicating a topological event. After marking a skipped node, the type of skip needs to be determined. A persistent skip indicates that the boundary pixel has permanently disappeared, while a transient skip indicates a brief recovery after a tracking failure. Persistent skips are determined by examining the observation probability sequence after the skip. If the observation probability in subsequent frames remains below the threshold, it is a persistent skip, and tracking of that boundary pixel terminates. A transient skip indicates that the observation probability recovers after the skip, meaning the boundary pixel has reappeared. Interpolation is needed to repair the trajectory during the skip.

[0080] Cubic spline interpolation is a numerical method for constructing smooth curves. It fits data points using piecewise cubic polynomials, ensuring the continuity of the first and second derivatives of the curve at connection points. For identified discontinuous jump nodes, the interpolation algorithm constructs a smooth transition between normal nodes before and after the jump. Assume the jump occurs at time t. arrive Between, the last normal node before the jump is The first normal node after the jump is Cubic spline interpolation was performed on... coordinates and Coordinates are processed independently, for coordinates, constructing a cubic polynomial The polynomial coefficients are determined by boundary conditions. Boundary conditions include endpoint position constraints. and And endpoint velocity constraints, where velocity is estimated using the displacements of adjacent normal nodes. The endpoint velocity is calculated as follows: and ,in and The coordinates of adjacent nodes before and after the jump are given. The four boundary conditions form a system of four linear equations, which are solved to obtain the polynomial coefficients. For each time point during the jump... Interpolated coordinates are calculated using polynomials. The same process is applied to Coordinates. The interpolated coordinate sequence replaces the abnormal coordinates of the jump nodes, allowing the trajectory to transition smoothly within the jump interval. Cubic spline interpolation ensures the continuity of position, velocity, and acceleration, avoiding abrupt velocity changes caused by linear interpolation and abrupt acceleration changes caused by quadratic interpolation.

[0081] Node displacement curves record the positional changes of individual boundary pixels in a time series, with the horizontal axis representing time and the vertical axis representing pixel coordinates. Each Markov tracking node, after Viterbi decoding and spline interpolation, generates a complete displacement curve containing position estimates for all moments from the initial frame to the last occluded frame. The displacement curves not only contain positional information but also allow for the calculation of velocity and acceleration through numerical differentiation; velocity is the first derivative of position with respect to time, and acceleration is the second derivative. Collecting the node displacement curves of all boundary pixels forms a comprehensive description of the boundary's motion. This collection process organizes multiple independent curves into a unified data structure. The data structure uses a two-dimensional array, with the first dimension indexing the boundary pixels and the second dimension indexing the time frames. Array elements represent the coordinates of the corresponding pixels at their respective moments. The collected set of displacement curves constitutes the topological deformation trajectory of the initial set of pixel coordinates. This trajectory fully describes the geometric evolution of the bubble boundary during occlusion. The topological deformation trajectory can be visualized as a dynamic curve graph, where each curve represents the motion path of a boundary pixel, and the density of the curves reflects the severity of the boundary deformation. The trajectory can also be converted into a time series of boundary shapes. At each moment, the coordinates of all boundary pixels are extracted and connected to form a closed curve. The time series shows the continuous change of the boundary shape.

[0082] In one implementation, the limb motion velocity vector field of the last frame before being occluded by the bubble is extracted. The limb motion velocity vector field is then cross-correlated with the normal deformation and tangential displacement across frames to generate an implicit motion trajectory equation. A compensation feature vector is constructed using the implicit motion trajectory equation and input into a preset temporal attention mechanism network to generate an aligned spatiotemporal feature matrix. The steps include the following: The spatial features of the limbs within the time window before being occluded by the bubble are cached, and the limb motion velocity vector field of the last frame before being occluded by the bubble is calculated and extracted. The limb motion velocity vector field, normal deformation, and tangential displacement are mapped to a unified spatiotemporal reference frame. Construct a cross-frame cross-correlation function in a spatiotemporal reference frame and determine the peak point of the response of the cross-frame cross-correlation function; Based on the response peak point, the three-dimensional motion coordinates of the limbs hidden behind the bubble are deduced in reverse, and the implicit motion trajectory equation is generated by fitting the three-dimensional motion coordinates of the limbs. Discretize and sample the implicit motion trajectory equation to construct a compensation feature vector; After replacing the missing features in the limb spatial features with the compensation feature vector, the input is fed into the preset temporal attention mechanism network to generate the aligned spatiotemporal feature matrix.

[0083] In this implementation, a sliding window strategy is used for caching, with a window length of 0.5 seconds corresponding to 15 frames of video data. The window stores the extracted limb spatial feature vector for each frame, with a dimension of 512, containing the diver's limb position, posture, and local texture information. The cache data structure uses a circular queue with a fixed capacity of 15 elements. When new frame features enter the queue, the oldest features are automatically dequeued, maintaining a constant window time span. When a bubble occlusion event is detected, the cache content is immediately frozen, update operations are stopped, and the complete feature sequence before occlusion is preserved. The limb motion velocity vector field describes the instantaneous motion state of each part of the limb in space, and the velocity vector field is calculated using a time difference method. The limb keypoint position in the last frame before occlusion is denoted as... ,in This refers to the number of keypoints, typically including 17 keypoints for the head, shoulders, elbows, wrists, hips, knees, and ankles. The velocity vector of each keypoint is calculated by the difference between its position in the current frame and the previous frame, using the following formula: ,in This is the inter-frame time interval.

[0084] The limb motion velocity vector field is defined in a two-dimensional pixel coordinate system on the image plane, with the velocity unit being pixels per second. The normal deformation and tangential displacement of the bubble are also defined in pixel coordinates, but their physical meanings differ: the normal deformation represents the shape change of the boundary, and the tangential displacement represents the positional drift of the boundary. The mapping process first unifies the spatial scale, converting all motion quantities into normalized coordinates relative to the image center. The normalized coordinates range from -1 to 1, eliminating the influence of image resolution. The temporal scale is aligned using unified timestamps; the time corresponding to all motion quantities is marked as a time offset relative to the start of occlusion, in seconds. The spatiotemporal reference frame uses a three-dimensional coordinate system, with the first two dimensions being spatial coordinates. The third dimension is the time coordinate. The limb velocity vector field is mapped to a vector field in the spacetime reference frame. Each spatial location and time corresponds to a velocity vector. The normal deformation and tangential displacement are mapped to a scalar field. and The scalar field value represents the deformation and displacement of the bubble boundary at the corresponding position and time. The mapping also needs to consider the rotation and translation of the coordinate system, achieving the transformation between different local coordinate systems through an affine transformation matrix. The affine transformation parameters are solved using the least squares method by minimizing the corresponding error estimates of the feature points before and after the mapping.

[0085] In a spatiotemporal reference frame, the limb motion velocity vector field and the bubble shape variable constitute two spatiotemporal signals. The cross-correlation function calculates the correlation coefficient between the two signals under different time delays. The cross-correlation function is defined as follows: ,in For time delay, and These are the normal and tangential unit vectors, respectively. Integration is performed in the spatial and temporal domains; the spatial domain covers the overlapping region of the limb and the bubble, and the temporal domain covers the time window before occlusion. The physical meaning of the cross-correlation function is the energy coupling strength between limb movement and bubble deformation. When the limb pushes the water, causing the bubble to deform, there is a causal relationship between the two, and the cross-correlation function peaks at the corresponding time delay. The response peak point is determined by finding the local maximum of the cross-correlation function, and the time delay corresponding to the peak is... This represents the propagation time of the effect of limb movement on bubble deformation. The propagation time reflects the dynamic characteristics of the fluid medium; the viscosity and density of water determine the speed of disturbance propagation. The amplitude of the peak value represents the coupling strength; a larger amplitude indicates a more significant influence of limb movement on bubble deformation. The calculation of the cross-correlation function is accelerated using a Fast Fourier Transform, converting time-domain convolution into frequency-domain multiplication, thus reducing computational complexity from... Reduce to ,in This is the signal length.

[0086] When a limb moves in water, it pushes against the surrounding fluid, which exerts pressure and shear force on the bubble, causing deformation and displacement at the bubble boundary. By observing the bubble's deformation and displacement, and combining this with a fluid dynamics model, the force applied to the bubble can be calculated in reverse, thus inferring the limb movement that generates this force. The reverse inference first establishes a model of the relationship between force and deformation. According to Stokes' law of drag, the drag on a spherical bubble is proportional to its relative velocity, with the proportionality coefficient related to the bubble radius and fluid viscosity. The normal deformation of the bubble is related to the normal pressure gradient, and the tangential displacement is related to the tangential velocity. Through least-squares fitting, the magnitude and direction of the force are inferred from the observed deformation and displacement. The spatial distribution of the force reflects the limb's position and motion state; the limb position corresponds to the source point of the force, and the limb velocity corresponds to the time rate of change of the force. The resulting three-dimensional motion coordinate sequence of the limb is denoted as... ,in The number of frames during the occlusion period. The coordinate sequence is fitted using a polynomial to generate an implicit motion trajectory equation. The fitting uses a cubic polynomial, and the equation has the following form: ,in These are the coefficients of a three-dimensional vector. The polynomial coefficients are solved by minimizing the fitting error, which is defined as the sum of squared Euclidean distances between the fitted curve and the observed coordinates.

[0087] The implicit motion trajectory equation is a continuous parameterized curve, which needs to be discretized and sampled to be converted into a feature vector that can be processed by a neural network. Discretization sampling uniformly selects sampling points on the time axis, with the sampling interval consistent with the video frame rate, ensuring that the sampling points are aligned with the video frames. For each moment during the occlusion period... The three-dimensional coordinates of the limbs are calculated using trajectory equations. The coordinates consist of three components: horizontal, vertical, and depth. The depth component is obtained through a monocular vision depth estimation network. Depth estimation is based on perspective projection and scale prior, using known limb dimensions as a reference. In addition to the position coordinates, the first and second derivatives of the trajectory are calculated, representing velocity and acceleration respectively. The derivatives are obtained through analytical differentiation, with the formula: and Position, velocity, and acceleration together constitute 9-dimensional kinematic features. These 9-dimensional features are mapped to 512-dimensional compensation feature vectors through a fully connected layer. The weights of the mapping layer are learned through pre-training, so that the compensation features are aligned with the normal limb spatial features in the semantic space.

[0088] The replacement operation of the compensating feature vector fills in the missing positions of the feature sequence with the inferred limb motion information, restoring the integrity of the feature sequence. The original limb spatial feature sequence contains gaps during occlusion; the feature vectors corresponding to the gaps are marked as invalid or filled with zero vectors. The replacement process traverses the feature sequence, identifies the position index of invalid features, and copies the corresponding compensating feature vector to that position. The replaced feature sequence is continuous and complete in the temporal dimension, with no missing frames, and each time step has a valid feature representation. The sequence length is typically set to 64 frames to cover the complete motion cycle. The complete feature sequence is input into a temporal attention mechanism network, which employs a multi-layer Transformer encoder architecture, as described above. Figure 3 It contains six stacked encoder layers. Each encoder layer consists of a multi-head self-attention sublayer and a feedforward neural network sublayer, with residual connections and layer normalization operations added between the two sublayers. The input feature sequence first passes through a position encoding module, which generates the position using sine and cosine functions, adding unique positional information to each time position, enabling the network to perceive the temporal order of the features. The position encoded vector is then added element-wise to the feature vector before entering the first encoder layer. In the multi-head self-attention sublayer, the input features undergo three linear transformations to generate a query matrix, a key matrix, and a value matrix, each with a dimension of 1. The multi-head mechanism divides the feature dimension into 8 subspaces, each independently calculating attention; the subspace dimension is 64. Each attention head calculates the dot product similarity between the query and the key. The similarity matrix is ​​scaled and softmax normalized to obtain the attention weight matrix. The weight matrix is ​​multiplied by the value matrix to obtain the weighted features. The outputs of the 8 attention heads are concatenated and fused through a linear layer. The fused result is added to the input features to form a residual connection, which is then normalized to stabilize the numerical distribution. The feedforward neural network sublayer contains two fully connected layers. The first layer expands the feature dimension to 2048 and applies the ReLU activation function, while the second layer compresses the dimension back to 512. The output of the feedforward network also undergoes residual connections and layer normalization. The feature sequence is processed sequentially through 6 encoder layers, each layer progressively refining the temporal dependencies and semantic representations. The final encoder layer output is the aligned spatiotemporal feature matrix, maintaining the same matrix dimension. The matrix maintains consistency over time.

[0089] The present invention also discloses a diver behavior anomaly detection system based on spatiotemporal attention mechanism, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the diver behavior anomaly detection method based on spatiotemporal attention mechanism as described above.

[0090] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0091] The memory can be an internal storage unit of a computer device, such as a hard disk or RAM, or an external storage device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) provided on the computer device. Furthermore, the memory can be a combination of internal storage units and external storage devices of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0092] The present invention also discloses a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to be configured to perform the diver behavior anomaly detection method based on spatiotemporal attention mechanism described in any of the above embodiments.

[0093] The computer program can be stored in a machine-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The machine-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the machine-readable medium includes, but is not limited to, the above-mentioned components.

[0094] The above-described diver behavior anomaly detection method based on spatiotemporal attention mechanism is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the above method.

[0095] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0096] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A method for detecting anomalies in diver behavior based on a spatiotemporal attention mechanism, characterized in that, Includes the following steps: A continuous video stream of the underwater working environment of divers is collected, and the dynamic displacement of suspended particles in the continuous video stream is tracked by high-pass filtering and optical flow method. A two-dimensional vector field of underwater local optical distortion is constructed based on the dynamic displacement. In the pre-defined feature extraction network with spatial attention mechanism, the inverse of the magnitude of the deformation vector in the two-dimensional vector field is used as a regularization term. At the same time, the direction of the deformation vector in the two-dimensional vector field is used to guide the receptive field of the feature extraction network to shift, thereby extracting the spatial features of the diver's limbs from the continuous video stream. When the spatial features of the limbs are detected to be obscured by bubbles, the normal deformation and tangential movement of the bubble boundary caused by the local water flow displacement are extracted. Extract the limb motion velocity vector field of the last frame before being occluded by the bubble, perform cross-frame cross-correlation calculation between the limb motion velocity vector field and the normal deformation and tangential movement, invert to generate implicit motion trajectory equation, and use the implicit motion trajectory equation to construct a compensation feature vector and input it into the preset time attention mechanism network to generate an aligned spatiotemporal feature matrix. The spatiotemporal feature matrix is ​​mapped to a spatiotemporal action topology sequence with a time series distribution, and the local behavioral mutation divergence of the spatiotemporal action topology sequence is obtained by calculating the divergence value of the gradient field of the spatiotemporal feature matrix between adjacent time frames. When the local behavioral abrupt change divergence is detected to exceed the preset stable range within a preset time window, it is determined that the target diver has engaged in abnormal behavior and an alarm signal is output.

2. The method for detecting abnormal diver behavior based on spatiotemporal attention mechanism according to claim 1, characterized in that, The process of acquiring a continuous video stream of the underwater working environment of divers, tracking the dynamic displacement of suspended particles in the continuous video stream using high-pass filtering and optical flow, and constructing a two-dimensional vector field of underwater local optical distortion based on the dynamic displacement includes the following steps: Acquire continuous video streams of the underwater working environment of divers and convert the continuous video streams into a sequence of grayscale images; A high-pass filtering algorithm was used to remove the background from a grayscale image sequence, separating the underwater suspended particle set. The optical flow method is used to track the two-dimensional displacement coordinates of each suspended particle in an array of suspended particles in adjacent grayscale image sequences; The pixel space corresponding to the grayscale image sequence is divided into multiple local grid regions; The average displacement coordinates of suspended particles in each local grid region are calculated based on two-dimensional displacement coordinates, and a local grid deformation vector is generated. The local mesh deformation vectors of all local mesh regions are spliced ​​and combined to form a two-dimensional vector field of underwater local optical distortion.

3. The method for detecting abnormal diver behavior based on spatiotemporal attention mechanism according to claim 2, characterized in that, The method of tracking the two-dimensional displacement coordinates of each suspended particle in an array of suspended particles in adjacent grayscale image sequences using optical flow includes the following steps: Assign a unique tracking identifier to each suspended particle in the set of suspended particles in the grayscale image sequence of the current frame; An optical flow evaluation model based on an image pyramid is constructed by inputting the current frame grayscale image sequence containing tracking identifiers and the adjacent next frame grayscale image sequence into the optical flow evaluation model. The initial coarse displacement of each suspended particle is calculated in the top-level image of the optical flow evaluation model; The initial coarse displacement is projected as a priori parameter into the underlying image of the optical flow evaluation model; By combining nonlocal spatial smoothing constraints, the initial coarse displacement in the underlying image is iteratively optimized to eliminate the interference of underwater high-frequency noise on the tracking trajectory and obtain the target displacement vector of each suspended particle. The two-dimensional displacement coordinates of each suspended particle in adjacent grayscale image sequences are determined based on the target displacement vector.

4. The method for detecting abnormal diver behavior based on spatiotemporal attention mechanism according to claim 3, characterized in that, The iterative optimization of the initial coarse displacement in the underlying high-resolution image by combining nonlocal spatial smoothing constraints to eliminate the interference of underwater high-frequency noise on the tracking trajectory, and obtaining the target displacement vector of each suspended particle, includes the following steps: A non-local neighborhood window is constructed with each suspended particle in the underlying image as the center. Calculate the gray-level similarity weight and spatial distance weight of adjacent pixels within a non-local neighborhood window; The gray-level similarity weight and spatial distance weight are combined to generate a nonlocal spatial smoothing constraint term. A global energy functional optimization objective is constructed by combining nonlocal spatial smoothing constraints with optical flow data terms. The global energy functional optimization objective is solved iteratively using the conjugate gradient descent method. When the convergence condition is met, the target displacement vector of each suspended particle is output.

5. The method for detecting abnormal diver behavior based on spatiotemporal attention mechanism according to claim 1, characterized in that, In the pre-defined feature extraction network with a spatial attention mechanism, the reciprocal of the magnitude of the deformation vector in the two-dimensional vector field is used as a regularization term. Simultaneously, the direction of the deformation vector in the two-dimensional vector field is used to guide the receptive field of the feature extraction network to shift. Extracting the spatial features of the diver's limbs from a continuous video stream includes the following steps: Continuous video frames are input into a pre-defined feature extraction network with a spatial attention mechanism to generate an initial feature map; Extract the magnitude and direction of the deformation vector for each local grid region from the two-dimensional vector field; Calculate the reciprocal of the magnitude of the deformation vector and map the reciprocal of the magnitude of the deformation vector to a regularization penalty mask; The regularization penalty mask is multiplied element-wise with the initial attention weight map of the feature extraction network to suppress the attention weights in high distortion regions. The deformation vector direction is converted into a two-dimensional offset field, and the two-dimensional offset field is injected into the deformable convolutional layer of the feature extraction network. Deformable convolutional layers are used, and the receptive field of the convolutional kernel is guided to shift in the opposite direction of the water flow distortion according to a two-dimensional offset field. Based on the offset receptive field and the suppressed attention weight, the initial feature map is resampled and fused to extract the spatial features of the diver's limbs after eliminating optical artifacts.

6. The method for detecting abnormal diver behavior based on spatiotemporal attention mechanism according to claim 1, characterized in that, When a limb spatial feature is detected to be obscured by a bubble, the extraction of the normal deformation and tangential movement of the bubble boundary caused by local water flow displacement includes the following steps: Real-time monitoring of the brightness of pixel regions containing spatial features of limbs, and calculation of local brightness gradients between adjacent video frames in a continuous video stream; When the local brightness gradient exceeds the preset bubble reflection threshold and the limb spatial features are lost, it is determined that the limb spatial features are occluded by bubbles. An edge detection algorithm is used to scan the contour of the feature-lost region and lock the initial set of pixel coordinates of the bubble boundary; Tracing the topological deformation trajectory of the initial set of pixel coordinates in a sequence of consecutively occluded video frames in a continuous video stream; The normal deformation magnitude of the bubble boundary in the normal direction and the displacement in the tangential direction are quantified based on the topological deformation trajectory, so as to obtain the normal deformation and tangential displacement of the bubble boundary caused by local water flow.

7. The method for detecting anomalies in diver behavior based on a spatiotemporal attention mechanism according to claim 6, characterized in that, The process of tracing the topological deformation trajectory of the initial set of pixel coordinates in a sequence of continuously occluded video frames in a continuous video stream includes the following steps: Assign an independent Markov tracking node to each boundary pixel in the initial set of pixel coordinates; The state transition probability matrix of the Markov tracking node is defined based on the time interval between adjacent occluded video frames in a continuous video stream. Extract the texture gradient features around the boundary pixels as observation variables and input them into the preset hidden Markov model; By combining the state transition probability matrix with the observed variables, the Viterbi algorithm is used to decode the optimal hidden state sequence of each Markov tracking node; Identify discontinuous jump nodes in the optimal hidden state sequence caused by bubble fusion or rupture; The cubic spline interpolation algorithm is used to smooth out discontinuous jump nodes and generate continuous node displacement curves. The node displacement curves of all boundary pixels are collected to form the topological deformation trajectory of the initial pixel coordinate set.

8. The method for detecting abnormal diver behavior based on spatiotemporal attention mechanism according to claim 1, characterized in that, The steps of extracting the limb motion velocity vector field from the last frame before the occlusion by the bubble, performing cross-frame cross-correlation calculations on the limb motion velocity vector field with the normal deformation and tangential displacement, generating an implicit motion trajectory equation, and constructing a compensation feature vector using the implicit motion trajectory equation and inputting it into a preset temporal attention mechanism network to generate an aligned spatiotemporal feature matrix include the following steps: The spatial features of the limbs within the time window before being occluded by the bubble are cached, and the limb motion velocity vector field of the last frame before being occluded by the bubble is calculated and extracted. The limb motion velocity vector field, normal deformation, and tangential displacement are mapped to a unified spatiotemporal reference frame. Construct a cross-frame cross-correlation function in a spatiotemporal reference frame and determine the peak point of the response of the cross-frame cross-correlation function; Based on the response peak point, the three-dimensional motion coordinates of the limbs hidden behind the bubble are deduced in reverse, and the implicit motion trajectory equation is generated by fitting the three-dimensional motion coordinates of the limbs. Discretize and sample the implicit motion trajectory equation to construct a compensation feature vector; After replacing the missing features in the limb spatial features with the compensation feature vector, the input is fed into the preset temporal attention mechanism network to generate the aligned spatiotemporal feature matrix.

9. A diver behavior anomaly detection system based on a spatiotemporal attention mechanism, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the diver behavior anomaly detection method based on spatiotemporal attention mechanism as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When executed by the processor, the instruction causes the processor to be configured to perform the diver behavior anomaly detection method based on spatiotemporal attention mechanism according to any one of claims 1 to 8.

Images