A human abnormal posture detection method and system based on a multi-view fusion network

Abstract

The application provides a human abnormal posture detection method and system based on a multi-view fusion network, and the method comprises the following steps: acquiring non-blind area video data through a multi-view synchronous acquisition system; sampling each view video, constructing a human joint point space-time trajectory through posture estimation and key point tracking; extracting visual features of the sampling frame and sampling according to the trajectory coordinates to form a trajectory visual feature sequence; using a sequence modeling network based on a selective state space to associate and enhance the trajectory features in space-time semantics; fusing multi-view trajectory features through a cross-view attention mechanism to generate unified multi-view fusion features; and finally realizing binary classification detection of abnormal postures through a classification network. Through multi-view cooperation, Mamba space-time modeling and attention feature fusion, the application improves the accuracy and robustness of human abnormal posture detection in complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video processing technology, and in particular to a method and system for detecting abnormal human poses based on a multi-view fusion network. Background Technology

[0002] Human anomaly pose detection is a key application of computer vision in fields such as intelligent security and public safety management. It aims to automatically identify postures and movements that deviate from normal behavioral patterns by analyzing video sequences. Currently, mainstream technologies in this field primarily rely on visual data acquired by a single camera, which faces several inherent limitations in practical deployments.

[0003] At the data perception level, a single fixed viewpoint is insufficient to cover the entire area of ​​complex monitoring scenarios. Human targets are prone to losing crucial pose information due to self-occlusion, environmental obstruction, or movement outside the shooting range, leading to missed detections or false positives in the detection system. At the information processing level, existing methods typically employ recurrent neural networks or Transformer architectures to perform temporal modeling of extracted human pose features. However, the former suffers from insufficient long-range dependency modeling capabilities, while the latter is accompanied by high computational complexity. Neither approach can achieve a balance between efficiency and accuracy in accurately depicting complex human motion patterns in long video sequences. Furthermore, to overcome the limitations of single-viewpoint systems, some studies have introduced multi-camera systems. However, most related multi-view information fusion methods remain at a superficial level, such as feature stitching or average pooling, failing to establish deep spatiotemporal correlations between human motion trajectories from different viewpoints. They also lack the ability to adaptively filter and weight key viewpoint information based on task importance, thus failing to fully realize the potential of multi-viewpoint systems.

[0004] In summary, existing technologies have significant shortcomings in addressing complex real-world scenarios, particularly in terms of the completeness of field of view coverage, the effectiveness of long-sequence motion modeling, and the intelligence of multi-source information fusion. Developing a detection scheme that can comprehensively solve these problems is of great significance for promoting the practical application and high performance of human abnormal posture detection technology. Summary of the Invention

[0005] In view of the above, the main objective of this invention is to propose a method and system for detecting abnormal human postures based on a multi-view fusion network, so as to solve the above-mentioned technical problems.

[0006] This invention proposes a method for detecting abnormal human poses based on a multi-view fusion network, the method comprising the following steps:

[0007] Step 1: Use a preset multi-view camera to capture a multi-view video set of human poses; use a pose estimation algorithm to locate the human pose key points in the multi-view video set of human poses, and use a key point tracking algorithm to associate the same key point across frames to construct a set of human pose key point trajectory segments from each viewpoint.

[0008] Step 2: Based on the set of trajectory segments of human posture key points from various perspectives, use the bilinear interpolation algorithm to perform feature sampling to obtain the visual feature sequence of the trajectory.

[0009] Step 3: Input the visual feature sequence of the trajectory into a sequence modeling network based on a selective state space model for processing, and capture the spatial semantic association between trajectories to obtain the enhanced hidden state sequence of the trajectory.

[0010] Step 4: Based on the cross-view attention mechanism, the enhanced hidden state sequence of the trajectory is fused to obtain unified multi-view fusion features;

[0011] Step 5: Input the unified multi-view fusion features into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal.

[0012] This invention also proposes a human abnormal pose detection system based on a multi-view fusion network, wherein the system employs the human abnormal pose detection method based on a multi-view fusion network as described above, and the system includes:

[0013] The multi-view trajectory perception and feature encoding module is used for:

[0014] A multi-view video set of human poses is obtained by using a pre-set multi-view camera; the human pose key points in the video set of the multi-view human poses are located by using a pose estimation algorithm; and the same key point is associated across frames by using a key point tracking algorithm to construct a set of human pose key point trajectory segments from each viewpoint.

[0015] Based on the set of trajectory fragments of human posture key points from various perspectives, feature sampling is performed using a bilinear interpolation algorithm to obtain the visual feature sequence of the trajectory.

[0016] The spatiotemporal modeling and multi-view fusion module is used for:

[0017] The visual feature sequence of the trajectory is input into a sequence modeling network based on a selective state space model for processing, and the spatial semantic association between trajectories is captured to obtain the enhanced hidden state sequence of the trajectory.

[0018] The enhanced hidden state sequence of the trajectory is fused based on the cross-view attention mechanism to obtain unified multi-view fusion features;

[0019] The abnormal pose classification decision module is used for:

[0020] The unified multi-view fusion features are input into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] 1. This invention solves the problem of blind spots in single-view vision, achieving all-round coverage perception of human posture. By deploying a multi-view synchronous acquisition system and adopting specific deployment angles and overlapping coverage strategies, it ensures that human posture is acquired without blind spots within the monitoring area, effectively avoiding the loss of posture information caused by occlusion, target out of frame, etc., and providing a comprehensive data foundation for accurate detection;

[0023] 2. It achieves efficient modeling of long-range spatiotemporal dynamics of complex postures. It innovatively introduces a sequence modeling network based on selective state space (Mamba), whose linear complexity and global receptive field characteristics can efficiently process long sequence trajectory features, accurately capture the spatiotemporal evolution law and motion pattern of human key points, and improve the ability to represent complex abnormal posture dynamic processes;

[0024] 3. Intelligent deep fusion of multi-view information was achieved. A cross-view fusion module based on an attention mechanism was designed, which can adaptively learn the association weights of trajectory features from each viewpoint, highlight the viewpoint information that contributes the most to anomaly detection, suppress redundant interference, and thus generate a more discriminative unified pose representation, significantly improving the accuracy of detection.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0026] Figure 1 The present invention proposes a step flowchart for a method for detecting abnormal human postures based on a multi-view fusion network;

[0027] Figure 2 This is a system structure diagram of a human abnormal posture detection system based on a multi-view fusion network proposed in this invention. Detailed Implementation

[0028] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0029] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0030] Please see Figure 1 This embodiment provides a method for detecting abnormal human poses based on a multi-view fusion network. The method includes the following steps:

[0031] Step 1: Use a preset multi-view camera to capture a set of multi-view videos of human posture; use a posture estimation algorithm to locate the key points of human posture in the multi-view video set, and use a key point tracking algorithm to associate the same key point across frames to obtain a set of spatiotemporal trajectory segments of human multi-joint points under each view.

[0032] In step 1, a multi-view video set of human posture is obtained by using a preset multi-view camera; the pose estimation algorithm is used to locate the key points of human posture in the video, and the key point tracking algorithm is used to associate the same key point across frames to obtain a set of spatiotemporal trajectory segments of human multi-joint points from each viewpoint. Specifically, it includes the following sub-steps:

[0033] A collection of multi-view videos of human postures captured using a pre-set multi-view camera. ;

[0034] in, A collection of multi-view videos representing human postures. This represents the video sequence from the first perspective. This represents a video sequence from a second perspective. Indicates the first A video sequence from multiple perspectives, This represents the video sequence from the last perspective. Index representing the viewpoint. Indicates the total number of viewpoints, and ;

[0035] Based on a multi-view video collection of human poses, the first Video sequences from various perspectives Sampling was performed to obtain the first A collection of sampled frames from various perspectives And select the first sampling frame. As the starting frame for trajectory tracking;

[0036] in, Indicates the first A set of sampled frames from various perspectives Indicates the first The first sampled frame from each viewpoint Indicates the first The second sampling frame from each viewpoint Indicates the first The first perspective Each sampling frame, Indicates the index of the sampled frame. Indicates the first The first perspective Each sampling frame, Indicates the total number of sampled frames;

[0037] The YOLO-Pose pose estimation algorithm is used to analyze and process the starting frame, and human pose keypoints with confidence scores lower than a preset confidence threshold are filtered out to obtain the set of valid human pose keypoints in the starting frame. The following relationship exists in the corresponding process:

[0038] ;

[0039] in, Indicates the first A set of effective human pose key points for each viewpoint starting frame. , and Both represent sets The elements in Indicates the first The first frame of the viewpoint Key points for effective human posture An index representing the key points of an effective human pose. This indicates the total number of valid human pose keypoints in the starting frame. Indicating key points of human posture The horizontal pixel coordinates in the starting frame Indicating key points of human posture Vertical pixel coordinates in the starting frame Indicating key points of human posture The confidence score, Indicating key points of human posture Type labels (such as "top of head", "right knee", etc.);

[0040] It should be noted that the key points of human posture include: top of the head, neck, shoulders, elbows, wrists, hips, knees, ankles, and nose tip; the preset confidence threshold is set to 0.7 in this embodiment.

[0041] Using key point tracking algorithm for the first The set of sampled frames from the first perspective and the first The effective human pose keypoint set of the first viewpoint starting frame is processed to obtain the first viewpoint. The set of trajectory segments of key points of human posture from various perspectives, and the following relationship exists in the correspondence process:

[0042] ;

[0043] in, Indicates the first A collection of trajectory segments of key points in human posture from multiple perspectives. This indicates that the data has been processed using the PIP3 keypoint tracking model.

[0044] It should be noted that the PIP3 keypoint tracking model utilizes ensembles. The key information of human posture in the set Subsequent sampling frames ( to Searching for and matching key points of the same human posture in the model to achieve trajectory continuity, therefore, the first... A collection of human posture key point trajectory segments from multiple perspectives The number of trajectories in the first and second trajectories Set of effective human pose key points in the starting frame of each viewpoint The number of effective human body key points is the same.

[0045] Furthermore, the process of obtaining a multi-view video set of human postures using preset multi-view cameras specifically includes: employing a distributed dual-view or triple-view camera deployment scheme, with 2-3 cameras, and controlling the horizontal viewing angle between adjacent cameras to be 45-60° to ensure no blind spots in the shooting area and at least dual-view overlap coverage of key areas (such as passageways and rest areas); uniformly adjusting the vertical viewing angle to 1.0m-2.5m above the ground to avoid obstruction by ground debris and deviation from the high-altitude viewing angle; configuring all cameras with the same parameters: sampling frame rate of 25-30fps, video resolution of 1920×1080 pixels, and image format of RGB; achieving strict timing alignment of multi-view video frames through the NTP timestamp synchronization protocol, with the error controlled within 10ms; and finally obtaining a multi-view video set of human postures.

[0046] Step 2: Based on the set of trajectory segments of human body pose key points from various perspectives, use bilinear interpolation algorithm to perform feature sampling to obtain the visual feature sequence of the trajectory.

[0047] In step 2, based on the set of trajectory segments of human pose key points from various viewpoints, feature sampling is performed using a bilinear interpolation algorithm to obtain the visual feature sequence of the trajectory. This specifically includes the following sub-steps:

[0048] The first The first perspective Sample frames The input is fed into a pre-trained visual encoding model for feature extraction to obtain the first... The first perspective The deep visual feature maps of each sampled frame have the following relationship in the correspondence process:

[0049] ;

[0050] in, Indicates the first The first perspective Deep visual feature maps of each sampled frame, This indicates that the code has undergone forward encoding processing using the SigLIP model.

[0051] Based on sets The Middle The trajectory in the first The two-dimensional spatial coordinates on each sample frame are used to extract the deep visual feature map using a bilinear interpolation algorithm. Feature sampling is performed to obtain the first... The trajectory in the first The visual feature vectors on each sampled frame correspond to the following relationship:

[0052] ;

[0053] in, Indicates the first The first perspective The trajectory in the first Visual feature vectors on each sampled frame This represents the bilinear interpolation operator. Represents a set The first in The trajectory in the first Two-dimensional spatial coordinates on each sampling frame Indicates the index of the trajectory, and ; Represents the total number of trajectories;

[0054] The first The first perspective The visual feature vectors of the trajectory on all sampling frames are concatenated in chronological order to obtain the first trajectory. The first perspective The visual feature sequence of the trajectory corresponds to the following relationship:

[0055] ;

[0056] in, Indicates the first The first perspective Visual feature sequence of trajectories Indicates the first The first perspective The visual feature vector of the trajectory on the first sampling frame. Indicates the first The first perspective The visual feature vector of the trajectory on the second sampling frame. Indicates the first The first perspective The trajectory in the first Visual feature vectors on each sampled frame.

[0057] Step 3: Input the visual feature sequence of the trajectory into a sequence modeling network based on a selective state space model for processing, and capture the spatial semantic association between trajectories to obtain the enhanced hidden state sequence of the trajectory.

[0058] In step 3, the visual feature sequence of the trajectory is input into a sequence modeling network based on a selective state space model for processing, and the spatial semantic associations between trajectories are captured to obtain the enhanced hidden state sequence of the trajectory. Specifically, this includes the following sub-steps:

[0059] For the first The first perspective The visual feature sequence of the trajectory is linearly transformed to obtain the first... The first perspective The initial embedding of the trajectories corresponds to the following relationship:

[0060] ;

[0061] in, Indicates the first The first perspective Preliminary embedding of the trajectory, Represents a learnable linear transformation matrix. Represents a learnable bias vector;

[0062] The learnable position encoding matrix is ​​compared with the first The first perspective The initial embeddings of the trajectories are summed to obtain the trace number. The first perspective The input sequence of trajectories corresponds to the following relationship in the process:

[0063] ;

[0064] in, Indicates the first The first perspective The input sequence of trajectories, This represents the learnable positional encoding matrix;

[0065] The first The first perspective The input sequence of the trajectory is fed into the Mamba model for processing to obtain the trajectory. The first perspective The enhanced hidden state sequence of the trajectory corresponds to the following relationship:

[0066] ;

[0067] in, Indicates the first The first perspective Enhanced hidden state sequence of trajectories, This represents the complete forward propagation function of the Mamba model;

[0068] Furthermore, the first The first perspective The input sequence of trajectories is fed into the Mamba model for processing, specifically including the following sub-steps:

[0069] Input sequence The input sequence is fed into the Mamba model and processed through a selective state-space kernel. Modeling is performed; the selective state-space core is achieved by discretizing parameters. With continuous state space parameters The continuous state-space equations are discretized, and parameters are dynamically adjusted using an input-dependent selection mechanism, thereby achieving [the desired effect] in the time dimension. Up-recursive computation of hidden state and output The following relationship exists in the correspondence process:

[0070] ;

[0071] in, Indicates the discretization step size. , and All of these represent continuous state-space parameters. Indicates time step The hidden state vector, Represents the discrete state transition matrix. This represents a discrete input matrix.

[0072] Step 4: Based on the cross-view attention mechanism, the enhanced hidden state sequence of the trajectory is fused to obtain unified multi-view fusion features.

[0073] In step 4, the enhanced hidden state sequence of the trajectory is fused based on a cross-view attention mechanism to obtain unified multi-view fusion features, specifically including the following sub-steps:

[0074] The first The enhanced hidden state sequences of all trajectories from each viewpoint are concatenated to obtain the first... The viewpoint-level feature vectors of each viewpoint correspond to the following relationship:

[0075] ;

[0076] in, Indicates the first Viewpoint-level feature vectors for each viewpoint This indicates that a splicing operation has been performed. Indicates the first The enhanced hidden state sequence of the first trajectory from each perspective. Indicates the first The enhanced hidden state sequence of the second trajectory from each perspective. Indicates the first The first perspective An enhanced hidden state sequence of trajectories;

[0077] The view-level feature vectors from all perspectives are stacked to construct the input feature matrix. The following relationship exists in this process:

[0078] ;

[0079] in, Represents the input feature matrix. This represents the viewpoint-level feature vector of the first viewpoint. This represents the viewpoint-level feature vector of the second viewpoint. This represents the viewpoint-level feature vector of the last viewpoint. Indicates transpose;

[0080] The input feature matrix is ​​processed using a cross-view attention mechanism to obtain view fusion features after attention aggregation;

[0081] A feedforward network is used to perform nonlinear transformation and dimensional mapping on the viewpoint fusion features after attention aggregation to obtain unified multi-viewpoint fusion features. The following relationship exists in the corresponding process:

[0082] ;

[0083] in, Represents a one-dimensional vector. This function represents the conversion of multidimensional data into one-dimensional data. This represents the viewpoint fusion feature after attention aggregation. Indicates hidden layer fusion features, This represents the modified linear unit function. This represents the first-layer weight matrix. This represents the first-level bias vector. This indicates a unified multi-perspective fusion feature. This represents the second-layer weight matrix. This represents the second-layer bias vector.

[0084] Step 5: Input the unified multi-view fusion features into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal.

[0085] In step 5, the unified multi-view fusion features are input into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal. This includes the following sub-steps:

[0086] A two-layer nonlinear transformation network is used to process the unified multi-view fusion features to obtain high-level discriminative features. The following relationship exists in the corresponding process:

[0087] ;

[0088] in, Indicates high-level discriminative features, , , and All represent learnable parameters;

[0089] The high-level discriminative features are transformed using a logistic regression layer to obtain the logistic score for anomaly detection. The following relationship exists in the corresponding process:

[0090] ;

[0091] in, The logical score representing the anomaly determination. and All represent learnable parameters;

[0092] The logical scores for anomaly detection are mapped using the Sigmoid function to obtain the anomaly probability. The following relationship exists in the mapping process:

[0093] ;

[0094] in, Indicates the probability of an anomaly. Represents an exponential function;

[0095] When the probability of an anomaly is less than 0.5, it is judged as a normal posture; otherwise, it is judged as an abnormal posture.

[0096] A binary cross-entropy loss function is constructed based on the detection results of abnormal human posture, and the following relationship exists in the corresponding process:

[0097] ;

[0098] in, Represents the binary cross-entropy loss. Indicates the number of samples. Indicates the index of the sample. Indicates the first The true label of each sample Indicates taking the logarithm. Indicates the first The predicted anomaly probability of each sample;

[0099] The human abnormal pose classification network is optimized using the binary cross-entropy loss function to obtain the optimized human abnormal pose classification network.

[0100] The unified multi-view fusion features are input into the optimized human abnormal pose classification network for detection and classification to obtain the final detection results.

[0101] It should be noted that during training, the Adam optimizer is used to minimize the binary cross-entropy loss function, and the learning rate is set to... The weight decay coefficient is This continues until the binary cross-entropy loss function converges or the preset number of training rounds is reached.

[0102] Please see Figure 2 This embodiment also provides a human abnormal pose detection system based on a multi-view fusion network, wherein the system applies the human abnormal pose detection method based on a multi-view fusion network as described above, and the system includes:

[0103] The multi-view trajectory perception and feature encoding module is used for:

[0104] A multi-view video set of human poses is obtained by using a pre-set multi-view camera; the human pose key points in the video set of the multi-view human poses are located by using a pose estimation algorithm; and the same key point is associated across frames by using a key point tracking algorithm to construct a set of human pose key point trajectory segments from each viewpoint.

[0105] Based on the set of trajectory fragments of human posture key points from various perspectives, feature sampling is performed using a bilinear interpolation algorithm to obtain the visual feature sequence of the trajectory.

[0106] The spatiotemporal modeling and multi-view fusion module is used for:

[0107] The visual feature sequence of the trajectory is input into a sequence modeling network based on a selective state space model for processing, and the spatial semantic association between trajectories is captured to obtain the enhanced hidden state sequence of the trajectory.

[0108] The enhanced hidden state sequence of the trajectory is fused based on the cross-view attention mechanism to obtain unified multi-view fusion features;

[0109] The abnormal pose classification decision module is used for:

[0110] The unified multi-view fusion features are input into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal.

[0111] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0112] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0113] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0114] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for detecting abnormal human poses based on a multi-view fusion network, characterized in that, The method includes the following steps: Step 1: Use a preset multi-view camera to capture a multi-view video set of human poses; use a pose estimation algorithm to locate the human pose key points in the multi-view video set of human poses, and use a key point tracking algorithm to associate the same key point across frames to construct a set of human pose key point trajectory segments from each viewpoint. Step 2: Based on the set of trajectory segments of human posture key points from various perspectives, use the bilinear interpolation algorithm to perform feature sampling to obtain the visual feature sequence of the trajectory. Step 3: Input the visual feature sequence of the trajectory into a sequence modeling network based on a selective state space model for processing, and capture the spatial semantic associations between trajectories to obtain the enhanced hidden state sequence of the trajectory. This includes the following sub-steps: For the The first perspective The visual feature sequence of the trajectory is linearly transformed to obtain the first... The first perspective The initial embedding of the trajectories corresponds to the following relationship: ； in, Indicates the first The first perspective Preliminary embedding of the trajectory, Indicates the first The first perspective Visual feature sequence of trajectories Represents a learnable linear transformation matrix. Represents a learnable bias vector; The learnable position encoding matrix is ​​compared with the first The first perspective The initial embeddings of the trajectories are summed to obtain the trace number. The first perspective The input sequence of trajectories corresponds to the following relationship in the process: ； in, Indicates the first The first perspective The input sequence of trajectories, This represents the learnable positional encoding matrix; The first The first perspective The input sequence of the trajectory is fed into the Mamba model for processing to obtain the trajectory. The first perspective The enhanced hidden state sequence of the trajectory corresponds to the following relationship: ； in, Indicates the first The first perspective Enhanced hidden state sequence of trajectories, This represents the complete forward propagation function of the Mamba model; Step 4: Fusion processing of the enhanced hidden state sequence of the trajectory based on the cross-view attention mechanism to obtain unified multi-view fusion features, specifically including the following sub-steps: The first The enhanced hidden state sequences of all trajectories from each viewpoint are concatenated to obtain the first... Viewpoint-level feature vectors for each viewpoint; Stack the view-level feature vectors from all viewpoints to construct the input feature matrix; The input feature matrix is ​​processed using a cross-view attention mechanism to obtain view fusion features after attention aggregation; A feedforward network is used to perform nonlinear transformation and dimensional mapping on the viewpoint fusion features after attention aggregation to obtain unified multi-viewpoint fusion features. Step 5: Input the unified multi-view fusion features into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal.

2. The human abnormal pose detection method based on multi-view fusion network according to claim 1, characterized in that, In step 1, a set of multi-view videos of human poses is captured using a preset multi-view camera; a pose estimation algorithm is used to locate key points of human poses in the videos of the multi-view video set; and a key point tracking algorithm is used to associate the same key point across frames to construct a set of trajectory segments of human pose key points from each viewpoint. Specifically, this includes the following sub-steps: A collection of multi-view videos of human postures captured using a pre-set multi-view camera. ; in, A collection of multi-view videos representing human postures. This represents the video sequence from the first perspective. This represents a video sequence from a second perspective. Indicates the first A video sequence from multiple perspectives, This represents the video sequence from the last perspective. Indicates the total number of viewpoints. An index indicating the viewpoint; Based on a multi-view video collection of human poses, the first Video sequences from various perspectives Sampling was performed to obtain the first A collection of sampled frames from various perspectives And select the first sampling frame. As the starting frame for trajectory tracking; in, Indicates the first A set of sampled frames from various perspectives Indicates the first The first sampled frame from each viewpoint Indicates the first The second sampling frame from each viewpoint Indicates the first The first perspective Each sampling frame, Indicates the index of the sampled frame. Indicates the first The first perspective Each sampling frame, Indicates the total number of sampled frames; The YOLO-Pose pose estimation algorithm is used to analyze and process the starting frame, and human pose key points with confidence scores lower than the preset confidence threshold are filtered out to obtain the set of effective human pose key points in the starting frame. Using key point tracking algorithm for the first The set of sampled frames from the first perspective and the first The effective human pose keypoint set of the first viewpoint starting frame is processed to obtain the first viewpoint. A collection of trajectory segments of key points in human posture from multiple perspectives.

3. The human abnormal pose detection method based on multi-view fusion network according to claim 2, characterized in that, In the process of analyzing and processing the starting frame using the YOLO-Pose pose estimation algorithm, and filtering out human pose keypoints with confidence scores lower than a preset confidence threshold to obtain the set of valid human pose keypoints in the starting frame, the following relationship exists: ； in, Indicates the first A set of effective human pose key points for each viewpoint starting frame. , and Both represent sets The elements in Indicates the first The first frame of the viewpoint Key points for effective human posture An index representing the key points of an effective human pose. This indicates the total number of valid human pose keypoints in the starting frame. Indicating key points of human posture The horizontal pixel coordinates in the starting frame Indicating key points of human posture Vertical pixel coordinates in the starting frame Indicating key points of human posture The confidence score, Indicating key points of human posture Type tags; Using key point tracking algorithm for the first The set of sampled frames from the first perspective and the first The effective human pose keypoint set of the first viewpoint starting frame is processed to obtain the first viewpoint. In the process of creating a set of trajectory fragments of human posture key points from multiple perspectives, the following relationship exists: ； in, Indicates the first A collection of trajectory segments of key points in human posture from multiple perspectives. This indicates that the data has been processed using the PIP3 keypoint tracking model.

4. The human abnormal pose detection method based on multi-view fusion network according to claim 3, characterized in that, In step 2, based on the set of trajectory segments of human pose key points from various viewpoints, feature sampling is performed using a bilinear interpolation algorithm to obtain the visual feature sequence of the trajectory. This specifically includes the following sub-steps: The first The first perspective Sample frames The input is fed into a pre-trained visual encoding model for feature extraction to obtain the first... The first perspective Deep visual feature map of each sampled frame ; in, Indicates the first The first perspective Deep visual feature maps of each sampled frame; Based on sets The Middle The trajectory in the first The two-dimensional spatial coordinates on each sample frame are used to extract the deep visual feature map using a bilinear interpolation algorithm. Feature sampling is performed to obtain the first... The trajectory in the first Visual feature vectors on each sampled frame; The first The first perspective The visual feature vectors of the trajectory on all sampling frames are concatenated in chronological order to obtain the first trajectory. The first perspective A sequence of visual features of a trajectory.

5. The human abnormal pose detection method based on multi-view fusion network according to claim 4, characterized in that, In the first The first perspective Sample frames The input is fed into a pre-trained visual encoding model for feature extraction to obtain the first... The first perspective In the process of generating deep visual feature maps for each sampled frame, the following relationship exists: ； in, This indicates that the code has undergone forward encoding processing using the SigLIP model. Based on sets The Middle The trajectory in the first The two-dimensional spatial coordinates on each sample frame are used to extract the deep visual feature map using a bilinear interpolation algorithm. Feature sampling is performed to obtain the first... The trajectory in the first In the steps of sampling the visual feature vectors on each frame, the following relationship exists: ； in, Indicates the first The first perspective The trajectory in the first Visual feature vectors on each sampled frame This represents the bilinear interpolation operator. Represents a set The first in The trajectory in the first Two-dimensional spatial coordinates on each sampling frame Index representing the trajectory; In the first The first perspective The visual feature vectors of the trajectory on all sampling frames are concatenated in chronological order to obtain the first trajectory. The first perspective In the steps of visual feature sequence of a trajectory, the following relationship exists: ； in, Indicates the first The first perspective The visual feature vector of the trajectory on the first sampling frame. Indicates the first The first perspective The visual feature vector of the trajectory on the second sampling frame. Indicates the first The first perspective The trajectory in the first Visual feature vectors on each sampled frame.

6. The human abnormal pose detection method based on multi-view fusion network according to claim 5, characterized in that, In the first The enhanced hidden state sequences of all trajectories from each viewpoint are concatenated to obtain the first... In the steps of generating view-level feature vectors from each viewpoint, the following relationship exists: ； in, Indicates the first Viewpoint-level feature vectors for each viewpoint This indicates that a splicing operation has been performed. Indicates the first The enhanced hidden state sequence of the first trajectory from each perspective. Indicates the first The enhanced hidden state sequence of the second trajectory from each perspective. Indicates the first The first perspective An enhanced hidden state sequence of trajectories; In the step of stacking the view-level feature vectors from all viewpoints to construct the input feature matrix, the following relationship exists: ； in, Represents the input feature matrix. This represents the viewpoint-level feature vector of the first viewpoint. This represents the viewpoint-level feature vector of the second viewpoint. This represents the viewpoint-level feature vector of the last viewpoint. Indicates transpose; In the step of using a feedforward network to perform nonlinear transformation and dimensional mapping on the attention-aggregated viewpoint fusion features to obtain unified multi-viewpoint fusion features, the following relationship exists: ； in, Represents a one-dimensional vector. This function represents the conversion of multidimensional data into one-dimensional data. This represents the viewpoint fusion feature after attention aggregation. Indicates hidden layer fusion features, This represents the modified linear unit function. This represents the first-layer weight matrix. This represents the first-level bias vector. This indicates a unified multi-perspective fusion feature. This represents the second-layer weight matrix. This represents the second-layer bias vector.

7. The human abnormal pose detection method based on multi-view fusion network according to claim 6, characterized in that, In step 5, the unified multi-view fusion features are input into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal. Specifically, it includes the following sub-steps: A two-layer nonlinear transformation network is used to process the unified multi-view fusion features to obtain high-level discriminative features. The following relationship exists in the corresponding process: ； in, Indicates high-level discriminative features, , , and All represent learnable parameters; The high-level discriminative features are transformed using a logistic regression layer to obtain the logistic score for anomaly detection. The following relationship exists in the corresponding process: ； in, The logical score representing the anomaly determination. and All represent learnable parameters; The logical scores for anomaly detection are mapped using the Sigmoid function to obtain the anomaly probability. The following relationship exists in the mapping process: ； in, Indicates the probability of an anomaly. Represents an exponential function; When the probability of an anomaly is less than 0.5, it is judged as a normal posture; otherwise, it is judged as an abnormal posture. A binary cross-entropy loss function is constructed based on the detection results of abnormal human posture, and the following relationship exists in the corresponding process: ； in, Represents the binary cross-entropy loss. Indicates the number of samples. Indicates the index of the sample. Indicates the first The true label of each sample Indicates taking the logarithm. Indicates the first The predicted anomaly probability of each sample; The human abnormal pose classification network is optimized using the binary cross-entropy loss function to obtain the optimized human abnormal pose classification network. The unified multi-view fusion features are input into the optimized human abnormal pose classification network for detection and classification to obtain the final detection results.

8. A human abnormal pose detection system based on a multi-view fusion network, characterized in that, The system employs the human abnormal pose detection method based on a multi-view fusion network as described in any one of claims 1 to 7, and the system comprises: The multi-view trajectory perception and feature encoding module is used for: A multi-view video set of human poses is obtained by using a pre-set multi-view camera; the human pose key points in the video set of the multi-view human poses are located by using a pose estimation algorithm; and the same key point is associated across frames by using a key point tracking algorithm to construct a set of human pose key point trajectory segments from each viewpoint. Based on the set of trajectory fragments of human posture key points from various perspectives, feature sampling is performed using a bilinear interpolation algorithm to obtain the visual feature sequence of the trajectory. The spatiotemporal modeling and multi-view fusion module is used for: The visual feature sequence of the trajectory is input into a sequence modeling network based on a selective state space model for processing, and the spatial semantic association between trajectories is captured to obtain the enhanced hidden state sequence of the trajectory. The enhanced hidden state sequence of the trajectory is fused based on the cross-view attention mechanism to obtain unified multi-view fusion features; The abnormal pose classification decision module is used for: The unified multi-view fusion features are input into the human abnormal pose classification network for detection and classification to obtain the detection result of whether the human pose is abnormal.

Images