A machine vision violence detection analysis method and system for small sample meta-learning

Abstract

The present application relates to the technical field of machine vision and small sample learning, and discloses a machine vision violence detection analysis method and system based on small sample meta learning, wherein the method comprises the following steps: extracting human target features and semantic action primitive dictionaries; calculating role probability distribution vectors and generating a role-labeled action primitive sequence set; generating a cross-role action primitive causal constraint graph; performing role constraint counterfactual intervention on rare role configuration modes to generate a counterfactual multi-person primitive sequence combination coordinated by roles; decoding to generate a space-time coordinated multi-person synthetic skeleton sequence set; constructing a space-time heterogeneous interaction graph; extracting space-time role interaction feature vectors and calculating meta learning prototype vectors by using a relationship type perception graph attention network; and generating a violence behavior detection judgment result based on distance measurement values.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of machine vision and few-shot learning technology, and in particular to a machine vision violence detection and analysis method and system based on few-shot meta-learning. Background Technology

[0002] With the continuous expansion of public safety monitoring networks and the in-depth application of video analytics technology, automatic detection of violent behavior based on machine vision has become an important research direction in the field of intelligent security. Multi-person collaborative violent behavior, due to its large number of participants, complex role divisions, and diverse action coupling relationships, places high demands on the spatiotemporal modeling capabilities of detection methods.

[0003] At the technical level, existing violence detection methods typically rely on large-scale labeled data for supervised training. However, in real-world scenarios, labeled samples for rare character configurations (such as multiple perpetrators attacking or one person threatening multiple others with a weapon) are extremely scarce, severely limiting the generalization ability of such methods under small sample conditions. While few-sample meta-learning methods can quickly adapt to new categories under limited sample conditions, the reliability of their prototype computation is highly dependent on the quality and diversity of synthetic samples.

[0004] Existing action sequence synthesis methods based on statistical co-occurrence relationships are unreliable under conditions of very few samples and fail to model the action coupling constraints between multiple characters, resulting in a lack of coordination and consistency among the action sequences of each character in the synthesized samples, thus failing to provide reliable training signals for meta-learning prototype computation. Furthermore, existing violence detection methods based on few-sample meta-learning treat all human target nodes in the interaction graph as homogeneous nodes and process them using a uniform feature aggregation method, causing the violence interaction features to be diluted by bystander noise, further weakening the detection accuracy.

[0005] The inventors of this disclosure have noted that current small-sample violence detection schemes do not take into account the cross-role causal coupling constraints between different participating roles in multi-person collaborative violence scenarios, resulting in insufficient coordination of role actions in synthetic samples under rare role configuration modes. At the same time, the homogenization aggregation of bystander nodes in heterogeneous interaction graphs introduces feature noise, which reduces the discriminativeness of meta-learning prototype vectors. Summary of the Invention

[0006] The purpose of this invention is to provide a machine vision violence detection and analysis method and system based on small sample meta-learning, which solves the technical problems of existing violence detection methods such as insufficient modeling of role interaction relationships in multi-person collaborative violence scenarios under small sample conditions, prototype vector estimation bias caused by sample scarcity, and lack of cross-role causal constraints.

[0007] This disclosure provides at least one embodiment of a machine vision violence detection and analysis method based on few-sample meta-learning. The method includes: acquiring a video frame sequence; extracting the appearance feature vector and skeleton joint coordinate sequence of a human target; performing atomic action segmentation and encoding on the skeleton motion sequence; generating a semantic action primitive dictionary through clustering; calculating a motion aggression index based on the skeleton motion sequence; generating a role probability distribution vector by combining motion direction features; generating a set of action primitive sequences labeled with the dominant role; performing cross-role temporal alignment of primitive sequences of different roles in the same violence scene; using a causal discovery algorithm to examine the causal direction and response delay between primitives of different roles; generating a cross-role action primitive causal constraint graph; and in the cross-role action primitive causal constraint graph... On the beamform, counterfactual intervention is performed on primitive sequences of rare character configuration patterns to generate a counterfactual multi-person primitive sequence combination with coordinated roles. Each character primitive sequence is decoded into a skeleton motion sequence and time alignment constraints, spatial distance constraints, and motion continuity constraints are applied to generate a set of multi-person synthetic skeleton sequences. The set of multi-person synthetic skeleton sequences is merged with the original samples to form an extended sample set, and a spatiotemporal heterogeneous interaction graph is generated based on spatial positional relationships and motion direction relationships. The spatiotemporal role interaction feature vector is extracted from the spatiotemporal heterogeneous interaction graph using a relation type-aware graph attention network, and violent interaction prototype vectors and normal interaction prototype vectors are calculated in a meta-learning framework. Spatiotemporal role interaction feature vectors are generated for the video to be detected, and violent behavior detection judgment results are generated based on the distance metric between the vectors and each prototype vector.

[0008] For example, in the machine vision violence detection and analysis method based on small sample meta-learning provided in at least one embodiment of this disclosure, the step of calculating the motion aggression index based on the skeleton motion sequence and generating the character probability distribution vector by combining the motion direction features includes: calculating the skeleton motion sequence of each human target in each atomic action segment, and calculating the average limb extension speed. Mean angle of forward lean of center of gravity Arm swing frequency After being standardized using Z-score, according to Calculate the sports aggression index ,in An index for human targets. , , The preset weighting coefficients satisfy the following conditions: Extracting motion direction features The By the The cosine similarity between the displacement vector of an individual human target within the current atomic action segment and the direction vector pointing from that human target to the average position of all other human targets in the scene is calculated; this motion aggression index is then used to... With the aforementioned motion direction features After concatenation, the vectors are mapped to a 3D vector through a fully connected layer and a softmax operation is applied to generate the character probability distribution vector. ,in , , These correspond to the probability values ​​for attacking, defending, and spectating roles, respectively.

[0009] For example, in the machine vision violence detection and analysis method based on few-sample meta-learning provided in at least one embodiment of this disclosure, the cross-role action primitive causal constraint graph is a directed graph structure, where each node corresponds to a primitive type labeled by a role. The cross-role action primitive causal constraint graph contains three types of directed edges: intra-role causal precedence edges, connecting two primitive type nodes under the same role label, indicating that the predecessor primitive causally precedes the successor primitive in temporal order; cross-role response causal edges, connecting two primitive type nodes under different role labels, indicating that the primitive of the source role causally triggers the primitive response of the target role, with a response delay parameter marked on the edge; and mutually exclusive constraint edges, connecting primitive type node pairs that cannot occur simultaneously at the physical level. The causal discovery algorithm confirms the existence of causal edges under the condition that the statistical significance level meets a preset threshold, where the preset threshold is the significance level parameter of the conditional independence test.

[0010] For example, in the machine vision violence detection and analysis method based on few-sample meta-learning provided in at least one embodiment of this disclosure, the role constraint counterfactual intervention includes: selecting the target primitive position in the primitive sequence of the target role, and searching for candidate alternative primitives with the same role label as the target primitive and simultaneously satisfying the following conditions in the cross-role action primitive causal constraint graph: there is an intra-role causal pre-edge between the candidate alternative primitive and the predecessor primitive of the target primitive; there is an intra-role causal pre-edge between the candidate alternative primitive and the successor primitive of the target primitive; and the candidate alternative primitive does not violate the mutual exclusion constraint edge limitation between primitives in the primitive sequences of other roles within the corresponding time window.

[0011] For example, in the machine vision violence detection and analysis method based on few-sample meta-learning provided in at least one embodiment of this disclosure, after the candidate substitution primitive is filtered through intra-role constraints, a cross-role causal consistency test is further performed, including: retrieving all cross-role response causal edges with the candidate substitution primitive node as the source node or target node in the cross-role action primitive causal constraint graph; if the candidate substitution primitive is the source node of the cross-role response causal edge, then verifying that the target primitive type pointed to by the causal edge exists in the primitive sequence of the target role within the time window specified by the response delay parameter; if the candidate substitution primitive is the target node of the cross-role response causal edge, then verifying that the source primitive type of the causal edge exists in the primitive sequence of the source role within the corresponding pre-time window; the substitution scheme is determined to be valid only when the candidate substitution primitive simultaneously satisfies the intra-role causal pre-constraint, mutual exclusion constraint, and cross-role response causal constraint.

[0012] For example, in the machine vision violence detection and analysis method based on few-sample meta-learning provided in at least one embodiment of this disclosure, the generation of spatiotemporal heterogeneous interaction graphs based on spatial positional relationships and motion direction relationships includes: taking each human target in each time frame as a graph node, the node features being the concatenation result of the human target's role probability distribution vector, appearance feature vector, and skeleton motion features after Z-score normalization; for any two human nodes in the current frame, determining the relationship type of the edge based on the combination of the dominant role labels of the two end nodes, the relationship type including interaction edges between attacking and defending roles, interaction edges between attacking and spectating roles, interaction edges between defending and spectating roles, and interaction edges between similar roles; the edge generation condition is that the spatial distance between the two end nodes does not exceed a preset interaction distance threshold, or the motion directions of the two end nodes show a trend of approaching each other; performing cross-frame matching along the time dimension for the same human target node in adjacent frames based on the cosine similarity of appearance feature vectors and spatial position continuity, and adding temporal connection edges between successfully matched cross-frame node pairs.

[0013] For example, in the machine vision violence detection and analysis method based on few-sample meta-learning provided in at least one embodiment of this disclosure, the relation type-aware graph attention network focuses on nodes. In relation types The following features are updated to ,in For nodes In relation types The updated feature vector is as follows. For nodes In relation types The set of neighboring nodes below, For relation type The corresponding learnable linear transformation matrix, For nodes The input feature vector, Non-linear activation function; attention weights Calculated as ,in For relation type The corresponding learnable parameter vector, This represents a vector concatenation operation. After the aggregation results under each relation type are fused, each edge is concatenated with the updated features of its two end nodes and the learnable type embedding vector of the relation type, and then linearly transformed to generate an edge embedding vector. The edge embedding vector sequence of the interaction edges between the attacking and defending roles is averaged along the time dimension to output the spatiotemporal role interaction feature vector.

[0014] For example, in the machine vision violence detection and analysis method based on few-sample meta-learning provided in at least one embodiment of this disclosure, the step of generating a violence behavior detection judgment result based on the distance metric between each prototype vector includes: calculating the Euclidean distance between the spatiotemporal role interaction feature vector of the video to be detected and the violence interaction prototype vector under each role configuration mode, and the Euclidean distance between the video and the normal interaction prototype vector; when the distance between the spatiotemporal role interaction feature vector and the violence interaction prototype vector of a certain role configuration mode is less than the distance between the video and the normal interaction prototype vector, and the absolute value of the difference between the two exceeds a preset judgment threshold, the video to be detected is judged to contain violence interaction behavior; and detection alarm information is output, wherein the detection alarm information includes the determined violence type label, the role identifier of each participant, and the spatial position information of each participant in the picture.

[0015] At least one embodiment of this disclosure also provides a machine vision violence detection and analysis system based on few-sample meta-learning. This system includes a video feature extraction unit, a role probability allocation unit, a causal constraint graph generation unit, a counterfactual intervention and synthesis unit, a heterogeneous interaction graph generation unit, a feature aggregation and prototype calculation unit, and a detection and judgment unit. The video feature extraction unit is configured to acquire video frame sequences, extract the appearance feature vectors and skeleton joint coordinate sequences of human targets, perform atomic action segmentation and encoding on the skeleton motion sequences, and generate a semantic action primitive dictionary through clustering. The role probability allocation unit is configured to calculate a motion aggression index based on the skeleton motion sequences, generate a role probability distribution vector by combining motion direction features, and generate a set of action primitive sequences labeled with the dominant role based on the dominant role label. The causal constraint graph generation unit is configured to perform cross-role temporal alignment of primitive sequences of different roles in the same violence scene, use a causal discovery algorithm to examine the causal direction and response delay between primitives of different roles, and generate... A cross-role action primitive causal constraint graph is constructed. The counterfactual intervention and synthesis unit is configured to perform role constraint counterfactual intervention on primitive sequences of rare role configuration patterns on the cross-role action primitive causal constraint graph, generate a combination of counterfactual multi-person primitive sequences for role coordination, and decode each role primitive sequence into a skeleton motion sequence and apply time alignment constraints, spatial distance constraints, and motion continuity constraints to generate a set of multi-person synthesized skeleton sequences. The heterogeneous interaction graph generation unit is configured to merge the set of multi-person synthesized skeleton sequences with the original samples into an extended sample set, and generate a spatiotemporal heterogeneous interaction graph based on spatial positional relationships and motion direction relationships. The feature aggregation and prototype calculation unit is configured to extract spatiotemporal role interaction feature vectors from the spatiotemporal heterogeneous interaction graph using a relation type-aware graph attention network, and calculate violent interaction prototype vectors and normal interaction prototype vectors in a meta-learning framework. The detection and judgment unit is configured to generate spatiotemporal role interaction feature vectors for the video to be detected, and generate violent behavior detection and judgment results based on the distance metric between the vectors and each prototype vector.

[0016] The machine vision violence detection and analysis method and system based on small sample meta-learning disclosed herein can achieve at least the following technical effects.

[0017] Firstly, by calculating the probability distribution vector of each human target, which includes three types of probability values: attacking role, defending role, and spectator role, and incorporating the role probability distribution information as a continuous feature into the node representation of the spatiotemporal heterogeneous interaction graph, different participating roles are differentiated in the graph structure, thereby reducing the interference of spectator nodes on violent interaction features during feature aggregation.

[0018] Secondly, by constructing a cross-role action primitive causal constraint graph, the causal direction, response delay, and mutual exclusion relationship of primitives between different roles are explicitly modeled. When performing role constraint counterfactual intervention on a very small number of samples with rare role configuration patterns, the generated counterfactual multi-person primitive sequence combination is constrained by the causal constraint graph, and the causal consistency between the action sequences of each role is guaranteed, thereby improving the role coordination of the synthesized samples.

[0019] Third, by jointly applying time alignment constraints, prior constraints on spatial distance between characters, and constraints on joint motion continuity during the generation of the synthetic skeleton sequence, the generated multi-person synthetic skeleton sequence satisfies the physical rationality requirements of multi-person collaborative violent behavior in both the time and spatial dimensions, thereby providing more reliable training samples for meta-learning prototype computation.

[0020] Fourth, the relation type-aware graph attention network uses independent attention computing heads to aggregate edges of different relation types in a differentiated manner, and extracts the edge embedding vector sequence of the interaction edges between the attacking and defending roles separately for the calculation of spatiotemporal role interaction feature vectors. This makes the feature representation used for prototype calculation more focused on reflecting the core violent interaction relationship, which helps to improve the feature discrimination between different role configuration modes.

[0021] Fifth, in the detection and judgment stage, the spatiotemporal role interaction feature vector of the video to be detected is compared with the violent interaction prototype vector and normal interaction prototype vector of each role configuration mode by distance measurement, and the detection alarm information containing violence type label, participant role identifier and spatial location information is output, so that the detection results are interpretable and facilitate the targeted handling of subsequent security responses. Attached Figure Description

[0022] Figure 1 This is a flowchart of the machine vision violence detection and analysis method based on few-sample meta-learning provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the overall processing flow of the violent behavior detection system provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the probability distribution comparison of various human target roles provided in the embodiments of the present invention; Figure 4 This is a schematic diagram illustrating the cross-role action primitive causal constraint graph structure provided in an embodiment of the present invention; Figure 5 This is a schematic diagram comparing the sample size of each role configuration mode before and after counterfactual intervention, provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the Euclidean distance distribution between the video to be detected and each prototype vector provided in an embodiment of the present invention; Figure 7This is a schematic diagram of the node and edge structure of a single-frame spatiotemporal heterogeneous interaction graph provided in an embodiment of the present invention (frame 43); Figure 8 This is a schematic diagram of the frame number distribution of the skeleton joint point sequence of each atomic motion segment of the human body 1 provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the comparison of the weighted components of the offensive index provided in the embodiment of the present invention (human body 1 segment 2). Detailed Implementation

[0023] Example 1 In real-world surveillance scenarios, participants in coordinated acts of violence often assume different roles. Perpetrators exhibit aggressive behavior, victims exhibit defensive and withdrawn behavior, and bystanders exhibit observing behavior. The actions of these different roles are coupled in both temporal and spatial dimensions. Existing violence detection methods based on few-sample meta-learning treat all human target nodes in the interaction graph as homogeneous nodes and process them using a uniform feature aggregation method, resulting in the violent interaction features being diluted by bystander noise. Furthermore, labeled samples for rare role configuration patterns (e.g., multiple perpetrators attacking, one person threatening multiple people with a weapon) are extremely scarce. Existing action sequence synthesis methods based on statistical co-occurrence relationships are unreliable under extremely limited sample conditions and do not model the action coupling constraints between multiple roles. This leads to a lack of coordination and consistency in the action sequences of each role in the synthesized samples, failing to provide reliable training signals for meta-learning prototype computation.

[0024] This disclosure provides at least one embodiment of a machine vision-based violence detection and analysis method using small-sample meta-learning, such as... Figure 1 As shown, the process includes the following steps S100 to S800.

[0025] Step S100: Obtain the video frame sequence and extract human target features and semantic action primitives dictionary.

[0026] In step S100, a continuous frame sequence of the video to be analyzed and all available multi-person violent behavior video samples is obtained. The bounding box coordinates, appearance feature vectors and skeleton joint coordinate sequences of the human target are extracted for each frame image. Atomic action segmentation and encoding are performed on the skeleton motion sequence of each human target. A semantic action primitive dictionary is generated by clustering.

[0027] For example, a target detection network is used to perform human target detection on each frame of the image. The input of the target detection network is a single frame of the image, and the output is the bounding box coordinates and the corresponding appearance feature vector for each detected human target. A pose estimation network is used to extract the skeleton joints of each detected human target. The input of the pose estimation network is the human target image region obtained by cropping the bounding box, and the output is the sequence of skeleton joint coordinates of the human target.

[0028] In at least one example of the embodiments of this disclosure, for each human target's skeleton motion sequence arranged along the time dimension, a temporal segmentation algorithm is used to divide the skeleton motion sequence into several atomic motion segments according to the abrupt changes in joint motion speed and direction. Each atomic motion segment corresponds to continuous skeleton motion within a time interval. Each atomic motion segment is encoded using a variational autoencoder. The input of the variational autoencoder is the temporal matrix of skeleton joint coordinates corresponding to the atomic motion segment, and the output is a fixed-dimensional semantic action primitive vector and its corresponding latent variable distribution parameters. During training, the sum of the reconstruction error of the skeleton joint coordinate temporal matrix and the KL divergence of the latent variable distribution is used as the loss function, and the Adam optimization algorithm is used to update the parameters. A clustering algorithm is performed on the semantic action primitive vectors extracted from all samples. The input of the clustering algorithm is the set of semantic action primitive vectors of all samples, and the output is several cluster center vectors. Each cluster center is defined as a semantic action primitive type, and the set of all cluster centers constitutes a semantic action primitive dictionary.

[0029] Step S200: Calculate the character probability distribution vector based on the skeleton motion sequence and generate a set of action primitive sequences labeled with the character.

[0030] In step S200, a motion aggression index is calculated based on the skeletal motion sequence of each human target. Combining motion direction features, a soft assignment method is used to generate a character probability distribution vector containing probability values ​​of three types: attacking role, defensive role, and spectator role. The dominant role label is attached to each action primitive corresponding to the human target to generate a set of action primitive sequences labeled with the role.

[0031] For example, for each human target's skeletal motion sequence within each atomic motion segment, in calculating the motion aggression index... Previously, for , , Z-score standardization was applied to eliminate the impact of dimensional differences among the three metrics on the weighted summation calculation. The standardized metric for motion aggression was then obtained. Defined as: in, For the first An indicator of the aggressiveness of an individual target's movement within the current atomic action segment. An index for human targets. For the first The average, standardized limb extension velocity of an individual target within the current atomic motion segment. This is the standardized mean forward tilt angle of the center of gravity. The standardized arm swing frequency, , , The weighting coefficients are preset, all three are positive and satisfy the following conditions: The limb extension speed is obtained by calculating the rate of change of distance between the distal joints of the limbs (hands and feet) and the center point of the torso between adjacent frames. The forward tilt angle of the center of gravity is obtained by the tilt angle of the line connecting the neck joint and the hip joint relative to the vertical direction. The arm swing frequency is obtained by extracting the main frequency component after performing a short-time Fourier transform on the motion velocity sequence of the hand joints.

[0032] In at least one example of the embodiments of this disclosure, motion direction features are further extracted for each human target. , Characterizing the first The overall displacement direction of an individual human target relative to the positions of other human targets in the scene, specifically, The similarity is calculated by the cosine similarity between the displacement vector of the human target within the current atomic action segment and the direction vector pointing from the human target to the average position of all other human targets in the scene. The value range is [value missing]. Positive values ​​indicate movement toward other human targets, while negative values ​​indicate movement away from human targets. This is the movement aggression index. Characteristics of the direction of motion After concatenation, the vectors are mapped to a 3D vector through a fully connected layer and a softmax operation is applied to generate the character probability distribution vector. ,in For the first The probability distribution vector of the role of an individual target. , , The probability values ​​correspond to the attacking, defending, and spectating roles, respectively, and their sum is 1. The fully connected layer uses human target samples with role annotations as the supervision signal, cross-entropy loss as the loss function, and is trained using the Adam optimization algorithm. The dominant role label is determined by the role type corresponding to the maximum probability in the role probability distribution vector. The dominant role label is attached to each action primitive corresponding to the human target, thereby generating a set of action primitive sequences with role annotations.

[0033] It should be noted that the above soft allocation method allows each human target to retain complete role probability distribution information while maintaining the dominant role label. The role probability distribution information is used as a continuous feature in the subsequent determination of edge type and feature splicing of spatiotemporal heterogeneous interaction graph, so that the role allocation result has the ability to smoothly transition to boundary situations (such as a participant alternating between attack and bystander characteristics at different times).

[0034] Step S300: Generate a cross-role action primitive causal constraint graph based on the set of action primitive sequences annotated by the character.

[0035] In step S300, the primitive sequences of different characters in the same violent scene are aligned across roles in time. The co-occurrence relationship and temporal response relationship of primitives between different roles are extracted. The cross-role causal discovery algorithm is used to examine the causal direction and response delay between primitives of different roles, and a cross-role action primitive causal constraint graph is generated.

[0036] For example, for human targets with different dominant role labels in the same violent scene, the action primitive sequences labeled by each role are temporally aligned according to frame-level timestamps, so that the primitive sequences of different roles are arranged correspondingly in a unified time coordinate system. On the aligned multi-role primitive sequences, the co-occurrence frequency matrix and temporal response distribution of primitive types among different roles are statistically analyzed, where the temporal response distribution records the conditional probability of a certain primitive of one role appearing in another role within a specific time window after a certain primitive of one role appears.

[0037] In at least one example of the embodiments of this disclosure, a cross-role causal discovery algorithm is used to perform causal direction checks on each pair of cross-role primitive type combinations. The cross-role causal discovery algorithm takes the occurrence sequence of each of the two role-labeled primitive types in the time dimension as input, and uses a causal inference method based on conditional independence checks to determine whether a directed causal relationship exists between them, the causal direction, and the response delay. The output is a causal direction identifier and a response delay parameter. For adjacent primitives within the same role, the single-role form of the cross-role causal discovery algorithm is used to check the causal preconditioning relationship between primitives within the role.

[0038] It should be noted that the cross-role action primitive causal constraint graph is defined as a directed graph structure, where each node corresponds to a primitive type of role labeling. That is, the same cluster primitive type is regarded as different nodes under different role labels. The cross-role action primitive causal constraint graph contains three types of directed edges: First, intra-role causal precedence edges, connecting two primitive type nodes under the same character label, indicating that the predecessor primitive causally precedes the successor primitive in terms of temporal sequence, such as the attacking character's "charge" primitive causally preceding the "punch" primitive; Second, cross-role response causal edges, connecting two primitive type nodes under different character labels, indicating that a certain primitive of the source character causally triggers a certain primitive response of the target character, with the edge marked with a response delay parameter, such as the attacking character's "punch" primitive causally triggering the defending character's "block" primitive, with a response delay of several frames; Third, mutually exclusive constraint edges, connecting primitive type node pairs that do not have a causal relationship but cannot occur simultaneously at the physical level, such as the victim's "fall down" primitive and the same victim's "run" primitive being marked as mutually exclusive. The cross-role causal discovery algorithm confirms the existence of causal edges when the statistical significance level meets a preset threshold. The preset threshold is the significance level parameter of the conditional independence test. Causal edges are not set between primitive type combinations that fail the test.

[0039] Step S400: Perform role constraint counterfactual intervention on rare role configuration patterns based on the cross-role action primitive causal constraint graph, and generate multiple sets of counterfactual multi-person primitive sequence combinations for role coordination.

[0040] In step S400, a combination of multi-role primitive sequences is extracted from a very small number of samples of rare role configuration patterns. Role constraint counterfactual intervention is performed on the primitive sequence of each role on the cross-role action primitive causal constraint graph. A primitive in a role sequence is replaced with a substitute primitive under the same role label that satisfies the causal constraint. At the same time, it is checked whether the replaced sequence maintains cross-role causal consistency with other role sequences. Replacement schemes that satisfy all constraints are retained.

[0041] For example, for a rare role configuration pattern (e.g., three attacking roles besieging one defending role), extract the role-labeled action primitive sequence for each participant from only 1 to 2 labeled samples. Select a target primitive position in the primitive sequence of a certain role (e.g., attacking role A), and search in the cross-role action primitive causal constraint graph for all candidate alternative primitives that have the same role label as the target primitive and meet the following conditions: There is an intra-role causal preceding edge between the candidate alternative primitive and the target primitive in the cross-role action primitive causal constraint graph, that is, the candidate alternative primitive can be a legitimate causal successor of the predecessor primitive. There is an intra-role causal preceding edge between the candidate alternative primitive and the target primitive in the cross-role action primitive causal constraint graph, that is, the candidate alternative primitive can serve as a legitimate causal preceding edge of the successor primitive. Candidate alternative primitives do not violate the mutual exclusion constraint edge between primitives in the corresponding time window sequence of other roles.

[0042] In at least one example of the embodiments of this disclosure, after the candidate substitution primitives pass the above-mentioned intra-role constraint screening, a cross-role causal consistency check is further performed. Specifically, all cross-role response causal edges with the candidate substitution primitive node as the source node or the target node are retrieved in the cross-role action primitive causal constraint graph, and it is verified whether the replaced sequence satisfies the following conditions at the corresponding time position: if the candidate substitution primitive is the source node of the cross-role response causal edge, then the target primitive type pointed to by the causal edge should exist in the primitive sequence of the target role within the time window specified by the response delay parameter; if the candidate substitution primitive is the target node of the cross-role response causal edge, then the source primitive type of the causal edge should exist in the primitive sequence of the source role within the corresponding pre-existing time window. The replacement scheme is determined to be valid only when the candidate substitution primitive simultaneously satisfies the intra-role causal pre-existing constraint, the mutual exclusion constraint, and the cross-role response causal constraint.

[0043] In another example, for the original primitive sequence combination of the same rare role configuration pattern, the above counterfactual intervention process is performed on multiple different target primitive positions and different role sequences, and the results of multiple independent interventions are combined to ensure that there are no contradictory causal constraint conflicts between the replacement schemes of each position in the same combined sequence combination, and finally multiple sets of role-coordinated counterfactual multi-person primitive sequence combinations are generated.

[0044] Step S500: Generate a set of spatially and temporally coordinated multi-person synthetic skeleton sequences based on the combination of counterfactual multi-person primitive sequences.

[0045] In step S500, for each set of counterfactual multi-person primitive sequence combinations, the primitive sequence of each character is decoded into a skeleton motion sequence using the decoder of the variational autoencoder. During the decoding process, the time alignment of each character's skeleton sequence is adjusted according to the response delay parameter in the cross-character action primitive causal constraint graph. The relative position of each character's root node is constrained according to the prior spatial interaction distance between characters, and the continuity constraint of joint motion is applied.

[0046] For example, for each character sequence in a set of counterfactual multi-person primitive sequences, the primitive vectors corresponding to each primitive in the sequence are arranged in temporal order and input into the decoder of the variational autoencoder trained in step S100. The input of the decoder of the variational autoencoder is the primitive vector sequence, and the output is the temporal matrix of the corresponding skeleton joint coordinates. When decoding the skeleton motion sequence of each character, according to the response delay parameters marked on the cross-character response causal edge in the cross-character action primitive causal constraint graph, a corresponding time offset is applied to the skeleton sequence of each character on the time axis, so that the causally related action segments present an alignment relationship that conforms to the causal response delay in time.

[0047] In at least one example of the embodiments of this disclosure, a priori constraints on inter-role distances are imposed on the synthesized multi-role skeleton sequence in the spatial dimension: based on the statistical distribution of spatial distances between each role participant in the original sample, the relative positions between the root nodes (e.g., hip joints) of each role skeleton are constrained to be within a reasonable range of spatial interaction distances. The range of spatial interaction distances is determined by the mean and standard deviation of the distances between the root nodes of each role participant in the original sample. Specifically, the distances between the root nodes of each role are constrained to fall within the interval defined by the mean plus or minus twice the standard deviation. Furthermore, at the connecting frames of adjacent primitives in the same role sequence, continuity constraints are imposed on the joint positions and joint velocities on both sides of the connecting frame to avoid abrupt changes in skeleton motion caused by primitive replacement. The combined effect of the above-mentioned temporal alignment, spatial distance constraints, and motion continuity constraints generates a set of spatially and temporally coordinated multi-role synthesized skeleton sequences.

[0048] Step S600: Generate a spatiotemporal heterogeneous interaction graph based on the extended sample set.

[0049] In step S600, the multi-person synthetic skeleton sequence set is merged with the original rare configuration sample into an extended sample set. For each human node in each time frame of the extended sample set, the splicing features of the role probability distribution vector, appearance feature vector and skeleton motion feature are extracted. Based on the spatial position relationship and motion direction relationship of each human target, a heterogeneous multi-relationship interaction graph is generated. The graph is connected into a spatiotemporal heterogeneous interaction graph by cross-frame node matching along the time dimension.

[0050] For example, for each time frame of each sample in the expanded sample set, each human target in that frame is treated as a node in the graph, and the node feature is the probability distribution vector of the human target's role in the current frame. The result of concatenating appearance feature vectors and skeletal motion features. Among them, the character probability distribution vector... The range of values ​​for each component is as follows: And the sum is 1. For the index of human targets in the current frame, the appearance feature vector and skeleton motion feature are Z-score normalized before concatenation to ensure that the three types of features are within a comparable numerical range after concatenation. For any two human nodes in the current frame, the relationship type of the edge is determined based on the combination of the dominant role labels of the two nodes. The relationship types include: interaction edges between attacking and defending roles, interaction edges between attacking and spectating roles, interaction edges between defending and spectating roles, and interaction edges between similar roles. The conditions for edge generation are that the spatial distance between the two nodes does not exceed a preset interaction distance threshold, or that the movement directions of the two nodes show a tendency to move towards each other. The preset interaction distance threshold is determined by the upper quantile of the spatial distance distribution between pairs of human targets that actually interact in the original training samples, and the tendency to move towards each other is determined by the cosine value of the angle between the displacement vectors of the two nodes and the direction of the line connecting the two nodes being greater than zero.

[0051] In at least one example of the embodiments of this disclosure, based on a single-frame heterogeneous interaction graph, cross-frame matching is performed on the same human target node in adjacent frames along the time dimension. The cross-frame matching is performed based on the cosine similarity between the appearance feature vectors of the nodes and the continuity of their spatial positions. Temporal connection edges are added between successfully matched cross-frame node pairs. Through the combined effect of the above-mentioned intra-frame heterogeneous relationship edges and cross-frame temporal connection edges, a spatiotemporal heterogeneous interaction graph is formed. The spatiotemporal heterogeneous interaction graph simultaneously encodes the spatial interaction relationships between multiple roles within a single frame and the temporal evolution relationships across frames.

[0052] Step S700: Extract spatiotemporal role interaction feature vectors and calculate meta-learning prototype vectors based on the spatiotemporal heterogeneous interaction graph using a relation type-aware graph attention network.

[0053] In step S700, message passing is performed on the spatiotemporal heterogeneous interaction graph using a relation type-aware graph attention network. Different types of edges are aggregated using independent attention weight calculation heads. The spatiotemporal interaction edge embedding vector sequence is extracted and generated into a spatiotemporal role interaction feature vector through temporal aggregation. In the meta-learning framework, violent interaction prototype vectors and normal interaction prototype vectors are calculated according to the role configuration mode.

[0054] It should be noted that the relation type-aware graph attention network comprises a relation type assignment unit, a multi-head attention aggregation unit, an edge embedding extraction unit, and a temporal aggregation unit. The data transfer relationships between these units are as follows: The relation type assignment unit receives the relation type identifier of each edge in the spatiotemporally heterogeneous interaction graph, assigns the edge to the corresponding attention computation head according to the relation type, and passes the assignment result to the multi-head attention aggregation unit. The multi-head attention aggregation unit contains an equal number of independent attention computation heads as the relation types. Each attention computation head has independent learnable parameters. It receives the source node features and target node features connected to all edges assigned to that attention computation head, calculates the attention weights, and after weighted aggregation of the source node features, outputs updated node features to the edge embedding extraction unit. For nodes... ,node In relation types The updated features after attention aggregation are as follows: in, The index of the target node. For identifying the relation type, For nodes In relation types The updated feature vector after attention aggregation For nodes In relation types The set of neighboring nodes below, For the set of neighboring nodes The traversal index of all nodes in the array. For relation type The corresponding learnable linear transformation matrix of the attention computation head, For relation type Next node For nodes Attention weights For nodes The input feature vector, This is a non-linear activation function. Attention weights. By node With nodes Feature vectors via relation types The corresponding attention computation head's learnable parameter vector is obtained by taking the inner product and then normalizing it using softmax. Specifically: in, For relation type The corresponding learnable parameter vector of the attention computation head, Indicates transpose. This represents a vector concatenation operation. Represents an exponential function. For nodes In relation types The set of neighboring nodes The traversal index of all nodes is calculated. The aggregation results under each relation type are merged into a comprehensive update feature of the nodes through concatenation or summation operations and then passed to the edge embedding extraction unit. The edge embedding extraction unit generates an edge embedding vector for each edge based on the update features of its two endpoints and the relation type of the edge. Specifically, the edge embedding vector is obtained by concatenating the comprehensive update feature vector of the two endpoints of the edge and the learnable type embedding vector of the relation type of the edge, followed by a linear transformation. The edge embedding extraction unit extracts the edge embedding vector sequence of the interaction edges between the attacking and defending roles separately and passes it to the temporal aggregation unit. The temporal aggregation unit receives the extracted edge embedding vector sequence, performs average pooling along the time dimension, and outputs a fixed-dimensional spatiotemporal role interaction feature vector. The relation type-aware graph attention network is trained in a plot-based training manner within the meta-learning framework. Each training plot consists of a support set and a query set. The spatiotemporal role interaction feature vectors of the support set samples are used to calculate the prototype vector. The Euclidean distance between the spatiotemporal role interaction feature vectors and the prototype vectors of the query set samples is normalized by softmax to obtain the classification probability. The cross-entropy loss is used as the loss function, and the Adam optimization algorithm is used for parameter updates.

[0055] It should be noted that the input to the average pooling operation in the temporal aggregation unit is a sequence of edge embedding vectors arranged along the time dimension. Each element in the sequence corresponds to the edge embedding vector of a certain interaction edge between the attacking and defending characters within a time frame. Average pooling takes the average of the edge embedding vectors of all frames along the time frame dimension, thereby compressing the variable-length temporal edge embedding sequence into a fixed-dimensional spatiotemporal character interaction feature vector. The spatiotemporal character interaction feature vector simultaneously encodes the temporal evolution information across frames and the spatial interaction information within a single frame.

[0056] For example, in the meta-learning framework, each sample in the extended sample set is classified according to the role configuration mode (e.g., two-player confrontation mode, multi-player siege mode, armed threat mode, etc.). For violent samples in each role configuration mode, the mean of their spatiotemporal role interaction feature vector is calculated as the violent interaction prototype vector in that mode. Similarly, for normal interaction samples, the mean of their spatiotemporal role interaction feature vector is calculated as the normal interaction prototype vector.

[0057] Step S800: Generate a spatiotemporal character interaction feature vector for the video to be detected and generate a violent behavior detection judgment result based on the distance metric value with the prototype vector.

[0058] In step S800, a multi-person skeleton sequence is extracted from the video to be detected and the probability distribution vector of the characters is calculated. A spatiotemporal heterogeneous interaction graph is generated. A spatiotemporal character interaction feature vector is generated through a relation type-aware graph attention network. The distance metric between the violent interaction prototype vector and the normal interaction prototype vector of each character configuration mode is calculated. A violent behavior detection judgment result is generated based on the distance metric.

[0059] For example, the same human target detection, skeleton extraction, and character probability distribution vector calculation processes as steps S100 to S200 are performed on the video to be detected. The extraction results are then processed using the same spatiotemporal heterogeneous interaction graph generation process as step S600. Finally, after processing by the same relation-type perceptual graph attention network as step S700, the spatiotemporal character interaction feature vector of the video to be detected is obtained. The Euclidean distance between the spatiotemporal character interaction feature vector and the violent interaction prototype vector under each character configuration mode, as well as the Euclidean distance between it and the normal interaction prototype vector, are calculated. When the distance between the spatiotemporal character interaction feature vector and the violent interaction prototype vector of a certain character configuration mode is less than the distance between it and the normal interaction prototype vector, and the absolute value of the difference exceeds a preset judgment threshold, the video to be detected is determined to contain violent interaction behavior. The preset judgment threshold is determined by the distance difference quantile that optimizes the detection accuracy on the validation set.

[0060] In at least one example of the embodiments of this disclosure, when a violent interaction is determined, a detection alarm message is output. The detection alarm message includes: the specific violent type label (the name of the role configuration mode to which the nearest violent interaction prototype vector belongs), the role identifier of each participant in the current frame (determined by the dominant role of the role probability distribution vector), and the spatial position information of each participant in the screen (represented by the bounding box coordinates).

[0061] The machine vision violence detection and analysis system based on small sample meta-learning provided in at least one embodiment of this disclosure includes at least one processor, a video feature extraction unit, a role probability allocation unit, a causal constraint graph generation unit, a counterfactual intervention and synthesis unit, a heterogeneous interaction graph generation unit, a feature aggregation and prototype calculation unit, and a detection and judgment unit.

[0062] The video feature extraction unit is configured to perform the relevant operations of step S100 in the machine vision violence detection and analysis method based on few-shot meta-learning described above. For a detailed explanation of the video feature extraction unit, please refer to the relevant description of step S100 in the embodiments of the machine vision violence detection and analysis method based on few-shot meta-learning described above; it will not be repeated here.

[0063] The role probability allocation unit is configured to perform the relevant operations of step S200 in the machine vision violence detection and analysis method based on few-sample meta-learning described above. For a detailed explanation of the role probability allocation unit, please refer to the relevant description of step S200 in the embodiment of the machine vision violence detection and analysis method based on few-sample meta-learning described above; it will not be repeated here.

[0064] The causal constraint graph generation unit is configured to perform the relevant operations of step S300 in the machine vision violence detection and analysis method based on few-shot meta-learning described above. For a detailed explanation of the causal constraint graph generation unit, please refer to the relevant description of step S300 in the embodiments of the machine vision violence detection and analysis method based on few-shot meta-learning described above; it will not be repeated here.

[0065] The counterfactual intervention and synthesis unit is configured to perform the relevant operations of steps S400 and S500 in the machine vision violence detection and analysis method based on few-sample meta-learning described above. For a detailed explanation of the counterfactual intervention and synthesis unit, please refer to the relevant descriptions of steps S400 and S500 in the embodiments of the machine vision violence detection and analysis method based on few-sample meta-learning described above; they will not be repeated here.

[0066] The heterogeneous interaction graph generation unit is configured to perform the relevant operations of step S600 in the machine vision violence detection and analysis method based on few-shot meta-learning described above. For a detailed explanation of the heterogeneous interaction graph generation unit, please refer to the relevant description of step S600 in the embodiments of the machine vision violence detection and analysis method based on few-shot meta-learning described above; it will not be repeated here.

[0067] The feature aggregation and prototype calculation unit is configured to perform the relevant operations of step S700 in the machine vision violence detection and analysis method based on few-shot meta-learning described above. For a detailed explanation of the feature aggregation and prototype calculation unit, please refer to the relevant description of step S700 in the embodiments of the machine vision violence detection and analysis method based on few-shot meta-learning described above; it will not be repeated here.

[0068] The detection and determination unit is configured to perform the relevant operations of step S800 in the machine vision violence detection and analysis method based on few-sample meta-learning described above. For a detailed explanation of the detection and determination unit, please refer to the relevant description of step S800 in the embodiments of the machine vision violence detection and analysis method based on few-sample meta-learning described above; it will not be repeated here.

[0069] The technical performance of the machine vision violence detection and analysis system based on few-sample meta-learning is the same as that of the machine vision violence detection and analysis method based on few-sample meta-learning described above, and will not be repeated here.

[0070] The following is an example of an application of the present invention, such as Figure 2-9 As shown, the implementation process is as follows: In 20XX, a security monitoring system at a city rail transit hub conducted real-time violent behavior detection on surveillance video from the platform waiting area. The hub's platform area experiences high daily passenger flow, and historical records show multiple instances of coordinated multi-person conflicts, including rare character configurations such as armed threats and group attacks. However, labeled samples of such incidents are extremely scarce, and the existing system's detection capabilities for these scenarios are insufficient. The system is deployed on edge computing nodes, with a video frame rate of 25 frames per second and a resolution of 1920×1080, covering a monitoring area of ​​approximately 200 square meters in the platform waiting area.

[0071] In a video clip triggered by a certain incident, the camera captured an abnormal physical contact event involving four human targets (numbered Human 1 to Human 4) in the waiting area of ​​the platform. The video clip consisted of 120 frames (approximately 4.8 seconds). The target detection network output the bounding box coordinates and appearance feature vectors of each human target for each frame, while the pose estimation network output a sequence of 17 skeleton joint coordinates for each human target. Temporal segmentation was performed on the skeleton motion sequences of the four human targets, dividing the skeleton motion sequences of each target into atomic action segments based on abrupt changes in joint motion speed and direction. These segments were then encoded into semantic action primitive vectors by a variational autoencoder. Finally, clustering was performed on the primitive vectors of all samples to generate a semantic action primitive dictionary containing 12 primitive types. Among these, primitive types relevant to this scene included "charging up," "punching," "blocking," "retreating," "encircling," and "standing still."

[0072] Table 1. Segmentation results of target atomic actions for each human body (examples of segments 1 to 3) In the aforementioned video clip from the platform waiting area, the motion aggression index of the four human targets was calculated within each atomic motion segment. and characteristics of motion direction This process generates a probability distribution vector for the role and determines the dominant role label.

[0073] Taking human body 1 in segment 2 (power-gathering action) as an example, after Z-score standardization, the average limb extension speed is... Mean angle of forward lean of center of gravity arm swing frequency Take weight , , ,but: Human body 1's displacement direction is towards the average direction of the other three human body target positions, and its movement direction characteristics are as follows. ,Will and After concatenation, a fully connected layer and softmax operation are applied to obtain the character probability distribution vector. The dominant role is the attacking role. Human body 4 lower and (The displacement direction is away from other targets), and the corresponding role probability distribution vector indicates that the dominant role is the defensive role.

[0074] Table 2. Probability distribution vectors of target roles for each human character and the dominant role label. The dominant role label was attached to each action primitive of each human target, generating a set of action primitive sequences labeled with the role. Human 1 and Human 2 were labeled as attacking roles, Human 4 as defending roles, and Human 3 as spectator roles. This was consistent with the manual verification results of the on-site monitoring personnel and conformed to the rare role configuration pattern of "two people attacking a single defender and one spectator".

[0075] The action primitive sequences of four human targets are aligned to a unified time coordinate system by frame-level timestamps. The co-occurrence frequency matrix and temporal response distribution between different character primitive types are statistically analyzed. Then, a cross-character causal discovery algorithm is used to perform causal direction verification on each pair of cross-character primitive type combinations.

[0076] In this scenario, the cross-role causality discovery algorithm results show that: the attacker's "charge" primitive has an intra-role causal precondition to the attacker's own "punch" primitive; the attacker's "punch" primitive has a cross-role response causal relationship to the defender's "block" primitive, with a response latency parameter of 8 frames; there is no mutual exclusion constraint between the defender's "backstep" primitive and the defender's "run" primitive, but the defender's "knock down" primitive and the defender's "run" primitive are marked as mutually exclusive edges because they cannot occur simultaneously at the physical level. All the above test results meet the preset significance level threshold (…). ).

[0077] Table 3. Examples of core edges in the causal constraint graph of cross-role action primitives The configuration mode of "two attacking characters besieging a single defensive character" has only two samples in the historical labeled sample library, making the sample size extremely scarce and unable to directly support the computation of the meta-learning prototype. From these two original samples, the character-labeled action primitive sequences of each participant are extracted. The "charging" primitive at position 2 in the sequence of attacking character 1 (human 1) is selected as the target intervention position. Candidate alternative primitives are then searched in the causal constraint graph of cross-character action primitives.

[0078] After filtering based on intra-role causal pre-constraints, candidate replacement primitives must have "encircle" as a valid causal antecedent and "punch" as a valid causal successor. The "pushing" primitive satisfies both intra-role constraints. Further cross-role causal consistency checks are performed: after replacing "accumulate force" with "pushing," cross-role response causal edges with "pushing" as the source node are retrieved. This confirms that the defensive role sequence contains a "retreat" primitive within the corresponding time window, satisfying the cross-role response causal constraints; simultaneously, it confirms that "pushing" does not violate any mutually exclusive constraints with primitives of other roles within the current time window. The replacement scheme passes all constraint checks and is deemed valid. The above process is performed on multiple intervention positions and two attacking role sequences, and a global causal conflict check is performed on the combined results, ultimately generating six sets of counterfactual multi-person primitive sequence combinations for role coordination.

[0079] For six sets of counterfactual multi-person primitive sequences, the primitive vectors in each character sequence are arranged temporally and then input into the decoder of the variational autoencoder. The decoder outputs the temporal matrix of the skeleton joint coordinates of each character. Based on the response delay parameters of the cross-character response causal edge annotation, an 8-frame time offset is applied to align the "punch" action fragment of the attacking character and the "block" action fragment of the defending character.

[0080] In the spatial dimension, based on the statistical distribution of the distance between hip joints of the four participants in the two original samples (mean 1.24 meters, standard deviation 0.19 meters), the distance between the root nodes of each role is constrained to fall within the interval [missing information]. ,Right now Within the context, continuity constraints are applied to the joint positions and velocities at the transition frames between adjacent primitives to eliminate abrupt changes in skeleton motion that may occur at the transition frames when "accumulating power" is replaced by "pushing". Through the combined effect of the above constraints, six sets of spatially and temporally coordinated multi-person synthetic skeleton sequences are generated, which are merged with two original samples to form an extended sample set (a total of eight "dual attack siege defense" configuration mode samples).

[0081] For each time frame of each sample in the expanded sample set, with 4 human targets as nodes, the node features are the probability distribution vectors of the roles. The concatenation result of appearance feature vector (128-dimensional, Z-score normalized) and skeleton motion feature (64-dimensional, Z-score normalized) results in a total dimension of 195. For any two nodes within a frame, the relationship type of the edge is determined based on the combination of the dominant role labels of the two end nodes, and edges are generated based on the spatial distance threshold (determined by the upper quantile of the distance distribution between node pairs that actually interact in the original sample, which is 1.80 meters in this scenario) and the condition that the motion directions are approaching each other.

[0082] Table 4 Examples of edge types in a single-frame heterogeneous interactive graph (taking frame 43 as an example) Cross-frame matching is performed on the same human target node in adjacent frames along the time dimension. The matching is based on the cosine similarity of the node appearance feature vector and the continuity of spatial position. Temporal connection edges are added between successfully matched cross-frame node pairs to form a spatiotemporal heterogeneous interaction graph covering 120 frames. At the same time, the spatial interaction relationship of multiple roles within the frame and the temporal evolution relationship across frames are encoded.

[0083] A relation type-aware graph attention network performs message passing on spatiotemporally heterogeneous interaction graphs. The target node is Human Body 4 (defense character node) in frame 43. In attack-defense relationship types Below, its set of neighboring nodes Includes Human Body 1 and Human Body 2. Attention Weights The updated defense node features are obtained by multiplying the feature vectors of human body 1 and human body 4 by the inner product of the parameter vectors corresponding to the relation type and then normalizing by softmax. The attention weight of human body 1 to human body 4 is 0.61, and the attention weight of human body 2 to human body 4 is 0.39. It is obtained by weighted aggregation and nonlinear activation of the features of two neighboring nodes.

[0084] The edge embedding extraction unit extracts the edge embedding vector sequence of the attack-defense interaction edge (covering all 120 frames) separately and then feeds it into the temporal aggregation unit. The temporal aggregation unit takes the average of the edge embedding vectors of all frames along the time frame dimension and outputs a 128-dimensional spatiotemporal role interaction feature vector.

[0085] In the meta-learning framework, after the samples in the extended sample set are classified according to the role configuration mode, the spatiotemporal role interaction feature vectors of the eight violent samples under the "dual attack siege defense" mode are averaged to obtain the violent interaction prototype vector of this mode; the spatiotemporal role interaction feature vectors of the normal waiting interaction samples (including multi-person platform scenes without attack behavior) are averaged to obtain the normal interaction prototype vector.

[0086] Table 5. Number of samples used for calculating the prototype vector of each role's configuration mode For newly added video segments to be detected (containing 3 human targets, 120 frames) captured by cameras in the waiting area of ​​the platform, perform the same detection, skeleton extraction and role probability distribution vector calculation as steps S100 to S200 to generate a spatiotemporal heterogeneous interaction graph. After processing by a relation type perception graph attention network, a 128-dimensional spatiotemporal role interaction feature vector is obtained.

[0087] The Euclidean distances between this feature vector and each violent interaction prototype vector and the normal interaction prototype vector are calculated, and the results are shown in the table below.

[0088] Table 6. Euclidean distances between the video to be detected and each prototype vector. The Euclidean distance (3.17) between the video to be detected and the violent prototype vector of "dual attack siege defense" is the smallest and less than the distance with the normal interaction prototype vector (8.96). The absolute value of the difference between the two is 5.79, which exceeds the preset judgment threshold (2.50) determined by the validation set. Therefore, the video segment is judged to contain violent interactive behavior. The system then outputs detection alarm information: the violence type label is "dual attack siege defense", the roles of human 1 and human 2 in the current frame are identified as attack roles, and the role of human 3 is identified as a defense role. The spatial position of each participant is represented by the corresponding bounding box coordinates. The alarm information is pushed to the platform security monitoring terminal in real time.

[0089] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A machine vision-based violence detection and analysis method using few-sample meta-learning, characterized in that, include: The process involves acquiring video frame sequences, extracting the appearance feature vectors of human targets and the coordinate sequence of skeleton joints, performing atomic action segmentation and encoding on the skeleton motion sequence, and generating a semantic action primitive dictionary through clustering. The motion aggression index is calculated based on the skeleton motion sequence, and a character probability distribution vector is generated by combining motion direction features. A set of action primitive sequences labeled with the dominant character is generated based on the dominant character label. Cross-role temporal alignment of primitive sequences of different characters in the same violent scene is performed. Causal discovery algorithm is used to examine the causal direction and response delay between primitives of different characters, and a cross-role action primitive causal constraint graph is generated. On the cross-role action primitive causal constraint graph, perform role constraint counterfactual intervention on primitive sequences of rare role configuration patterns to generate a counterfactual multi-person primitive sequence combination for role coordination. The primitive sequences of each character are decoded into skeleton motion sequences and time alignment constraints, spatial distance constraints and motion continuity constraints are applied to generate a set of multi-person synthetic skeleton sequences. The multi-person synthetic skeleton sequence set is merged with the original sample to form an extended sample set, and a spatiotemporal heterogeneous interaction graph is generated based on spatial positional relationship and motion direction relationship. The spatiotemporal heterogeneous interaction graph is used to extract spatiotemporal role interaction feature vectors using a relation type-aware graph attention network, and the violent interaction prototype vector and the normal interaction prototype vector are calculated in the meta-learning framework. as well as Spatiotemporal character interaction feature vectors are generated for the video to be detected, and violent behavior detection results are generated based on the distance metric between the vectors and each prototype vector.

2. The machine vision violence detection and analysis method based on few-sample meta-learning according to claim 1, wherein, The calculation of motion aggression indicators based on skeleton motion sequences, combined with motion direction features to generate a character probability distribution vector, includes: For each human target, within each atomic motion segment of the skeletal motion sequence, the mean limb extension velocity... Mean angle of forward lean of center of gravity Arm swing frequency After Z-score standardization, the motion aggression index was calculated as follows. : ， in, An index for human targets. , , The preset weighting coefficients satisfy the following conditions: ; Extracting motion direction features The By the The cosine similarity between the displacement vector of an individual human target within the current atomic action segment and the direction vector pointing to the average position of all other human targets in the scene is calculated. The aforementioned sports aggression index With the aforementioned motion direction features After concatenation, the vectors are mapped to a 3D vector through a fully connected layer and a softmax operation is applied to generate the character probability distribution vector. ,in , , These correspond to the probability values ​​for attacking, defending, and spectating roles, respectively.

3. The machine vision violence detection and analysis method based on few-sample meta-learning according to claim 1, wherein, The cross-role action primitive causal constraint graph is a directed graph structure, where each node corresponds to a primitive type labeled by a role. The cross-role action primitive causal constraint graph contains three types of directed edges: A causal precedence edge within a character connects two primitive type nodes under the same character tag, indicating that the predecessor primitive causally precedes the successor primitive in temporal order; Cross-role response causal edge connects two primitive type nodes under different role labels, indicating that the primitive of the source role causally triggers the primitive response of the target role, and the edge is marked with the response delay parameter; Mutually exclusive edges connect primitive type nodes that cannot occur simultaneously at the physical level. The causal discovery algorithm confirms the existence of a causal edge when the statistical significance level meets a preset threshold, where the preset threshold is the significance level parameter of the conditional independence test.

4. The machine vision violence detection and analysis method based on few-sample meta-learning according to claim 3, wherein, The role constraint counterfactual intervention includes: Select the target primitive position in the primitive sequence of the target character, and search for candidate alternative primitives with the same character label as the target primitive and simultaneously satisfying the following conditions in the cross-character action primitive causal constraint graph: There is an in-role causal pre-edge between the candidate alternative primitive and the predecessor primitive of the target primitive; There is an in-role causal front edge between the candidate alternative primitive and the successor primitive of the target primitive; The candidate alternative primitives do not violate the mutual exclusion constraint edge between primitives in the primitive sequences of other roles within the corresponding time window.

5. The machine vision violence detection and analysis method based on few-sample meta-learning according to claim 4, wherein, After the candidate alternative primitives are filtered through intra-role constraints, a cross-role causal consistency test is further performed, including: Retrieve all cross-role response causal edges with the candidate substitution primitive node as the source node or target node in the cross-role action primitive causal constraint graph. If the candidate alternative primitive is the source node of a cross-role response causal edge, then verify that the target role's primitive sequence contains the target primitive type pointed to by the causal edge within the time window specified by the response delay parameter. If the candidate alternative primitive is the target node of the cross-role response causal edge, then verify that the source primitive type of the causal edge exists in the corresponding preceding time window in the primitive sequence of the source role. The replacement scheme is deemed valid only if the candidate alternative primitive simultaneously satisfies the intra-role causal precondition, mutual exclusion constraint, and cross-role response causal constraint.

6. The machine vision violence detection and analysis method based on few-sample meta-learning according to claim 1, wherein, The generation of spatiotemporal heterogeneous interaction graphs based on spatial positional relationships and motion direction relationships includes: Each human target in each time frame is treated as a graph node. The node features are the concatenation results of the human target’s role probability distribution vector, appearance feature vector and skeleton motion features after Z-score normalization. For any two human body nodes in the current frame, the relationship type of the edge is determined based on the combination of the dominant role labels of the two end nodes. The relationship type includes the interaction edge between the attacking role and the defending role, the interaction edge between the attacking role and the spectator role, the interaction edge between the defending role and the spectator role, and the interaction edge between the same type of roles. The conditions for generating an edge are that the spatial distance between the two endpoints does not exceed a preset interaction distance threshold, or the movement directions of the two endpoints show a tendency to move towards each other. Cross-frame matching is performed on the same human target node in adjacent frames along the time dimension based on the cosine similarity of appearance feature vectors and spatial position continuity. Temporal connection edges are added between successfully matched cross-frame node pairs.

7. The machine vision violence detection and analysis method based on few-sample meta-learning according to claim 1, wherein, The relation type-aware graph attention network for nodes In relation types The following features are updated as follows: ， in, For nodes In relation types The updated feature vector is as follows. For nodes In relation types The set of neighboring nodes below, For relation type The corresponding learnable linear transformation matrix, For nodes The input feature vector, Non-linear activation function; attention weights The calculation is as follows: ， in, For relation type The corresponding learnable parameter vector, This represents a vector concatenation operation; After the aggregation results of each relation type are fused, each edge is concatenated with the updated features of its two end nodes and the learnable type embedding vector of the relation type, and then linearly transformed to generate an edge embedding vector. The edge embedding vector sequence of the interaction edges between the attacking role and the defending role is averaged along the time dimension to output the spatiotemporal role interaction feature vector.

8. The machine vision violence detection and analysis method based on few-sample meta-learning according to claim 1, wherein, The step of generating a violent behavior detection result based on the distance metric between each prototype vector includes: Calculate the Euclidean distance between the spatiotemporal character interaction feature vector of the video to be detected and the violent interaction prototype vector under each character configuration mode, as well as the Euclidean distance with the normal interaction prototype vector. When the distance between the spatiotemporal role interaction feature vector and the violent interaction prototype vector of a certain role configuration mode is less than the distance between the violent interaction prototype vector and the normal interaction prototype vector, and the absolute value of the difference between the two exceeds the preset judgment threshold, the video to be detected is determined to contain violent interaction behavior. Output detection alarm information, which includes the determined violence type label, the role identifier of each participant, and the spatial position information of each participant in the picture.

9. A machine vision violence detection and analysis system based on few-sample meta-learning, used to execute the machine vision violence detection and analysis method based on few-sample meta-learning as described in any one of claims 1 to 8, characterized in that, include: The video feature extraction unit is configured to acquire video frame sequences, extract the appearance feature vectors and skeleton joint coordinate sequences of human targets, perform atomic action segmentation and encoding on the skeleton motion sequence, and generate a semantic action primitive dictionary through clustering. The role probability allocation unit is configured to calculate the motion aggression index based on the skeleton motion sequence, generate the role probability distribution vector by combining motion direction features, and generate a set of action primitive sequences labeled with the dominant role tag. The causal constraint graph generation unit is configured to perform cross-role temporal alignment of primitive sequences of different roles in the same violent scene, use causal discovery algorithm to examine the causal direction and response delay between primitives of different roles, and generate a cross-role action primitive causal constraint graph. The counterfactual intervention and synthesis unit is configured to perform role constraint counterfactual intervention on primitive sequences of rare role configuration patterns on the cross-role action primitive causal constraint graph, generate a combination of counterfactual multi-person primitive sequences for role coordination, decode each role primitive sequence into a skeleton motion sequence and apply time alignment constraints, spatial distance constraints and motion continuity constraints to generate a set of multi-person synthesized skeleton sequences. The heterogeneous interaction graph generation unit is configured to merge the multi-person synthetic skeleton sequence set with the original sample into an extended sample set, and generate a spatiotemporal heterogeneous interaction graph based on spatial positional relationship and motion direction relationship. The feature aggregation and prototype calculation unit is configured to extract spatiotemporal role interaction feature vectors from the spatiotemporal heterogeneous interaction graph using a relation type-aware graph attention network, and to calculate violent interaction prototype vectors and normal interaction prototype vectors in a meta-learning framework. The detection and judgment unit is configured to generate spatiotemporal role interaction feature vectors for the video to be detected, and generate violent behavior detection and judgment results based on the distance metric between the vectors and each prototype vector.

Images