Method and device for behavior recognition based on self-supervised neural network of dominant game

Abstract

The embodiment of the application relates to the technical field of artificial intelligence, and discloses a behavior recognition method of a self-supervised neural network based on a dominant game and related devices, the method comprising: obtaining visual data to be recognized; inputting the visual data to be recognized into a self-supervised neural network based on a dominant game to obtain a behavior recognition result; the self-supervised neural network based on the dominant game comprising a key region division module based on spatiotemporal double-weight collaborative positioning, a partition data enhancement module based on a double-scale game, and a replacement queue based on an entropy-driven dominant game; the self-supervised neural network based on the dominant game being obtained through training according to preset sample data; the preset sample data being a skeleton behavior data set; the skeleton behavior data set comprising positive sample data and negative sample data; and outputting the behavior recognition result. The application achieves the effect of significantly improving recognition accuracy, generalization ability, and training efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a behavior recognition method and related apparatus based on a self-supervised neural network of dominant game theory. Background Technology

[0002] Currently, contrastive learning, as an important branch of self-supervised learning, has shown great potential in the field of skeleton behavior recognition due to its intuitive learning mechanism and excellent feature extraction capabilities. Its core idea is to drive the model to learn more discriminative features through constructed "positive sample pairs" and "negative sample pairs." However, current algorithms suffer from problems in sample construction and management, such as the inaccurate optimization of the quality and value of positive and negative samples, leading to insufficient effectiveness in model feature learning and limited ability to optimize the discrimination boundary. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention provide a behavior recognition method, apparatus, computer device and storage medium based on dominant game theory and self-supervised neural network, which is used to solve the problem that positive samples are prone to semantic distortion or information dilution, making it difficult to simultaneously meet the requirements of high semantic integrity and high information diversity, thus restricting the learning efficiency of the model on the core features of the action.

[0004] According to one aspect of the present invention, a behavior recognition method based on a self-supervised neural network of dominant game theory is provided, the method comprising: Acquire the visual data to be recognized; The visual data to be identified is input into a self-supervised neural network based on dominant game theory to obtain behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data. The preset sample data is a skeleton behavior dataset, which includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data; the partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees; the replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, and selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, which is then used as the negative sample set for training. Output the behavior recognition result.

[0005] In an optional approach, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions according to the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data, further including: Calculate the difference and jointness of each skeleton node in the positive sample data; The spatiotemporal composite weight of the skeleton node is obtained by fusing the difference degree and the joint degree. Based on the spatiotemporal composite weight and the skeleton node key features, a key action region mask is generated. The key motion regions and non-key motion regions of the positive sample data are determined based on the key motion region mask.

[0006] In an alternative approach, calculating the dissimilarity and articulation of each skeleton node in the positive sample data further includes: Calculate the global statistical baseline pose based on the skeleton behavior dataset; The difference of each node is calculated based on the negative gradient of the cosine similarity between the global statistical baseline pose and each skeleton behavior data in the input skeleton behavior data with respect to the node features. Based on the adjacency matrix of the undirected graph corresponding to each skeleton behavior data in the skeleton behavior data, the number of connecting edges of each node in each skeleton behavior data is counted, and the joint degree of each node is calculated after normalizing the number of connecting edges.

[0007] In one alternative approach, the dual-scale game-based partitioned data augmentation module enhances the key motion regions and non-key motion regions in the positive sample data to different degrees, further including: Based on the key motion region, style transfer is performed on nodes with joint degree greater than the first joint degree threshold in each positive sample data to generate style-transferred positive sample data. Data augmentation of key and non-key motion regions in the style-transferred positive sample data was performed to different degrees on two scales: similarity and dissimilarity, to obtain the data-augmented positive sample dataset.

[0008] In one alternative approach, based on the key motion region, style transfer is performed on nodes in each positive sample data whose joint degree is greater than a first joint degree threshold to generate style-transferred positive sample data, including: The following formula will be used to calculate the first... j Positive samples The style migration to the first i Positive samples middle: ; in, These are positive samples after style transfer. Positive samples nodes The degree of joint movement, and Positive samples respectively and positive samples The spatial mean, and Samples and The time variance.

[0009] In one alternative approach, data augmentation is applied to key and non-key motion regions of the style-transferred positive sample data to varying degrees, based on both similarity and dissimilarity scales, resulting in a data-augmented positive sample dataset, including: Based on the payoff function of the two-scale dominant game, the target strong data augmentation strategy used in the key motion regions of the positive sample data after style transfer, and the target ordinary data augmentation strategy used in the non-key motion regions are determined; the target strong data augmentation strategy is one of a plurality of strong data augmentation strategies, and the target ordinary data augmentation strategy is one of a plurality of ordinary data augmentation strategies. The payoff function of the two-scale dominant game is: ; in, For key motion regions, the mask coefficients are... =1, otherwise the mask coefficient Set to 0; Indicates the first i Nodes of positive samples Use enhancement strategies Similarity score at time Indicates the first iNodes of positive samples Use enhancement strategies Dissimilarity score at time; when At that time, take ;when At that time, take .

[0010] Based on the target strong data augmentation strategy, the key motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented key motion regions. Based on the target general data augmentation strategy, the non-critical motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented non-critical motion regions. The data-augmented key motion regions and the data-augmented non-key motion regions are stitched together to obtain the data-augmented positive sample.

[0011] In one alternative approach, the replacement queue based on entropy-driven dominant game uses the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, calculates the similarity normalized probability of each negative sample memory, selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, and uses the optimal negative sample memory as the training negative sample set, including: The payoff function for an entropy-driven dominant game can be calculated using the following formula: ; Among them, for dominant game The set of strategies in a game is ; For the negative sample memory size, The strategy represents replacing samples in the negative sample memory with the current sample. ; This represents the normalized probability of similarity. The normalized probability of similarity is expressed as: ; in, The cosine similarity between two negative samples is represented by... and Indicates negative samples Negative samples The negative sample features in the replaced negative sample memory, They represent the first Positive sample features during the training iteration.

[0012] According to another aspect of the present invention, a behavior recognition device based on a self-supervised neural network of dominant game theory is provided, comprising: The acquisition module is used to acquire the visual data to be recognized; The behavior recognition module is used to input the visual data to be recognized into a self-supervised neural network based on dominant game theory to obtain the behavior recognition result. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data augmentation module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data. The preset sample data is a skeleton behavior dataset. The skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data; the partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees; the replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, and selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, which is then used as the negative sample set for training. The output module is used to output the behavior recognition results.

[0013] According to another aspect of the present invention, a computer device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform the operation of the behavior recognition method based on the dominant game-theoretic self-supervised neural network.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored therein, which, when executed on a computer device, causes the computer device to perform the operation of the behavior recognition method based on the dominant game-theoretic self-supervised neural network.

[0015] This invention provides an embodiment that acquires visual data to be identified; inputs the visual data to be identified into a self-supervised neural network based on dominant game theory to obtain behavior recognition results; and outputs the behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data; the preset sample data is a skeleton behavior dataset; the skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data. The partitioned data enhancement module based on dual-scale game theory enhances the key and non-key motion regions in the positive sample data to different degrees. The replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory using the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, and selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, which is then used as the negative sample set for training. In this way, based on the "dominant game" theory in game theory, a two-dimensional dominant system of "positive sample optimization - negative sample selection" is constructed. By accurately locating key regions and enhancing positive sample quality through partitioned differentiation, and by using an entropy priority queue to select high-value, difficult negative samples to strengthen constraints, while integrating a contrastive loss optimization module to guide model learning, the accuracy, generalization ability, and training efficiency of skeleton behavior recognition are significantly improved.

[0016] Furthermore, by calculating the difference between the sample and the global statistical baseline posture through the key region segmentation module based on spatiotemporal dual-weight collaborative localization, and combining the joint degree calculated by the joint degree weighted difference activation module, the core motion joint group is located from the perspectives of difference degree and joint degree. Relying on the skeleton partitioning data enhancement module based on dual-scale game theory, strong enhancement is applied to the key region to preserve the action semantics, and weak enhancement is applied to the non-key region to maintain the structural features. This achieves the synergistic optimization of high information content and strong representativeness of positive samples, improves the sample semantics and enriches the diversity of features.

[0017] Furthermore, using the entropy value of the negative sample memory after sample replacement as the profit objective, the dominant strategy in game theory is deeply coupled with the screening of negative sample value. The degree of dominance of negative samples is quantified by information entropy to obtain dominant negative samples. Dominant negative samples are retained first to maintain the strong constraint of the queue, while inferior samples are eliminated. The management of negative samples is changed from indiscriminate treatment to dynamic screening based on dominant value, ensuring that the model always obtains the most critical comparison signal for improving feature discrimination ability.

[0018] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0019] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the behavior recognition method based on a self-supervised neural network using dominant game theory provided in an embodiment of the present invention is shown. Figure 2 The diagram shows the structure of the self-supervised neural network in the behavior recognition method based on dominant game theory provided in this embodiment of the invention. Figure 3 This diagram illustrates the structure of the spatiotemporal dual-weight collaborative localization algorithm in the behavior recognition method based on dominant game theory and self-supervised neural networks provided in this embodiment of the invention. Figure 4 This diagram illustrates style transfer in the behavior recognition method based on dominant game theory and self-supervised neural network provided in an embodiment of the present invention. Figure 5 A schematic diagram of the structure of the remote sensing target detection device provided in an embodiment of the present invention is shown; Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0020] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0021] Figure 1 The flowchart illustrates a behavior recognition method based on a self-supervised neural network using dominant game theory, provided in an embodiment of the present invention. This method is executed by a computer device. The computer device can be a desktop computer, tablet computer, laptop computer, smart terminal, distributed device, wearable device, etc., and the embodiments of the present invention do not impose specific limitations. Figure 1 As shown, the method includes the following steps: Step 110: Obtain the visual data to be recognized.

[0022] The visual data to be identified is video data or video frame data. Specifically, the data modalities of the visual data to be identified include RGB video, depth sequences, skeleton sequences, and infrared data, covering content such as daily behaviors, health-related actions, and two-person interactions.

[0023] Step 120: Input the visual data to be identified into a self-supervised neural network based on dominant game theory to obtain the behavior recognition result.

[0024] Among them, such as Figure 2 As shown, the self-supervised neural network based on dominant game theory includes a key region partitioning module (DW-KRM) based on spatio-temporal-dual-weight synergistic localization, a partitioned data augmentation module (DGA) based on dual-scale game theory, and an entropy-driven dominance game replacement queue (EDGRQ). In this embodiment, the self-supervised neural network based on dominant game theory follows the MoCov2 learning paradigm, using game theory's dominant game theory as its core to construct a two-dimensional dominant system of positive sample optimization and negative sample selection. In the positive sample construction stage, the dual-scale dominant game based on the spatio-temporal-dual-weight synergistic localization algorithm includes a key region partitioning module based on spatio-temporal-dual-weight synergistic localization and a partitioned data augmentation module based on dual-scale game theory. More targeted data augmentation is set according to the optimal strategy of the game, exploring richer motion patterns while maintaining the semantics of the samples. In the negative sample management stage, the replacement queue based on entropy-driven dominant game theory prioritizes retaining high-entropy, difficult-to-negative samples and eliminating low-entropy, inferior samples according to the optimal strategy of the game, providing a strong constraint comparison signal for the model.

[0025] The self-supervised neural network based on dominant game theory is trained using preset sample data. This preset sample data is a skeleton behavior dataset. The skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. Specifically, the skeleton behavior dataset is based on large-scale human behavior recognition resources using RGB+D technology. The skeleton behavior dataset covers 60 different action categories, containing 56,578 action sequences, recorded using three Microsoft Kinect v2 cameras, and provides four data modalities: RGB video, depth sequence, skeleton sequence, and infrared data, covering daily behaviors, health-related actions, and two-person interactions. The dataset includes data collected from 40 subjects using Kinect v2 sensors from three different perspectives. The dataset provides two standard partitioning schemes: cross-subject (X-sub) and cross-view (X-view). X-sub divides the 40 subjects into a training group (1-20) and a test group (21-40), while X-view uses data from two perspectives as the training set and the remaining perspectives as the test set. This dataset reflects individual differences and perspective variations, providing a comprehensive and diverse dataset foundation for behavior recognition algorithms. Furthermore, the skeleton behavior dataset includes action sequences recorded by 66 subjects, covering 51 action types, with a total of 20,734 samples, including 1,074 independent action instances. Under the C-view protocol, the dataset is divided according to the camera's perspective: the training set consists of samples captured by cameras from the center and right perspectives, while the test set consists of samples from cameras from the left perspective.

[0026] During iterative training, the key region segmentation module based on spatiotemporal dual-weight co-localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data. The partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees. The replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, and selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, which is then used as the negative sample set for training. First, the key region segmentation module based on spatiotemporal dual-weight co-localization calculates the Global Statistical Benchmark Pose (GSBP), and then calculates the GSBP based on the GSBP and the input skeleton behavior data. The cosine similarity between nodes is used to calculate the negative gradient of node features, resulting in the Discrepancy-Degree (DD) for each node. Next, in the Joint-Degree Activation Module (JDAM), the number of edges connecting each node is counted based on the adjacency matrix of the skeleton undirected graph. After normalization, the Joint Degree (JD) of each node is calculated. The spatiotemporal composite weight is derived from DD and JD. By combining node key features, key motion regions that determine the basic properties of motion are located. Then, in the DGA module, based on the key motion regions located by DW-KRM, style transfer is first performed on nodes with high JD. Then, based on two scales—similarity and dissimilarity—different degrees of data augmentation are applied to key and non-key motion regions respectively, enriching the diversity of samples while preserving semantics. Finally, in the replacement queue based on entropy-driven dominant game, a dominant game is conducted with entropy as the game utility and dominant strategy as the convergence mechanism to enhance the value of negative samples.

[0027] In self-supervised contrastive learning, positive samples need to possess high information content and strong representativeness. High information content requires positive samples to contain rich, fine-grained motion information, while strong representativeness requires positive samples to accurately reflect the core features of the action category based on the key nodes of the skeleton. Therefore, based on this problem, this paper proposes a spatiotemporal dual-weight cooperative localization algorithm. i Positive samples After inputting the key region segmentation module based on spatiotemporal dual-weight collaborative localization, the key action region mask is generated by fusing two types of weights: the difference degree (DD) of nodes and the joint degree (JD) calculated in the joint degree weighted difference activation module (JDAM). This enables adaptive focusing on key motion areas.

[0028] In one embodiment of the present invention, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions according to the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data, further comprising: Step 001: Calculate the difference and articulation of each skeleton node in the positive sample data.

[0029] The calculation of the dissimilarity and articulation of each skeleton node in the positive sample data further includes: calculating a global statistical baseline pose based on the skeleton behavior dataset; calculating the dissimilarity of each node based on the negative gradient of the cosine similarity between the global statistical baseline pose and each skeleton behavior data in the input skeleton behavior data with respect to node features; and calculating the articulation of each node based on the adjacency matrix of the skeleton undirected graph corresponding to each skeleton behavior data in the skeleton behavior data, and normalizing the number of connecting edges.

[0030] Specifically, such as Figure 3 As shown, to facilitate the subsequent explanation of the spatiotemporal dual-weight cooperative localization algorithm, we will first introduce the definitions of global statistical reference pose, difference degree, and joint degree: Global Statistical Benchmark Pose (GSBP) definition: Given a skeleton behavior dataset... ,in The total number of positive samples in the positive sample data. , They represent the first i Positive samples The size of the channels, frames, and joint sets determines the Global Statistical Baseline Pose (GSBP). The algebraic average of the dataset along the sample dimensions, i.e.: ; in, is the Global Statistical Baseline Pose (GSBP), and N is the total number of positive samples in the dataset; For the first i One positive sample.

[0031] Definition of Discrepancy-Degree (DD): Let... and Positive samples and Global Statistical Reference Attitude (GSBP) Keyed encoders and multilayer perceptron The obtained features, their similarity Represented as: ; in, Let L be the L-2 norm of the eigenvectors. one sample node Difference For similarity For nodes Key encoder After encoding, the channel Key features The negative derivative of the difference is used to calculate the degree of difference, as shown in the following formula: ; in, They represent the degree of difference, respectively. The size of the channel, frame, and joint sets, among which of and They are consistent; no distinction will be made here. yes The Yuan.

[0032] Joint-degree (JD) is defined as follows: Given a skeleton behavior dataset... No. The first sample The undirected skeleton graph of the frame is for The set of nodes, for The edge set, and the corresponding adjacency matrix is ,in If the total number of elements is equal to the number of nodes, then define the node. The degree is as follows: ; Among them, when node and When there are edges connecting them, ,otherwise . Node joint degree Define as a node degree The normalized value, i.e.: ; The three definitions above together form the basis of the spatiotemporal dual-weight cooperative positioning algorithm, providing spatiotemporal dual constraints for subsequent regional positioning.

[0033] Step 002: Combine the difference degree and the joint degree to obtain the spatiotemporal composite degree weight of the skeleton node.

[0034] First, the first Positive samples and GSBP Input key encoder and multilayer perceptron , to obtain features and Calculate the sample difference .

[0035] Then, the difference calculated above is used. Input a partitioned data augmentation module based on a two-scale game theory approach, and calculate the jointness weight vector. Its weight That is The difference was calculated. and joint degree Define the spatiotemporal composite weight To measure neurons Importance, spatiotemporal composite weight for: ; in, For activation function, They represent The size of the frame and joint set.

[0036] Step 003: Generate a key action region mask based on the spatiotemporal composite weight and the skeleton node key features.

[0037] Finally, a weighted combination of the composite weight and features is calculated, activated, and multiplied by the importance adjacency matrix to generate a key action region mask, represented as: ; in, This is the adjacency matrix of the skeleton behavior data, used for importance smoothing; it is pre-defined in the dataset. Through this linear combination, features that negatively impact similarity are selected, including key action region masks. Exceeding the importance threshold The area designated as the critical action area is the area designated as the critical action area, while the remaining area is the non-critical area.

[0038] Step 004: Determine the critical motion regions and non-critical motion regions of the positive sample data based on the critical motion region mask.

[0039] Among them, after obtaining the mask of the critical action region, the critical action region is Exceeding the importance threshold The key area is the area that is not critical, while the remaining area is the non-critical area.

[0040] Spatiotemporal composite weight From the formula, the spatiotemporal composite weight Both the interpretation of the pipe Joint flexibility was also taken into consideration. This approach explores nodes with higher dissimilarity in the skeleton from a temporal perspective, and focuses on nodes with higher articulation from a spatial perspective, i.e., the topological structure of the skeleton itself. The weight distribution of key regions is determined collaboratively by the weights of dissimilarity and articulation.

[0041] From a game theory perspective, DW-KRM can be viewed as a locally dominant game. In this game, each skeleton node is a player whose strategy is to maximize dissimilarity over time. Joint degree with spatial dimension Attention weights are allocated among nodes to maximize their information gain in the global representation. The spatiotemporal composite weight of nodes is also considered. It can be viewed as its payoff function in the game. The competition and cooperation among all nodes regarding weight allocation constitute a non-cooperative dominant game system. Ultimately, the weight distribution among nodes converges to a locally dominant equilibrium, meaning that without changing the weights of other nodes, no node can gain a higher return by adjusting its own strategy. This game-like feature selection mechanism gives DW-KRM adaptability and robustness when extracting key regions.

[0042] Step 005: After determining the key motion region, adjust the positive sample data according to the key motion region. Nodes with joint degree greater than the first joint degree threshold are subjected to style transfer, generating positive sample data after style transfer.

[0043] In order to enhance the semantic rationality and diversity of the samples, the embodiments of the present invention will include the first... i sample Style transfer to samples In the middle. Adjust the joint degree. Higher nodes are considered as sample content, often focusing on dynamic information strongly correlated with action categories, and incorporating joint degrees. Lower-level nodes are considered as the style of the sample, corresponding only to global structural information unrelated to action semantics. Another sample, different from the input model's samples, is randomly selected. samples Calculate separately and Spatial mean , and time variance .like Figure 4 As shown, it specifically includes: The following formula will be used to calculate the first... j Positive samples The style migration to the first i Positive samples middle: ; in, These are positive samples after style transfer. Positive samples nodes The degree of joint movement, and Positive samples respectively and positive samples The spatial mean, and Samples and The time variance.

[0044] Step 006: Perform different degrees of data augmentation on the key motion regions and non-key motion regions of the style-transferred positive sample data from two scales: similarity and dissimilarity, to obtain the data-augmented positive sample dataset.

[0045] Specifically, the key motion regions of the positive sample data after style transfer are augmented according to the target strong data augmentation strategy to obtain the data-augmented key motion regions; the non-key motion regions of the positive sample data after style transfer are augmented according to the target ordinary data augmentation strategy to obtain the data-augmented non-key motion regions; and the data-augmented key motion regions and the data-augmented non-key motion regions are spliced ​​together to obtain the data-augmented positive sample.

[0046] In this embodiment of the invention, based on the payoff function of a two-scale dominant game, a targeted strong data augmentation strategy for key motion regions of the style-transferred positive sample data and a targeted ordinary data augmentation strategy for non-key motion regions are determined. The targeted strong data augmentation strategy is one of multiple strong data augmentation strategies, and the targeted ordinary data augmentation strategy is one of multiple ordinary data augmentation strategies.

[0047] Specifically, after style transfer, although the samples maintain semantic consistency, they still lack strategic optimization of action patterns. Therefore, this embodiment of the invention further introduces a dual-scale dominant game mechanism to achieve adaptive selection of enhanced strategies. The game modeling is described as follows: Set up a single-player game The players in this module are key area data enhancement modules. In order to provide different enhancement methods, in one specific embodiment of the present invention, eight strong data augmentation strategies are specifically designed. Compared with two common data augmentation methods The strategies are listed below: (1) Strong data augmentation strategy Four spatial data transformations (shearing, spatial flipping, rotation, axis masking), two temporal data transformations (cropping, time flipping), and two spatiotemporal data transformations (Gaussian noise, Gaussian blur). (2) Common data augmentation strategies Add Gaussian noise with large variance and skeleton mixing.

[0048] The goal of the game is to use strong data augmentation strategies as much as possible in key motion areas, while reducing the similarity between the samples before and after augmentation to explore more motion patterns, and to maintain the similarity between the samples before and after augmentation as much as possible in non-key motion areas, so as to preserve semantics to the greatest extent.

[0049] The payoff function of the two-scale dominant game is: ; in, For key motion regions, the mask coefficients are... =1, otherwise the mask coefficient Set to 0; Indicates the first i Nodes of positive samples Use enhancement strategies Similarity score at time Indicates the first i Nodes of positive samples Use enhancement strategies Dissimilarity score at time; when At that time, take ;when At that time, take .

[0050] Regulation: , And always set .when At that time, take ;when At that time, take , It can always strictly achieve the maximum value, therefore, the strategy It is a game One of the dominant strategies, in which This indicates the strategy used by key area nodes. This indicates the strategy used by non-critical area nodes.

[0051] For mask coefficients In the region, we analyzed the style-transferred samples. Strengthening data augmentation to obtain samples For mask coefficients In the region, we sample Implement general data augmentation, Obtaining samples The two augmented samples are concatenated to obtain the augmented positive sample: ; in, This represents the final augmented data. Given a binary mask matrix, if the skeleton... No. Frame key points Joint importance weight Exceeding the importance threshold Then the element Set to 1, otherwise Set to 0, It is the Hadamard product of a matrix, that is, the element-wise multiplication of a matrix.

[0052] Style transfer improved the sample Semantic diversity and rationality; by employing different data augmentation strategies for different regions, the sample's... The semantics were analyzed, and more behavioral patterns were explored. Meanwhile, due to game theory... There exists a unique dominant strategy ,therefore It is a dominant game, and as long as ,Strategy The dominant strategy is always maintained, which demonstrates the strong robustness of the dominant strategy proposed in this paper.

[0053] In contrastive learning, traditional methods typically employ a first-in, first-out (FIFO) strategy for updates, treating all negative samples equally. However, this approach fails to fully explore the differences between negative samples, especially those with higher difficulty. Therefore, after completing local and global enhancement of positive sample representations, the model needs to maintain a dynamic equilibrium of information distribution at the negative sample construction level. This invention, based on an entropy-driven dominant game replacement queue (EDGRQ), optimizes negative sample quality at the sample strategy level, ensuring that DoGCLR always uses the optimal negative samples to train the model in each iteration. Specifically, the EDGRQ uses the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, calculates the normalized similarity probability of each negative sample memory, selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, and uses this optimal negative sample memory as the training negative sample set, including: The payoff function for an entropy-driven dominant game can be calculated using the following formula: ; Among them, for dominant game The set of players in the game is This indicates that the game has only one player in the negative sample memory bank; the game's strategy set is... ; For the negative sample memory size, The strategy represents replacing samples in the negative sample memory with the current sample. ; This represents the normalized probability of similarity.

[0054] The normalized similarity probability is expressed as: ; in, The cosine similarity between two negative samples is represented by... and Indicates negative samples Negative samples The negative sample features in the replaced negative sample memory, They represent the first Positive sample features during the next training iteration. The functional characteristics, when hour, For function The global optimal value is found, and the game converges to the dominant strategy at this point.

[0055] Therefore, it can be guaranteed that in the first... In each training iteration, the negative samples in the negative sample memory are... There is always the greatest uncertainty, which avoids the quality fluctuations of the queue caused by indiscriminate processing in the traditional FIFO strategy, and maximizes the utilization efficiency of high-value negative samples. This allows the model to always focus on the key discrimination task during the contrastive learning process, and significantly enhances the discriminativeness and generalization of feature representation.

[0056] This invention addresses the problems of existing contrastive learning methods, such as loss of motion semantic information, lack of effective enhancement strategies, and poor negative sample selection due to uniform processing of different regions of the skeleton sequence. It proposes a self-supervised contrastive learning neural network for skeleton behavior recognition (DoGCLR) based on dominant game theory. First, a spatiotemporal dual-weight localization algorithm is used to accurately locate key motion regions using two temporal and spatial weights: difference degree and node degree. Second, a dual-scale dominant game theory approach is used to apply adaptive normalized style transfer enhancement to key regions and ordinary data enhancement to non-key regions, maximizing the exploration of more motion patterns while preserving action semantics. Finally, a negative sample replacement queue based on entropy-driven dominant game theory is designed. The information entropy of different negative sample memories is used as the evaluation criterion, retaining the negative sample memory with the highest entropy value to replace the FIFO queue in the original work. This allows the model to learn sample features using more challenging negative samples. Experiments demonstrate the superiority of the DoGCLR algorithm over earlier algorithms, and ablation experiments verify the effectiveness of the proposed module. In summary, the algorithm of this invention steadily improves the quality of positive samples and the supply of difficult negative samples, strengthens the modeling capability of key motion regions, and enables subsequent motion region localization algorithms to further improve the accuracy of behavior recognition algorithms.

[0057] After training the self-supervised neural network based on dominant game theory in the above manner, the visual data to be identified is input into the self-supervised neural network based on dominant game theory to obtain the behavior recognition result. The behavior recognition result can be behavior types such as daily behavior, health-related actions, and two-person interactions.

[0058] Step 130: Output the behavior recognition result.

[0059] After obtaining the behavior recognition results, the behavior recognition results are output and displayed to the user.

[0060] This invention provides an embodiment that acquires visual data to be identified; inputs the visual data to be identified into a self-supervised neural network based on dominant game theory to obtain behavior recognition results; and outputs the behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data; the preset sample data is a skeleton behavior dataset; the skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data. The partitioned data enhancement module based on dual-scale game theory enhances the key and non-key motion regions in the positive sample data to different degrees. The replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory using the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, and selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, which is then used as the negative sample set for training. In this way, based on the "dominant game" theory in game theory, a two-dimensional dominant system of "positive sample optimization - negative sample selection" is constructed. By accurately locating key regions and enhancing positive sample quality through partitioned differentiation, and by using an entropy priority queue to select high-value, difficult negative samples to strengthen constraints, while integrating a contrastive loss optimization module to guide model learning, the accuracy, generalization ability, and training efficiency of skeleton behavior recognition are significantly improved.

[0061] Furthermore, by calculating the difference between the sample and the global statistical baseline posture through the key region segmentation module based on spatiotemporal dual-weight collaborative localization, and combining the joint degree calculated by the joint degree weighted difference activation module, the core motion joint group is located from the perspectives of difference degree and joint degree. Relying on the skeleton partitioning data enhancement module based on dual-scale game theory, strong enhancement is applied to the key region to preserve the action semantics, and weak enhancement is applied to the non-key region to maintain the structural features. This achieves the synergistic optimization of high information content and strong representativeness of positive samples, improves the sample semantics and enriches the diversity of features.

[0062] Furthermore, using the entropy value of the negative sample memory after sample replacement as the profit objective, the dominant strategy in game theory is deeply coupled with the screening of negative sample value. The degree of dominance of negative samples is quantified by information entropy to obtain dominant negative samples. Dominant negative samples are retained first to maintain the strong constraint of the queue, while inferior samples are eliminated. The management of negative samples is changed from indiscriminate treatment to dynamic screening based on dominant value, ensuring that the model always obtains the most critical comparison signal for improving feature discrimination ability.

[0063] Figure 5 A schematic diagram of the structure of a behavior recognition device 200 based on a dominant game-theoretic self-supervised neural network provided in an embodiment of the present invention is shown. Figure 5 As shown, the device 300 includes: The acquisition module 210 is used to acquire the visual data to be recognized; The behavior recognition module 220 is used to input the visual data to be recognized into a self-supervised neural network based on dominant game theory to obtain behavior recognition results; wherein, the self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory; the self-supervised neural network based on dominant game theory is trained according to preset sample data; the preset sample data is a skeleton behavior dataset; the skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks; During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data; the partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees; the replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, and selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, which is then used as the negative sample set for training. Output module 230 is used to output the behavior recognition result.

[0064] In an optional approach, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions according to the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data, further including: Calculate the difference and jointness of each skeleton node in the positive sample data; The spatiotemporal composite weight of the skeleton node is obtained by fusing the difference degree and the joint degree. Based on the spatiotemporal composite weight and the skeleton node key features, a key action region mask is generated. The key motion regions and non-key motion regions of the positive sample data are determined based on the key motion region mask.

[0065] In an alternative approach, calculating the dissimilarity and articulation of each skeleton node in the positive sample data further includes: Calculate the global statistical baseline pose based on the skeleton behavior dataset; The difference of each node is calculated based on the negative gradient of the cosine similarity between the global statistical baseline pose and each skeleton behavior data in the input skeleton behavior data with respect to the node features. Based on the adjacency matrix of the undirected graph corresponding to each skeleton behavior data in the skeleton behavior data, the number of connecting edges of each node in each skeleton behavior data is counted, and the joint degree of each node is calculated after normalizing the number of connecting edges.

[0066] In one alternative approach, the dual-scale game-based partitioned data augmentation module enhances the key motion regions and non-key motion regions in the positive sample data to different degrees, further including: Based on the key motion region, style transfer is performed on nodes with joint degree greater than the first joint degree threshold in each positive sample data to generate style-transferred positive sample data. Data augmentation of key and non-key motion regions in the style-transferred positive sample data was performed to different degrees on two scales: similarity and dissimilarity, to obtain the data-augmented positive sample dataset.

[0067] In one alternative approach, based on the key motion region, style transfer is performed on nodes in each positive sample data whose joint degree is greater than a first joint degree threshold to generate style-transferred positive sample data, including: The following formula will be used to calculate the first... j Positive samples The style migration to the first i Positive samples middle: ; in, These are positive samples after style transfer. Positive samples nodes The degree of joint movement, and Positive samples respectively and positive samples The spatial mean, and Samples and The time variance.

[0068] In one alternative approach, data augmentation is applied to key and non-key motion regions of the style-transferred positive sample data to varying degrees, based on both similarity and dissimilarity scales, resulting in a data-augmented positive sample dataset, including: Based on the payoff function of the two-scale dominant game, the target strong data augmentation strategy used in the key motion regions of the positive sample data after style transfer, and the target ordinary data augmentation strategy used in the non-key motion regions are determined; the target strong data augmentation strategy is one of a plurality of strong data augmentation strategies, and the target ordinary data augmentation strategy is one of a plurality of ordinary data augmentation strategies. The payoff function of the two-scale dominant game is: ; in, For key motion regions, the mask coefficients are... =1, otherwise the mask coefficient Set to 0; Indicates the first i Nodes of positive samples Use enhancement strategies Similarity score at time Indicates the first i Nodes of positive samples Use enhancement strategies Dissimilarity score at time; when At that time, take ;when At that time, take .

[0069] Based on the target strong data augmentation strategy, the key motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented key motion regions. Based on the target general data augmentation strategy, the non-critical motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented non-critical motion regions. The data-augmented key motion regions and the data-augmented non-key motion regions are stitched together to obtain the data-augmented positive sample.

[0070] In one alternative approach, the replacement queue based on entropy-driven dominant game uses the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, calculates the similarity normalized probability of each negative sample memory, selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, and uses the optimal negative sample memory as the training negative sample set, including: The payoff function for an entropy-driven dominant game can be calculated using the following formula: ; Among them, for dominant game The set of strategies in a game is ; For the negative sample memory size, The strategy represents replacing samples in the negative sample memory with the current sample. ; This represents the normalized probability of similarity. The normalized probability of similarity is expressed as: ; in, The cosine similarity between two negative samples is represented by... and Indicates negative samples Negative samples The negative sample features in the replaced negative sample memory, They represent the first Positive sample features during the training iteration.

[0071] This invention provides an embodiment that acquires visual data to be identified; inputs the visual data to be identified into a self-supervised neural network based on dominant game theory to obtain behavior recognition results; and outputs the behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data; the preset sample data is a skeleton behavior dataset; the skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data. The partitioned data enhancement module based on dual-scale game theory enhances the key and non-key motion regions in the positive sample data to different degrees. The replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory using the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, and selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, which is then used as the negative sample set for training. In this way, based on the "dominant game" theory in game theory, a two-dimensional dominant system of "positive sample optimization - negative sample selection" is constructed. By accurately locating key regions and enhancing positive sample quality through partitioned differentiation, and by using an entropy priority queue to select high-value, difficult negative samples to strengthen constraints, while integrating a contrastive loss optimization module to guide model learning, the accuracy, generalization ability, and training efficiency of skeleton behavior recognition are significantly improved.

[0072] Furthermore, by calculating the difference between the sample and the global statistical baseline posture through the key region segmentation module based on spatiotemporal dual-weight collaborative localization, and combining the joint degree calculated by the joint degree weighted difference activation module, the core motion joint group is located from the perspectives of difference degree and joint degree. Relying on the skeleton partitioning data enhancement module based on dual-scale game theory, strong enhancement is applied to the key region to preserve the action semantics, and weak enhancement is applied to the non-key region to maintain the structural features. This achieves the synergistic optimization of high information content and strong representativeness of positive samples, improves the sample semantics and enriches the diversity of features.

[0073] Furthermore, using the entropy value of the negative sample memory after sample replacement as the profit objective, the dominant strategy in game theory is deeply coupled with the screening of negative sample value. The degree of dominance of negative samples is quantified by information entropy to obtain dominant negative samples. Dominant negative samples are retained first to maintain the strong constraint of the queue, while inferior samples are eliminated. The management of negative samples is changed from indiscriminate treatment to dynamic screening based on dominant value, ensuring that the model always obtains the most critical comparison signal for improving feature discrimination ability.

[0074] Figure 6 The diagram shows a structural schematic of a computer device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0075] like Figure 6 As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0076] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps described in the embodiment of the behavior recognition method for a self-supervised neural network based on dominant game theory.

[0077] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0078] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0079] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0080] Specifically, program 410 can be called by processor 402 to cause the computer device to perform the following operations: Acquire the visual data to be recognized; The visual data to be identified is input into a self-supervised neural network based on dominant game theory to obtain behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data. The preset sample data is a skeleton behavior dataset, which includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data; the partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees; the replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, and selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, which is then used as the negative sample set for training. Output the behavior recognition result.

[0081] In an optional approach, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions according to the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data, further including: Calculate the difference and jointness of each skeleton node in the positive sample data; The spatiotemporal composite weight of the skeleton node is obtained by fusing the difference degree and the joint degree. Based on the spatiotemporal composite weight and the skeleton node key features, a key action region mask is generated. The key motion regions and non-key motion regions of the positive sample data are determined based on the key motion region mask.

[0082] In an alternative approach, calculating the dissimilarity and articulation of each skeleton node in the positive sample data further includes: Calculate the global statistical baseline pose based on the skeleton behavior dataset; The difference of each node is calculated based on the negative gradient of the cosine similarity between the global statistical baseline pose and each skeleton behavior data in the input skeleton behavior data with respect to the node features. Based on the adjacency matrix of the undirected graph corresponding to each skeleton behavior data in the skeleton behavior data, the number of connecting edges of each node in each skeleton behavior data is counted, and the joint degree of each node is calculated after normalizing the number of connecting edges.

[0083] In one alternative approach, the dual-scale game-based partitioned data augmentation module enhances the key motion regions and non-key motion regions in the positive sample data to different degrees, further including: Based on the key motion region, style transfer is performed on nodes with joint degree greater than the first joint degree threshold in each positive sample data to generate style-transferred positive sample data. Data augmentation of key and non-key motion regions in the style-transferred positive sample data was performed to different degrees on two scales: similarity and dissimilarity, to obtain the data-augmented positive sample dataset.

[0084] In one alternative approach, based on the key motion region, style transfer is performed on nodes in each positive sample data whose joint degree is greater than a first joint degree threshold to generate style-transferred positive sample data, including: The following formula will be used to calculate the first... j Positive samples The style migration to the first i Positive samples middle: ; in, These are positive samples after style transfer. Positive samples nodes The degree of joint movement, and Positive samples respectively and positive samples The spatial mean, and Samples and The time variance.

[0085] In one alternative approach, data augmentation is applied to key and non-key motion regions of the style-transferred positive sample data to varying degrees, based on both similarity and dissimilarity scales, resulting in a data-augmented positive sample dataset, including: Based on the payoff function of the two-scale dominant game, the target strong data augmentation strategy used in the key motion regions of the positive sample data after style transfer, and the target ordinary data augmentation strategy used in the non-key motion regions are determined; the target strong data augmentation strategy is one of a plurality of strong data augmentation strategies, and the target ordinary data augmentation strategy is one of a plurality of ordinary data augmentation strategies. The payoff function of the two-scale dominant game is: ; in, For key motion regions, the mask coefficients are... =1, otherwise the mask coefficient Set to 0; Indicates the first i Nodes of positive samples Use enhancement strategies Similarity score at time Indicates the first i Nodes of positive samples Use enhancement strategies Dissimilarity score at time; when At that time, take ;when At that time, take .

[0086] Based on the target strong data augmentation strategy, the key motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented key motion regions. Based on the target general data augmentation strategy, the non-critical motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented non-critical motion regions. The data-augmented key motion regions and the data-augmented non-key motion regions are stitched together to obtain the data-augmented positive sample.

[0087] In one alternative approach, the replacement queue based on entropy-driven dominant game uses the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, calculates the similarity normalized probability of each negative sample memory, selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, and uses the optimal negative sample memory as the training negative sample set, including: The payoff function for an entropy-driven dominant game can be calculated using the following formula: ; Among them, for dominant game The set of strategies in a game is ; For the negative sample memory size, The strategy represents replacing samples in the negative sample memory with the current sample. ; This represents the normalized probability of similarity. The normalized probability of similarity is expressed as: ; in, The cosine similarity between two negative samples is represented by... and Indicates negative samples Negative samples The negative sample features in the replaced negative sample memory, They represent the first Positive sample features during the training iteration.

[0088] This invention provides an embodiment that acquires visual data to be identified; inputs the visual data to be identified into a self-supervised neural network based on dominant game theory to obtain behavior recognition results; and outputs the behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data; the preset sample data is a skeleton behavior dataset; the skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data. The partitioned data enhancement module based on dual-scale game theory enhances the key and non-key motion regions in the positive sample data to different degrees. The replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory using the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, and selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, which is then used as the negative sample set for training. In this way, based on the "dominant game" theory in game theory, a two-dimensional dominant system of "positive sample optimization - negative sample selection" is constructed. By accurately locating key regions and enhancing positive sample quality through partitioned differentiation, and by using an entropy priority queue to select high-value, difficult negative samples to strengthen constraints, while integrating a contrastive loss optimization module to guide model learning, the accuracy, generalization ability, and training efficiency of skeleton behavior recognition are significantly improved.

[0089] Furthermore, by calculating the difference between the sample and the global statistical baseline posture through the key region segmentation module based on spatiotemporal dual-weight collaborative localization, and combining the joint degree calculated by the joint degree weighted difference activation module, the core motion joint group is located from the perspectives of difference degree and joint degree. Relying on the skeleton partitioning data enhancement module based on dual-scale game theory, strong enhancement is applied to the key region to preserve the action semantics, and weak enhancement is applied to the non-key region to maintain the structural features. This achieves the synergistic optimization of high information content and strong representativeness of positive samples, improves the sample semantics and enriches the diversity of features.

[0090] Furthermore, using the entropy value of the negative sample memory after sample replacement as the profit objective, the dominant strategy in game theory is deeply coupled with the screening of negative sample value. The degree of dominance of negative samples is quantified by information entropy to obtain dominant negative samples. Dominant negative samples are retained first to maintain the strong constraint of the queue, while inferior samples are eliminated. The management of negative samples is changed from indiscriminate treatment to dynamic screening based on dominant value, ensuring that the model always obtains the most critical comparison signal for improving feature discrimination ability.

[0091] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform the behavior recognition method based on a dominant game-theoretic self-supervised neural network in any of the above method embodiments.

[0092] Executable instructions can be used to cause computer devices to perform the following operations: Acquire the visual data to be recognized; The visual data to be identified is input into a self-supervised neural network based on dominant game theory to obtain behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data. The preset sample data is a skeleton behavior dataset, which includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data; the partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees; the replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, and selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, which is then used as the negative sample set for training. Output the behavior recognition result.

[0093] In an optional approach, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions according to the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data, further including: Calculate the difference and jointness of each skeleton node in the positive sample data; The spatiotemporal composite weight of the skeleton node is obtained by fusing the difference degree and the joint degree. Based on the spatiotemporal composite weight and the skeleton node key features, a key action region mask is generated. The key motion regions and non-key motion regions of the positive sample data are determined based on the key motion region mask.

[0094] In an alternative approach, calculating the dissimilarity and articulation of each skeleton node in the positive sample data further includes: Calculate the global statistical baseline pose based on the skeleton behavior dataset; The difference of each node is calculated based on the negative gradient of the cosine similarity between the global statistical baseline pose and each skeleton behavior data in the input skeleton behavior data with respect to the node features. Based on the adjacency matrix of the undirected graph corresponding to each skeleton behavior data in the skeleton behavior data, the number of connecting edges of each node in each skeleton behavior data is counted, and the joint degree of each node is calculated after normalizing the number of connecting edges.

[0095] In one alternative approach, the dual-scale game-based partitioned data augmentation module enhances the key motion regions and non-key motion regions in the positive sample data to different degrees, further including: Based on the key motion region, style transfer is performed on nodes with joint degree greater than the first joint degree threshold in each positive sample data to generate style-transferred positive sample data. Data augmentation of key and non-key motion regions in the style-transferred positive sample data was performed to different degrees on two scales: similarity and dissimilarity, to obtain the data-augmented positive sample dataset.

[0096] In one alternative approach, based on the key motion region, style transfer is performed on nodes in each positive sample data whose joint degree is greater than a first joint degree threshold to generate style-transferred positive sample data, including: The following formula will be used to calculate the first... j Positive samples The style migration to the first i Positive samples middle: ; in, These are positive samples after style transfer. Positive samples nodes The degree of joint movement, and Positive samples respectively and positive samples The spatial mean, and Samples and The time variance.

[0097] In one alternative approach, data augmentation is applied to key and non-key motion regions of the style-transferred positive sample data to varying degrees, based on both similarity and dissimilarity scales, resulting in a data-augmented positive sample dataset, including: Based on the payoff function of the two-scale dominant game, the target strong data augmentation strategy used in the key motion regions of the positive sample data after style transfer, and the target ordinary data augmentation strategy used in the non-key motion regions are determined; the target strong data augmentation strategy is one of a plurality of strong data augmentation strategies, and the target ordinary data augmentation strategy is one of a plurality of ordinary data augmentation strategies. The payoff function of the two-scale dominant game is: ; in, For key motion regions, the mask coefficients are... =1, otherwise the mask coefficient Set to 0; Indicates the first i Nodes of positive samples Use enhancement strategies Similarity score at time Indicates the first iNodes of positive samples Use enhancement strategies Dissimilarity score at time; when At that time, take ;when At that time, take .

[0098] Based on the target strong data augmentation strategy, the key motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented key motion regions. Based on the target general data augmentation strategy, the non-critical motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented non-critical motion regions. The data-augmented key motion regions and the data-augmented non-key motion regions are stitched together to obtain the data-augmented positive sample.

[0099] In one alternative approach, the replacement queue based on entropy-driven dominant game uses the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, calculates the similarity normalized probability of each negative sample memory, selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, and uses the optimal negative sample memory as the training negative sample set, including: The payoff function for an entropy-driven dominant game can be calculated using the following formula: ; Among them, for dominant game The set of strategies in a game is ; For the negative sample memory size, The strategy represents replacing samples in the negative sample memory with the current sample. ; This represents the normalized probability of similarity. The normalized probability of similarity is expressed as: ; in, The cosine similarity between two negative samples is represented by... and Indicates negative samples Negative samples The negative sample features in the replaced negative sample memory, They represent the first Positive sample features during the training iteration.

[0100] This invention provides an embodiment that acquires visual data to be identified; inputs the visual data to be identified into a self-supervised neural network based on dominant game theory to obtain behavior recognition results; and outputs the behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data; the preset sample data is a skeleton behavior dataset; the skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data. The partitioned data enhancement module based on dual-scale game theory enhances the key and non-key motion regions in the positive sample data to different degrees. The replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory using the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game, and selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, which is then used as the negative sample set for training. In this way, based on the "dominant game" theory in game theory, a two-dimensional dominant system of "positive sample optimization - negative sample selection" is constructed. By accurately locating key regions and enhancing positive sample quality through partitioned differentiation, and by using an entropy priority queue to select high-value, difficult negative samples to strengthen constraints, while integrating a contrastive loss optimization module to guide model learning, the accuracy, generalization ability, and training efficiency of skeleton behavior recognition are significantly improved.

[0101] Furthermore, by calculating the difference between the sample and the global statistical baseline posture through the key region segmentation module based on spatiotemporal dual-weight collaborative localization, and combining the joint degree calculated by the joint degree weighted difference activation module, the core motion joint group is located from the perspectives of difference degree and joint degree. Relying on the skeleton partitioning data enhancement module based on dual-scale game theory, strong enhancement is applied to the key region to preserve the action semantics, and weak enhancement is applied to the non-key region to maintain the structural features. This achieves the synergistic optimization of high information content and strong representativeness of positive samples, improves the sample semantics and enriches the diversity of features.

[0102] Furthermore, using the entropy value of the negative sample memory after sample replacement as the profit objective, the dominant strategy in game theory is deeply coupled with the screening of negative sample value. The degree of dominance of negative samples is quantified by information entropy to obtain dominant negative samples. Dominant negative samples are retained first to maintain the strong constraint of the queue, while inferior samples are eliminated. The management of negative samples is changed from indiscriminate treatment to dynamic screening based on dominant value, ensuring that the model always obtains the most critical comparison signal for improving feature discrimination ability.

[0103] This invention provides a behavior recognition device based on a self-supervised neural network using dominant game theory, for executing the aforementioned behavior recognition method based on a self-supervised neural network using dominant game theory.

[0104] This invention provides a computer program that can be called by a processor to enable a computer device to execute the behavior recognition method based on a dominant game and a self-supervised neural network in any of the above method embodiments.

[0105] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to perform the behavior recognition method based on a dominant game and a self-supervised neural network in any of the above method embodiments.

[0106] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0107] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0108] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0109] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0110] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A behavior recognition method based on a self-supervised neural network of dominant game theory, characterized in that, The method includes: Acquire the visual data to be recognized; The visual data to be identified is input into a self-supervised neural network based on dominant game theory to obtain behavior recognition results. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data enhancement module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data. The preset sample data is a skeleton behavior dataset, which includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data; the partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees; the replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, and uses the optimal negative sample memory bank as the training negative sample set. Output the behavior recognition result.

2. The method according to claim 1, characterized in that, The key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions according to the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data, and further includes: Calculate the difference and jointness of each skeleton node in the positive sample data; The spatiotemporal composite weight of the skeleton node is obtained by fusing the difference degree and the joint degree. Based on the spatiotemporal composite weight and the skeleton node key features, a key action region mask is generated. The key motion regions and non-key motion regions of the positive sample data are determined based on the key motion region mask.

3. The method according to claim 2, characterized in that, The calculation of the difference and jointness of each skeleton node in the positive sample data further includes: Calculate the global statistical baseline pose based on the skeleton behavior dataset; The difference of each node is calculated based on the negative gradient of the cosine similarity between the global statistical baseline pose and each skeleton behavior data in the input skeleton behavior data with respect to the node features. Based on the adjacency matrix of the undirected graph corresponding to each skeleton behavior data in the skeleton behavior data, the number of connecting edges of each node in each skeleton behavior data is counted, and the joint degree of each node is calculated after normalizing the number of connecting edges.

4. The method according to claim 3, characterized in that, The dual-scale game-based partitioned data augmentation module enhances the key motion regions and non-key motion regions in the positive sample data to different degrees, further including: Based on the key motion region, style transfer is performed on nodes with joint degree greater than the first joint degree threshold in each positive sample data to generate style-transferred positive sample data. Data augmentation of key and non-key motion regions in the style-transferred positive sample data was performed to different degrees on two scales: similarity and dissimilarity, to obtain the data-augmented positive sample dataset.

5. The method according to claim 4, characterized in that, Based on the key motion region, style transfer is performed on nodes with joint degrees greater than the first joint degree threshold in each positive sample data, generating style-transferred positive sample data, including: The following formula will be used to calculate the first... j Positive samples The style migration to the first i Positive samples middle: ； in, These are positive samples after style transfer. Positive sample nodes The degree of joint movement, and Positive samples respectively and positive samples The spatial mean, and Samples and The time variance.

6. The method according to claim 5, characterized in that, Data augmentation of key and non-key motion regions in the style-transfer positive sample data was performed to varying degrees on both similarity and dissimilarity scales, resulting in the data-augmented positive sample dataset, including: Based on the payoff function of the two-scale dominant game, the target strong data augmentation strategy used in the key motion regions of the positive sample data after style transfer, and the target ordinary data augmentation strategy used in the non-key motion regions are determined; the target strong data augmentation strategy is one of a plurality of strong data augmentation strategies, and the target ordinary data augmentation strategy is one of a plurality of ordinary data augmentation strategies. The payoff function of the two-scale dominant game is: ； in, For key motion regions, the mask coefficients are... =1, otherwise the mask coefficient Set to 0; Indicates the first i Nodes of positive samples Use enhancement strategies Similarity score at time Indicates the first i Nodes of positive samples Use enhancement strategies Dissimilarity score at time; when At that time, take ;when At that time, take ; Based on the target strong data augmentation strategy, the key motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented key motion regions. Based on the target general data augmentation strategy, the non-critical motion regions of the positive sample data after style transfer are augmented to obtain the data-augmented non-critical motion regions. The data-augmented key motion regions and the data-augmented non-key motion regions are stitched together to obtain the data-augmented positive sample.

7. The method according to any one of claims 1-6, characterized in that, The replacement queue based on entropy-driven dominant game uses the information entropy of each negative sample memory as the payoff function of the entropy-driven dominant game. It calculates the normalized similarity probability of each negative sample memory, selects the negative sample memory with the highest information entropy value as the optimal negative sample memory, and uses the optimal negative sample memory as the training negative sample set, including: The payoff function for an entropy-driven dominant game can be calculated using the following formula: ； Among them, for dominant game The set of strategies in a game is ; For the negative sample memory size, The strategy represents replacing samples in the negative sample memory with the current sample. ; This represents the normalized probability of similarity. The normalized probability of similarity is expressed as: ； in, The cosine similarity between two negative samples is represented by... and Indicates negative samples Negative samples The negative sample features in the replaced negative sample memory, They represent the first Positive sample features during the training iteration.

8. A behavior recognition device based on a self-supervised neural network of dominant game theory, characterized in that, The device includes: The acquisition module is used to acquire the visual data to be recognized; The behavior recognition module is used to input the visual data to be recognized into a self-supervised neural network based on dominant game theory to obtain the behavior recognition result. The self-supervised neural network based on dominant game theory includes a key region segmentation module based on spatiotemporal dual-weight collaborative localization, a partitioned data augmentation module based on dual-scale game theory, and a replacement queue based on entropy-driven dominant game theory. The self-supervised neural network based on dominant game theory is trained using preset sample data. The preset sample data is a skeleton behavior dataset. The skeleton behavior dataset includes positive sample data and negative sample data stored in multiple negative sample memory banks. During iterative training, the key region segmentation module based on spatiotemporal dual-weight collaborative localization determines key motion regions and non-key motion regions based on the spatiotemporal composite weight and node key features of each node in the input skeleton behavior data; the partitioned data augmentation module based on dual-scale game theory enhances the key motion regions and non-key motion regions in the positive sample data to different degrees; the replacement queue based on entropy-driven dominant game theory calculates the similarity normalization probability of each negative sample memory bank using the information entropy of each negative sample memory bank as the payoff function of the entropy-driven dominant game, selects the negative sample memory bank with the highest information entropy value as the optimal negative sample memory bank, and uses the optimal negative sample memory bank as the training negative sample set. The output module is used to output the behavior recognition results.

9. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the behavior recognition method based on the dominant game-theoretic self-supervised neural network as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the behavior recognition method based on a dominant game-theoretic self-supervised neural network as described in any one of claims 1-7.

Images