Interactive behavior recognition method and system based on multimodal group behavior recognition technology

Abstract

The present invention belongs to the field of image recognition, and proposes an interactive behavior recognition method and system based on multimodal group behavior recognition technology. A multimodal group behavior recognition network is designed, including an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and multimodal feature processing is performed respectively to avoid the problem of low recognition accuracy caused by factors such as interactive behaviors between groups and complex interactive environments when recognition is performed through single modal information. The method further conforms to actual maintenance conditions, improves the overall recognition accuracy, and improves the recognition efficiency through lightweight design. The relationship and interaction between groups and between groups and interactive targets are enhanced, and the recognition accuracy is further improved. The recognition obstacles caused by occlusion and overlap are avoided. The present invention improves the recognition accuracy and efficiency of group target interactive behaviors.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition, and in particular relates to an interactive behavior recognition method and system based on multimodal group behavior recognition technology. Background Art

[0002] With the rapid development of the railway industry, the demand and importance of railway maintenance are gradually increasing. Because there are many workers performing maintenance work during the railway maintenance process, and they need to cooperate with each other to complete the complex maintenance process, supervision and monitoring of maintenance work and maintenance workers is particularly important. Among them, safety monitoring during the maintenance process is an important part of the railway maintenance process.

[0003] In the prior art, the railway maintenance process is often monitored by combining manual inspection and video surveillance. However, the cost of manual inspection and monitoring is too high, and the efficiency is low, and it is impossible to monitor comprehensively. Therefore, computer vision technology has become an important monitoring and identification means. However, the existing computer vision technology is limited by various factors such as hardware equipment conditions and complex maintenance environments, and cannot meet the monitoring requirements of the railway maintenance process. In particular, there are many workers in the maintenance process and there will be complex interactions and cooperation between the workers, resulting in overlap and occlusion between the workers and the interacting workers, and between the workers and the maintenance interaction objects, which greatly increases the difficulty of monitoring and identification and greatly affects the accuracy of identification.

[0004] Therefore, how to design an identification method to avoid the influence of the interactive behavior of many maintenance workers in a complex environment has become an urgent problem to be solved. Summary of the invention

[0005] Based on this, the present invention proposes an interactive behavior recognition method and system based on multimodal group behavior recognition technology. A multimodal group behavior recognition network is designed, including an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and multimodal feature processing is performed respectively to avoid the problem of low recognition accuracy due to the interactive behavior between groups and the complex interactive environment when recognition is performed through single modal information. It is further in line with the actual maintenance situation, improves the overall recognition accuracy, and improves the recognition efficiency through lightweight design. The relationship and interaction between groups and between groups and interactive targets are enhanced, which further improves the recognition accuracy and avoids the recognition obstacles caused by occlusion and overlap. The present invention improves the recognition accuracy and efficiency of group target interactive behavior.

[0006] The present invention proposes an interactive behavior recognition method based on multimodal group behavior recognition technology, comprising:

[0007] The target video image is collected in real time and preprocessed, and the preprocessed target video image is input into a multimodal group behavior recognition network, wherein the multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and the network structure among the image branch subnetwork, the text branch subnetwork and the key point branch subnetwork is a mutually parallel structure;

[0008] The image branch sub-network extracts features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, and then performs feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship-enhanced image features;

[0009] The text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operations on the basic target group image features according to the lightweight attention enhancement module to obtain attention enhanced image features, and performs information interaction processing on the action text features and the attention enhanced image features according to the knowledge distillation algorithm to obtain interactive text features, and then performs feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features, and the group relationship parameter adjustment module includes a relationship enhancement submodule and a parameter adjustment submodule, and the relationship enhancement submodule shares parameters with the group relationship enhancement module;

[0010] The key point branch sub-network extracts key point features from the target video image to obtain key point features, and then interactively enhances the key point features according to the interactive module to obtain interactively enhanced key point features;

[0011] Group behavior recognition is performed based on the relationship-enhanced image features, the relationship-enhanced text features, and the interaction-enhanced key point features to obtain a final classification result.

[0012] In summary, according to the above-mentioned interactive behavior recognition method based on multimodal group behavior recognition technology, a multimodal group behavior recognition network is designed, including an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and multimodal feature processing is performed respectively to avoid the problem of low recognition accuracy due to the interactive behavior between groups and the complex interactive environment when identifying through a single modal information. It further conforms to the actual maintenance situation and improves the overall recognition accuracy. Through lightweight design, the recognition efficiency is improved, and the relationship and interaction between groups and between groups and interactive targets are enhanced, which further improves the recognition accuracy and avoids the recognition obstacles caused by occlusion and overlap. The present invention improves the recognition accuracy and efficiency of group target interactive behaviors.Specifically, the target video image is collected in real time and preprocessed, and the preprocessed target video image is input into a multimodal group behavior recognition network. The multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork. The network structure between the image branch subnetwork, the text branch subnetwork and the key point branch subnetwork is a parallel structure, which combines multimodal information, enriches the recognition features, and further improves the recognition accuracy. The image branch subnetwork extracts features of the target video image according to a lightweight image feature processing module to obtain basic target group image features, and then performs feature relationship enhancement according to a group relationship enhancement module of time-space dual-stream interaction to obtain relationship-enhanced image features. The lightweight design speeds up the processing speed and improves the overall recognition efficiency. The text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operations on the basic target group image features according to a lightweight attention enhancement module to obtain attention-enhanced image features. The action text features and the attention-enhanced image features are processed for information interaction according to a knowledge distillation algorithm. The method comprises the following steps: processing the interactive text features to obtain interactive text features, and then performing feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features. The group relationship parameter adjustment module includes a relationship enhancement submodule and a parameter adjustment submodule. The parameters of the relationship enhancement submodule and the group relationship enhancement module are shared. According to the information interactive transmission between text and image features, repeated processing is avoided, the processing speed is improved, and the recognition efficiency is improved. At the same time, the relationship information between group targets is enhanced, the recognition obstacles caused by the complex relationship between target groups are avoided, and the recognition accuracy is improved. The key point branch subnetwork extracts key point features from the target video image to obtain key point features, and then interactively enhances the key point features according to the interactive module to obtain interactively enhanced key point features, which enhances the relationship between the target group and the interactive object, and further improves the recognition accuracy. According to the relationship enhanced image features, the relationship enhanced text features and the interactively enhanced key point features, group behavior recognition is performed to obtain the final classification result. The present invention improves the recognition accuracy and efficiency of group target interactive behaviors.

[0013] Furthermore, the image branch subnetwork extracts features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, specifically including:

[0014] The image branch subnetwork includes a lightweight image feature processing module, wherein the lightweight image feature processing module includes a lightweight network layer and a bounding box feature alignment block;

[0015] The lightweight network layer includes a depth-separable convolution block, an inverted residual block, and a squeeze excitation block to extract basic features, and adjust the feature map bounding box coordinates according to the spatial scaling algorithm. The algorithm relationship of the feature map bounding box coordinate adjustment is as follows:

[0016] ,

[0017] ,

[0018] The bounding box feature alignment block performs an alignment operation on the adjusted bounding box coordinates and the extracted basic features to obtain basic target group image features.

[0019] Furthermore, the step of performing feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship enhanced image features specifically includes:

[0020] The image branch subnetwork includes a group relationship enhancement module, and the group relationship enhancement module includes a space-time branch path and a time-space branch path;

[0021] The space-time branch path includes a spatial attention residual submodule, and the spatial attention residual submodule includes a spatial attention block, a multi-layer perception block and a residual branch;

[0022] The time-space branch path includes a time attention residual submodule, and the time attention residual submodule includes a time attention block, a multi-layer perception block and a residual branch;

[0023] The group relationship enhancement module performs feature relationship enhancement based on a spatiotemporal dual-stream interaction algorithm;

[0024] The spatiotemporal dual-stream interaction algorithm is specifically as follows:

[0025] ,

[0026] ,

[0027] ,

[0028] ,

[0029] ,

[0030] ,

[0031] ,

[0032] ,

[0033] ,

[0034] ,

[0035] in, Represents a PE function, represents the PE function of even dimensions, represents the PE function of odd dimensions, represents the center coordinates of the bounding box, represents the feature dimension, represents the dimension of the position encoding, represents the input features of the space-time branching path, represents the input features of the time-space branching path, and Represent the PE function output features in the spatial attention block and the temporal attention block, respectively. and Represent the PE functions in the spatial attention block and the temporal attention block, respectively. and Represent the multi-head attention enhancement features in the spatial attention block and the temporal attention block, respectively. represents a linear layer, represents the multi-head attention mechanism layer, and Represent the final outputs of the spatial attention block and the temporal attention block, respectively. represents the feed-forward layer, represents the temporal attention residual submodule, represents the spatial attention residual submodule, represents a multi-layer perception block, and They represent the final output features of the space-time branch path and the time-space branch path respectively;

[0036] Feature fusion is performed based on the final output features of the space-time branch path and the time-space branch path to obtain relation-enhanced image features.

[0037] Furthermore, the text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operations on the basic target group image features according to the lightweight attention enhancement module to obtain the step of attention enhanced image features, specifically including:

[0038] The text branch sub-network performs text prompt enhancement on the preset action label according to the text prompt module to obtain an enhanced action label, and then generates features for the enhanced action label according to the text encoding block to obtain action text features;

[0039] After acquiring the basic target group image features, an attention enhancement operation is performed according to a lightweight attention enhancement module, wherein the lightweight attention enhancement module includes a traditional self-attention layer and a transfer attention layer;

[0040] The attention score of the multi-head attention is obtained according to the traditional self-attention layer. The specific algorithm for obtaining the attention score is as follows:

[0041] ,

[0042] in, represents the attention score of multi-head attention, represents the query matrix, represents the key matrix, represents transpose, represents the dimension of the position encoding;

[0043] The attention score is linearly transformed, and the specific algorithm of the linear transformation is as follows:

[0044] ,

[0045] in, represents the attention score after linear transformation, represents a linear transformation;

[0046] The linearly transformed attention score is input into the transfer attention layer to obtain the attention-enhanced image features. The specific algorithm for obtaining the attention-enhanced image features is as follows:

[0047] ,

[0048] ,

[0049] in, represents the output features of the transfer attention layer, represents the activation function, represents the value matrix, represents the attention-enhanced image features, represents matrix multiplication, Represents the output features of the traditional self-attention layer.

[0050] Furthermore, the step of performing information interaction processing on the action text features and the attention-enhanced image features according to the knowledge distillation algorithm to obtain interactive text features, and then performing feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship-enhanced text features specifically includes:

[0051] According to the knowledge distillation algorithm, information interaction processing is performed on the action text features and the attention enhanced image features. The specific algorithm of the information interaction processing is as follows:

[0052] ,

[0053] ,

[0054] ,

[0055] in, Indicates information interaction processing of action text features. Indicates information interaction processing of attention-enhanced image features, Represents the action text features to be processed, represents the attention-enhanced image features to be processed, represents the feature ordinal number, represents the total number of knowledge distillation training pairs, represents the ordinal number of knowledge distillation training pairs, represents the cosine similarity function, represents transpose, is a learnable temperature parameter, The value range is [0,1], and the initial value is 0.5;

[0056] The knowledge distillation loss is calculated according to the KL divergence algorithm. The specific algorithm for calculating the knowledge distillation loss is as follows:

[0057] ,

[0058] ,

[0059] in, represents the knowledge distillation loss, represents the KL divergence algorithm, represents the predicted probability, Indicates authenticity;

[0060] According to the group relationship parameter adjustment module, feature relationship enhancement and parameter adjustment operations are performed to obtain relationship enhanced text features. The specific algorithm of the parameter adjustment operation is as follows:

[0061] ,

[0062] in, Represents the final output parameter, represents the initial parameters, represents the impact factor, and represents a randomly initialized matrix from a normal distribution, Represents the input features of the group relationship adjustment module.

[0063] Furthermore, the key point branch sub-network extracts key point features from the target video image to obtain key point features, specifically comprising:

[0064] The key point branch sub-network includes a key point feature extraction module, and the key point feature extraction module includes a posture estimation network layer and an embedding layer;

[0065] The key point feature extraction module extracts key point features from the target video image. The key point feature extraction algorithm is as follows:

[0066] ,

[0067] in, represents key point features, represents the feed-forward layer, represents a cascade function, Feature vectors representing different joint types, represents the normalized and standardized spatial position encoding, Indicates absolute position encoding.

[0068] Furthermore, the step of interactively enhancing the key point features according to the interactive module to obtain interactively enhanced key point features specifically includes:

[0069] The key point features are interactively enhanced according to the interactive module to obtain interactively enhanced key point features. The specific algorithm for obtaining the interactively enhanced key point features is as follows:

[0070] ,

[0071] in, represents interactively enhanced key point features, represents the activation function, represents the self-attention enhancement operation, Represents the interaction between key points. represents the norm function, Represents node characteristics, Represents feature weight.

[0072] The present invention proposes an interactive behavior recognition method based on multimodal group behavior recognition technology, which is characterized by comprising:

[0073] An acquisition module is used to acquire a target video image in real time and perform preprocessing, and input the preprocessed target video image into a multimodal group behavior recognition network, wherein the multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork, and a key point branch subnetwork, and the network structure among the image branch subnetwork, the text branch subnetwork, and the key point branch subnetwork is a mutually parallel structure;

[0074] An image feature processing module is used for the image branch sub-network to extract features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, and then perform feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship-enhanced image features;

[0075] A text feature processing module, used for the text branch sub-network to obtain action text features according to preset action labels, and then perform attention enhancement operations on the basic target group image features according to the lightweight attention enhancement module to obtain attention enhanced image features, and perform information interaction processing on the action text features and the attention enhanced image features according to the knowledge distillation algorithm to obtain interactive text features, and then perform feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features, wherein the group relationship parameter adjustment module includes a relationship enhancement sub-module and a parameter adjustment sub-module, and the relationship enhancement sub-module shares parameters with the group relationship enhancement module;

[0076] A key point feature processing module, used for the key point branch sub-network to extract key point features from the target video image to obtain key point features, and then interactively enhance the key point features according to the interactive module to obtain interactively enhanced key point features;

[0077] The classification module is used to perform group behavior recognition according to the relationship-enhanced image features, the relationship-enhanced text features and the interaction-enhanced key point features to obtain a final classification result.

[0078] The present invention also provides a storage medium, which stores one or more programs. When the programs are executed by a processor, the interactive behavior recognition method based on the multimodal group behavior recognition technology as described above is implemented.

[0079] The present invention further provides a computer device, the computer device comprising a memory and a processor, wherein:

[0080] The memory is used to store computer programs;

[0081] When the processor is used to execute the computer program stored in the memory, it implements the interactive behavior recognition method based on the multimodal group behavior recognition technology as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A flowchart of an interactive behavior recognition method based on multimodal group behavior recognition technology proposed in the first embodiment of the present invention;

[0083] Figure 2 A flowchart of an interactive behavior recognition method based on multimodal group behavior recognition technology proposed in the second embodiment of the present invention;

[0084] Figure 3 This is a schematic diagram of the structure of an interactive behavior recognition system based on multimodal group behavior recognition technology proposed in the third embodiment of the present invention.

[0085] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0086] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0087] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0088] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0089] See also Figure 1 , which is a flow chart of an interactive behavior recognition method based on multimodal group behavior recognition technology proposed in the first embodiment of the present invention, and the interactive behavior recognition method based on multimodal group behavior recognition technology includes steps S01 to S05, wherein:

[0090] Step S01: collecting target video images in real time and preprocessing them, and inputting the preprocessed target video images into a multimodal group behavior recognition network;

[0091] It should be noted that the multimodal group behavior recognition network described in this embodiment includes an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and the network structure between the image branch subnetwork, the text branch subnetwork and the key point branch subnetwork is a mutually parallel structure.

[0092] Step S02: the image branch sub-network extracts features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, and then performs feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship enhanced image features;

[0093] It should be noted that in this embodiment, the image branch sub-network includes a lightweight image feature processing module, and the lightweight image feature processing module includes a lightweight network layer and a bounding box feature alignment block;

[0094] The lightweight network layer includes a depth-separable convolution block, an inverted residual block, and a squeeze excitation block to extract basic features, and adjust the feature map bounding box coordinates according to the spatial scaling algorithm. The algorithm relationship of the feature map bounding box coordinate adjustment is as follows:

[0095] ,

[0096] ,

[0097] The bounding box feature alignment block performs an alignment operation on the adjusted bounding box coordinates and the extracted basic features to obtain basic target group image features;

[0098] The image branch subnetwork includes a group relationship enhancement module, and the group relationship enhancement module includes a space-time branch path and a time-space branch path;

[0099] The space-time branch path includes a spatial attention residual submodule, and the spatial attention residual submodule includes a spatial attention block, a multi-layer perception block and a residual branch;

[0100] The time-space branch path includes a time attention residual submodule, and the time attention residual submodule includes a time attention block, a multi-layer perception block and a residual branch;

[0101] The group relationship enhancement module performs feature relationship enhancement based on a spatiotemporal dual-stream interaction algorithm;

[0102] The spatiotemporal dual-stream interaction algorithm is specifically as follows:

[0103] ,

[0104] ,

[0105] ,

[0106] ,

[0107] ,

[0108] ,

[0109] ,

[0110] ,

[0111] ,

[0112] ,

[0113] in, Represents a PE function, represents the PE function of even dimensions, represents the PE function of odd dimensions, represents the center coordinates of the bounding box, represents the feature dimension, represents the dimension of the position encoding, represents the input features of the space-time branching path, represents the input features of the time-space branching path, and Represent the PE function output features in the spatial attention block and the temporal attention block, respectively. and Represent the PE functions in the spatial attention block and the temporal attention block, respectively. and Represent the multi-head attention enhancement features in the spatial attention block and the temporal attention block, respectively. represents a linear layer, represents the multi-head attention mechanism layer, and Represent the final outputs of the spatial attention block and the temporal attention block, respectively. represents the feed-forward layer, represents the temporal attention residual submodule, represents the spatial attention residual submodule, represents a multi-layer perception block, and They represent the final output features of the space-time branch path and the time-space branch path respectively;

[0114] Feature fusion is performed based on the final output features of the space-time branch path and the time-space branch path to obtain relation-enhanced image features.

[0115] Step S03: the text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operation on the basic target group image features according to the lightweight attention enhancement module to obtain attention enhanced image features, and performs information interaction processing on the action text features and the attention enhanced image features according to the knowledge distillation algorithm to obtain interactive text features, and then performs feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features;

[0116] It should be noted that the group relationship parameter adjustment module in this embodiment includes a relationship enhancement submodule and a parameter adjustment submodule. The relationship enhancement submodule shares parameters with the group relationship enhancement module. The text branch subnetwork performs text prompt enhancement on the preset action label according to the text prompt module to obtain an enhanced action label, and then generates features for the enhanced action label according to the text encoding block to obtain action text features.

[0117] After acquiring the basic target group image features, an attention enhancement operation is performed according to a lightweight attention enhancement module, wherein the lightweight attention enhancement module includes a traditional self-attention layer and a transfer attention layer;

[0118] The attention score of the multi-head attention is obtained according to the traditional self-attention layer. The specific algorithm for obtaining the attention score is as follows:

[0119] ,

[0120] in, represents the attention score of multi-head attention, represents the query matrix, represents the key matrix, represents transpose, represents the dimension of the position encoding;

[0121] The attention score is linearly transformed, and the specific algorithm of the linear transformation is as follows:

[0122] ,

[0123] in, represents the attention score after linear transformation, represents a linear transformation;

[0124] The linearly transformed attention score is input into the transfer attention layer to obtain the attention-enhanced image features. The specific algorithm for obtaining the attention-enhanced image features is as follows:

[0125] ,

[0126] ,

[0127] in, represents the output features of the transfer attention layer, represents the activation function, represents the value matrix, represents the attention-enhanced image features, represents matrix multiplication, Represents the output features of the traditional self-attention layer;

[0128] According to the knowledge distillation algorithm, information interaction processing is performed on the action text features and the attention enhanced image features. The specific algorithm of the information interaction processing is as follows:

[0129] ,

[0130] ,

[0131] ,

[0132] in, Indicates information interaction processing of action text features. Indicates information interaction processing of attention-enhanced image features, Represents the action text features to be processed, represents the attention-enhanced image features to be processed, represents the feature ordinal number, represents the total number of knowledge distillation training pairs, represents the ordinal number of knowledge distillation training pairs, represents the cosine similarity function, represents transpose, is a learnable temperature parameter, The value range is [0,1], and the initial value is 0.5;

[0133] The knowledge distillation loss is calculated according to the KL divergence algorithm. The specific algorithm for calculating the knowledge distillation loss is as follows:

[0134] ,

[0135] ,

[0136] in, represents the knowledge distillation loss, represents the KL divergence algorithm, represents the predicted probability, Indicates authenticity;

[0137] According to the group relationship parameter adjustment module, feature relationship enhancement and parameter adjustment operations are performed to obtain relationship enhanced text features. The specific algorithm of the parameter adjustment operation is as follows:

[0138] ,

[0139] in, Represents the final output parameter, represents the initial parameters, represents the impact factor, and represents a randomly initialized matrix from a normal distribution, Represents the input features of the group relationship adjustment module.

[0140] Step S04: the key point branch sub-network extracts key point features from the target video image to obtain key point features, and then interactively enhances the key point features according to the interactive module to obtain interactively enhanced key point features;

[0141] It should be noted that in this embodiment, the key point branch sub-network includes a key point feature extraction module, and the key point feature extraction module includes a posture estimation network layer and an embedding layer;

[0142] The key point feature extraction module extracts key point features from the target video image. The key point feature extraction algorithm is as follows:

[0143] ,

[0144] in, represents key point features, represents the feed-forward layer, represents a cascade function, Feature vectors representing different joint types, represents the normalized and standardized spatial position encoding, Indicates absolute position encoding;

[0145] The key point features are interactively enhanced according to the interactive module to obtain interactively enhanced key point features. The specific algorithm for obtaining the interactively enhanced key point features is as follows:

[0146] ,

[0147] in, represents interactively enhanced key point features, represents the activation function, represents the self-attention enhancement operation, Represents the interaction between key points. represents the norm function, Represents node characteristics, Represents feature weight.

[0148] Step S05: performing group behavior recognition based on the relationship-enhanced image features, relationship-enhanced text features, and interaction-enhanced key point features to obtain a final classification result.

[0149] In summary, according to the above-mentioned interactive behavior recognition method based on multimodal group behavior recognition technology, a multimodal group behavior recognition network is designed, including an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and multimodal feature processing is performed respectively to avoid the problem of low recognition accuracy due to the interactive behavior between groups and the complex interactive environment when identifying through a single modal information. It further conforms to the actual maintenance situation and improves the overall recognition accuracy. Through lightweight design, the recognition efficiency is improved, and the relationship and interaction between groups and between groups and interactive targets are enhanced, which further improves the recognition accuracy and avoids the recognition obstacles caused by occlusion and overlap. The present invention improves the recognition accuracy and efficiency of group target interactive behaviors.Specifically, the target video image is collected in real time and preprocessed, and the preprocessed target video image is input into a multimodal group behavior recognition network. The multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork. The network structure between the image branch subnetwork, the text branch subnetwork and the key point branch subnetwork is a parallel structure, which combines multimodal information, enriches the recognition features, and further improves the recognition accuracy. The image branch subnetwork extracts features of the target video image according to a lightweight image feature processing module to obtain basic target group image features, and then performs feature relationship enhancement according to a group relationship enhancement module of time-space dual-stream interaction to obtain relationship-enhanced image features. The lightweight design speeds up the processing speed and improves the overall recognition efficiency. The text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operations on the basic target group image features according to a lightweight attention enhancement module to obtain attention-enhanced image features. The action text features and the attention-enhanced image features are processed for information interaction according to a knowledge distillation algorithm. The method comprises the following steps: processing the interactive text features to obtain interactive text features, and then performing feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features. The group relationship parameter adjustment module includes a relationship enhancement submodule and a parameter adjustment submodule. The parameters of the relationship enhancement submodule and the group relationship enhancement module are shared. According to the information interactive transmission between text and image features, repeated processing is avoided, the processing speed is improved, and the recognition efficiency is improved. At the same time, the relationship information between group targets is enhanced, the recognition obstacles caused by the complex relationship between target groups are avoided, and the recognition accuracy is improved. The key point branch subnetwork extracts key point features from the target video image to obtain key point features, and then interactively enhances the key point features according to the interactive module to obtain interactively enhanced key point features, which enhances the relationship between the target group and the interactive object, and further improves the recognition accuracy. According to the relationship enhanced image features, the relationship enhanced text features and the interactively enhanced key point features, group behavior recognition is performed to obtain the final classification result. The present invention improves the recognition accuracy and efficiency of group target interactive behaviors.

[0150] See also Figure 2 , which is a flow chart of an interactive behavior recognition method based on multimodal group behavior recognition technology proposed in the second embodiment of the present invention, the interactive behavior recognition method based on multimodal group behavior recognition technology includes steps S11 to S17, wherein:

[0151] Step S11: collecting target video images in real time and preprocessing them, and inputting the preprocessed target video images into a multimodal group behavior recognition network;

[0152] Step S12: The lightweight network layer extracts basic features and adjusts the coordinates of the feature map bounding box according to the spatial scaling algorithm. The bounding box feature alignment block aligns the adjusted bounding box coordinates with the extracted basic features to obtain basic target group image features.

[0153] Step S13: The group relationship enhancement module performs feature relationship enhancement based on the spatiotemporal dual-stream interaction algorithm, and performs feature fusion according to the final output features of the space-time branch path and the time-space branch path to obtain relationship enhancement image features;

[0154] Step S14: The text prompt module performs text prompt enhancement on the preset action label to obtain an enhanced action label, and then generates features for the enhanced action label according to the text coding block to obtain action text features. After obtaining the basic target group image features, the lightweight attention enhancement module performs attention enhancement operations, the traditional self-attention layer obtains the attention score of the multi-head attention, performs a linear transformation on the attention score, and inputs the attention score after the linear transformation into the transfer attention layer to obtain the attention enhanced image features;

[0155] Step S15: performing information interaction processing on the action text features and the attention-enhanced image features according to the knowledge distillation algorithm, calculating the knowledge distillation loss according to the KL divergence algorithm, and performing feature relationship enhancement and parameter adjustment operations according to the group relationship parameter adjustment module to obtain relationship-enhanced text features;

[0156] Step S16: The key point feature extraction module extracts key point features from the target video image, and interactively enhances the key point features according to the interactive module to obtain interactively enhanced key point features;

[0157] Step S17: performing group behavior recognition based on the relationship-enhanced image features, the relationship-enhanced text features, and the interaction-enhanced key point features to obtain a final classification result.

[0158] In summary, according to the above-mentioned interactive behavior recognition method based on multimodal group behavior recognition technology, a multimodal group behavior recognition network is designed, including an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and multimodal feature processing is performed respectively to avoid the problem of low recognition accuracy due to the interactive behavior between groups and the complex interactive environment when identifying through a single modal information. It further conforms to the actual maintenance situation and improves the overall recognition accuracy. Through lightweight design, the recognition efficiency is improved, and the relationship and interaction between groups and between groups and interactive targets are enhanced, which further improves the recognition accuracy and avoids the recognition obstacles caused by occlusion and overlap. The present invention improves the recognition accuracy and efficiency of group target interactive behaviors.Specifically, the target video image is collected in real time and preprocessed, and the preprocessed target video image is input into a multimodal group behavior recognition network. The multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork. The network structure between the image branch subnetwork, the text branch subnetwork and the key point branch subnetwork is a parallel structure, which combines multimodal information, enriches the recognition features, and further improves the recognition accuracy. The image branch subnetwork extracts features of the target video image according to a lightweight image feature processing module to obtain basic target group image features, and then performs feature relationship enhancement according to a group relationship enhancement module of time-space dual-stream interaction to obtain relationship-enhanced image features. The lightweight design speeds up the processing speed and improves the overall recognition efficiency. The text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operations on the basic target group image features according to a lightweight attention enhancement module to obtain attention-enhanced image features. The action text features and the attention-enhanced image features are processed for information interaction according to a knowledge distillation algorithm. The method comprises the following steps: processing the interactive text features to obtain interactive text features, and then performing feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features. The group relationship parameter adjustment module includes a relationship enhancement submodule and a parameter adjustment submodule. The parameters of the relationship enhancement submodule and the group relationship enhancement module are shared. According to the information interactive transmission between text and image features, repeated processing is avoided, the processing speed is improved, and the recognition efficiency is improved. At the same time, the relationship information between group targets is enhanced, the recognition obstacles caused by the complex relationship between target groups are avoided, and the recognition accuracy is improved. The key point branch subnetwork extracts key point features from the target video image to obtain key point features, and then interactively enhances the key point features according to the interactive module to obtain interactively enhanced key point features, which enhances the relationship between the target group and the interactive object, and further improves the recognition accuracy. According to the relationship enhanced image features, the relationship enhanced text features and the interactively enhanced key point features, group behavior recognition is performed to obtain the final classification result. The present invention improves the recognition accuracy and efficiency of group target interactive behaviors.

[0159] See also Figure 3 , which is a schematic diagram of the structure of an interactive behavior recognition system based on multimodal group behavior recognition technology proposed in the third embodiment of the present invention, and the system includes:

[0160] The acquisition module 10 is used to acquire the target video image in real time and perform preprocessing, and input the preprocessed target video image into the multimodal group behavior recognition network, wherein the multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and the network structure among the image branch subnetwork, the text branch subnetwork and the key point branch subnetwork is a mutually parallel structure;

[0161] An image feature processing module 20 is used for the image branch sub-network to extract features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, and then perform feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship enhanced image features;

[0162] A text feature processing module 30 is used for the text branch sub-network to obtain action text features according to preset action labels, and then perform an attention enhancement operation on the basic target group image features according to a lightweight attention enhancement module to obtain attention enhanced image features, and perform information interaction processing on the action text features and the attention enhanced image features according to a knowledge distillation algorithm to obtain interactive text features, and then perform feature relationship enhancement and parameter adjustment operations on the interactive text features according to a group relationship parameter adjustment module to obtain relationship enhanced text features, wherein the group relationship parameter adjustment module includes a relationship enhancement sub-module and a parameter adjustment sub-module, and the relationship enhancement sub-module shares parameters with the group relationship enhancement module;

[0163] A key point feature processing module 40 is used for the key point branch sub-network to extract key point features from the target video image to obtain key point features, and then interactively enhance the key point features according to the interactive module to obtain interactively enhanced key point features;

[0164] The classification module 50 is used to perform group behavior recognition according to the relationship-enhanced image features, the relationship-enhanced text features and the interaction-enhanced key point features to obtain a final classification result.

[0165] The present invention also proposes a computer storage medium on which one or more programs are stored, and when the program is executed by a processor, the interactive behavior recognition method based on the multimodal group behavior recognition technology is implemented.

[0166] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned interactive behavior recognition method based on multimodal group behavior recognition technology.

[0167] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0168] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0169] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0170] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0171] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An interactive behavior recognition method based on multimodal group behavior recognition technology, characterized in that: include: The target video image is collected in real time and preprocessed, and the preprocessed target video image is input into a multimodal group behavior recognition network, wherein the multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork and a key point branch subnetwork, and the network structure among the image branch subnetwork, the text branch subnetwork and the key point branch subnetwork is a mutually parallel structure; The image branch sub-network extracts features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, and then performs feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship-enhanced image features; The step of performing feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship enhanced image features specifically includes: The image branch subnetwork includes a group relationship enhancement module, and the group relationship enhancement module includes a space-time branch path and a time-space branch path; The space-time branch path includes a spatial attention residual submodule, and the spatial attention residual submodule includes a spatial attention block, a multi-layer perception block and a residual branch; The time-space branch path includes a time attention residual submodule, and the time attention residual submodule includes a time attention block, a multi-layer perception block and a residual branch; The group relationship enhancement module performs feature relationship enhancement based on a spatiotemporal dual-stream interaction algorithm; The spatiotemporal dual-stream interaction algorithm is specifically as follows: ， ， ， ， ， ， ， ， ， ， in, Represents a PE function, represents the PE function of even dimensions, represents the PE function of odd dimensions, represents the center coordinates of the bounding box, represents the feature dimension, represents the dimension of the position encoding, represents the input features of the space-time branching path, represents the input features of the time-space branching path, and Represent the PE function output features in the spatial attention block and the temporal attention block, respectively. and Represent the PE functions in the spatial attention block and the temporal attention block, respectively. and Represent the multi-head attention enhancement features in the spatial attention block and the temporal attention block, respectively. represents a linear layer, represents the multi-head attention mechanism layer, and Represent the final outputs of the spatial attention block and the temporal attention block, respectively. represents the feed-forward layer, represents the temporal attention residual submodule, represents the spatial attention residual submodule, represents a multi-layer perception block, and They represent the final output features of the space-time branch path and the time-space branch path respectively; Perform feature fusion according to the final output features of the space-time branch path and the time-space branch path to obtain relation-enhanced image features; The text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operations on the basic target group image features according to the lightweight attention enhancement module to obtain attention enhanced image features, and performs information interaction processing on the action text features and the attention enhanced image features according to the knowledge distillation algorithm to obtain interactive text features, and then performs feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features, and the group relationship parameter adjustment module includes a relationship enhancement submodule and a parameter adjustment submodule, and the relationship enhancement submodule shares parameters with the group relationship enhancement module; The key point branch sub-network extracts key point features from the target video image to obtain key point features, and then interactively enhances the key point features according to the interactive module to obtain interactively enhanced key point features; Group behavior recognition is performed based on the relationship-enhanced image features, the relationship-enhanced text features, and the interaction-enhanced key point features to obtain a final classification result.

2. The interactive behavior recognition method based on multimodal group behavior recognition technology according to claim 1, characterized in that: The image branch subnetwork extracts features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, specifically comprising: The image branch subnetwork includes a lightweight image feature processing module, wherein the lightweight image feature processing module includes a lightweight network layer and a bounding box feature alignment block; The lightweight network layer includes a depth-separable convolution block, an inverted residual block, and a squeeze excitation block to extract basic features, and adjust the feature map bounding box coordinates according to the spatial scaling algorithm. The algorithm relationship of the feature map bounding box coordinate adjustment is as follows: ， ， The bounding box feature alignment block performs an alignment operation on the adjusted bounding box coordinates and the extracted basic features to obtain basic target group image features.

3. The interactive behavior recognition method based on multimodal group behavior recognition technology according to claim 1, characterized in that: The text branch subnetwork obtains action text features according to preset action labels, and then performs attention enhancement operations on the basic target group image features according to the lightweight attention enhancement module to obtain the steps of attention enhanced image features, specifically including: The text branch sub-network performs text prompt enhancement on the preset action label according to the text prompt module to obtain an enhanced action label, and then generates features for the enhanced action label according to the text encoding block to obtain action text features; After acquiring the basic target group image features, an attention enhancement operation is performed according to a lightweight attention enhancement module, wherein the lightweight attention enhancement module includes a traditional self-attention layer and a transfer attention layer; The attention score of the multi-head attention is obtained according to the traditional self-attention layer. The specific algorithm for obtaining the attention score is as follows: ， in, represents the attention score of multi-head attention, represents the query matrix, represents the key matrix, represents transpose, represents the dimension of the position encoding; The attention score is linearly transformed, and the specific algorithm of the linear transformation is as follows: ， in, represents the attention score after linear transformation, represents a linear transformation; The linearly transformed attention score is input into the transfer attention layer to obtain the attention-enhanced image features. The specific algorithm for obtaining the attention-enhanced image features is as follows: ， ， in, represents the output features of the transfer attention layer, represents the activation function, represents the value matrix, represents the attention-enhanced image features, represents matrix multiplication, Represents the output features of the traditional self-attention layer.

4. The interactive behavior recognition method based on multimodal group behavior recognition technology according to claim 1, characterized in that: The step of performing information interaction processing on the action text features and the attention-enhanced image features according to the knowledge distillation algorithm to obtain interactive text features, and then performing feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship-enhanced text features specifically includes: According to the knowledge distillation algorithm, information interaction processing is performed on the action text features and the attention enhanced image features. The specific algorithm of the information interaction processing is as follows: ， ， ， in, Indicates information interaction processing of action text features. Indicates information interaction processing of attention-enhanced image features, Represents the action text features to be processed, represents the attention-enhanced image features to be processed, represents the feature ordinal number, represents the total number of knowledge distillation training pairs, represents the ordinal number of knowledge distillation training pairs, represents the cosine similarity function, represents transpose, is a learnable temperature parameter, The value range is [0,1], and the initial value is 0.5; The knowledge distillation loss is calculated according to the KL divergence algorithm. The specific algorithm for calculating the knowledge distillation loss is as follows: ， ， in, represents the knowledge distillation loss, represents the KL divergence algorithm, represents the predicted probability, Indicates authenticity; According to the group relationship parameter adjustment module, feature relationship enhancement and parameter adjustment operations are performed to obtain relationship enhanced text features. The specific algorithm of the parameter adjustment operation is as follows: ， in, Represents the final output parameter, represents the initial parameters, represents the impact factor, and represents a randomly initialized matrix from a normal distribution, Represents the input features of the group relationship adjustment module.

5. The interactive behavior recognition method based on multimodal group behavior recognition technology according to claim 1, characterized in that: The key point branch sub-network extracts key point features from the target video image to obtain key point features, specifically comprising: The key point branch sub-network includes a key point feature extraction module, and the key point feature extraction module includes a posture estimation network layer and an embedding layer; The key point feature extraction module extracts key point features from the target video image. The key point feature extraction algorithm is as follows: ， in, represents key point features, represents the feed-forward layer, represents a cascade function, Feature vectors representing different joint types, represents the normalized and standardized spatial position encoding, Indicates absolute position encoding.

6. The interactive behavior recognition method based on multimodal group behavior recognition technology according to claim 1, characterized in that: The step of interactively enhancing the key point features according to the interactive module to obtain interactively enhanced key point features specifically includes: The key point features are interactively enhanced according to the interactive module to obtain interactively enhanced key point features. The specific algorithm for obtaining the interactively enhanced key point features is as follows: ， in, represents interactively enhanced key point features, represents the activation function, represents the self-attention enhancement operation, Represents the interaction between key points. represents the norm function, Represents node characteristics, Represents feature weight.

7. An interactive behavior recognition system based on multimodal group behavior recognition technology, characterized in that: include: An acquisition module is used to acquire a target video image in real time and perform preprocessing, and input the preprocessed target video image into a multimodal group behavior recognition network, wherein the multimodal group behavior recognition network includes an image branch subnetwork, a text branch subnetwork, and a key point branch subnetwork, and the network structure among the image branch subnetwork, the text branch subnetwork, and the key point branch subnetwork is a mutually parallel structure; An image feature processing module is used for the image branch sub-network to extract features from the target video image according to the lightweight image feature processing module to obtain basic target group image features, and then perform feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship-enhanced image features; The step of performing feature relationship enhancement according to the spatiotemporal dual-stream interactive group relationship enhancement module to obtain relationship enhanced image features specifically includes: The image branch subnetwork includes a group relationship enhancement module, and the group relationship enhancement module includes a space-time branch path and a time-space branch path; The space-time branch path includes a spatial attention residual submodule, and the spatial attention residual submodule includes a spatial attention block, a multi-layer perception block and a residual branch; The time-space branch path includes a time attention residual submodule, and the time attention residual submodule includes a time attention block, a multi-layer perception block and a residual branch; The group relationship enhancement module performs feature relationship enhancement based on a spatiotemporal dual-stream interaction algorithm; The spatiotemporal dual-stream interaction algorithm is specifically as follows: ， ， ， ， ， ， ， ， ， ， in, Represents a PE function, represents the PE function of even dimensions, represents the PE function of odd dimensions, represents the center coordinates of the bounding box, represents the feature dimension, represents the dimension of the position encoding, represents the input features of the space-time branching path, represents the input features of the time-space branching path, and Represent the PE function output features in the spatial attention block and the temporal attention block, respectively. and Represent the PE functions in the spatial attention block and the temporal attention block, respectively. and Represent the multi-head attention enhancement features in the spatial attention block and the temporal attention block, respectively. represents a linear layer, represents the multi-head attention mechanism layer, and Represent the final outputs of the spatial attention block and the temporal attention block, respectively. represents the feed-forward layer, represents the temporal attention residual submodule, represents the spatial attention residual submodule, represents a multi-layer perception block, and They represent the final output features of the space-time branch path and the time-space branch path respectively; Perform feature fusion according to the final output features of the space-time branch path and the time-space branch path to obtain relation-enhanced image features; A text feature processing module, used for the text branch sub-network to obtain action text features according to preset action labels, and then perform attention enhancement operations on the basic target group image features according to the lightweight attention enhancement module to obtain attention enhanced image features, and perform information interaction processing on the action text features and the attention enhanced image features according to the knowledge distillation algorithm to obtain interactive text features, and then perform feature relationship enhancement and parameter adjustment operations on the interactive text features according to the group relationship parameter adjustment module to obtain relationship enhanced text features, wherein the group relationship parameter adjustment module includes a relationship enhancement sub-module and a parameter adjustment sub-module, and the relationship enhancement sub-module shares parameters with the group relationship enhancement module; A key point feature processing module, used for the key point branch sub-network to extract key point features from the target video image to obtain key point features, and then interactively enhance the key point features according to the interactive module to obtain interactively enhanced key point features; The classification module is used to perform group behavior recognition according to the relationship-enhanced image features, the relationship-enhanced text features and the interaction-enhanced key point features to obtain a final classification result.

8. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the interactive behavior recognition method based on multimodal group behavior recognition technology as described in any one of claims 1 to 6.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the interactive behavior recognition method based on multimodal group behavior recognition technology described in any one of claims 1 to 6 is implemented.

Images