Memory network enhancement method for experimental interactive behavior recognition and application

Abstract

The application discloses a memory network enhancement method and application for experimental interactive behavior recognition, belongs to the interactive behavior recognition technology of a person in video image recognition, uses a video-level graph network to extract global features of each category in a data set, and initializes memory items of a memory network as supplementary information to reduce intra-class distance; uses a region-level graph network to extract local features of a single video, captures detailed information of interactive behavior, and fuses the local features with global features in the memory network to enrich sample features; designs memory embedding loss to expand inter-class distance between similar samples, realizes accurate recognition of interactive behavior, and further improves model robustness. The application realizes accurate recognition of experimental interactive behavior categories in an experimental video, and the model has excellent overall performance.

Description

TECHNICAL FIELD

[0001] The application discloses a memory network enhancement method and application for experimental interactive behavior recognition, and belongs to the application of interactive behavior recognition technology in teaching practice. BACKGROUND

[0002] The existing interactive behavior recognition method can be divided into image-based interactive behavior recognition method and video-based interactive behavior recognition method according to the data processing type. The image-based interactive behavior recognition method refers to recognizing the interaction relationship between each pair of people and objects from the image. At present, this kind of method mainly understands the context information in the scene through multi-stream network or graph network, however, this method cannot capture the dynamic changes and time sequence information of actions, and is difficult to be applied to real scenes that need to detect continuous actions. The video-based interactive behavior recognition method can recognize and locate the time period of all interactive behaviors of people in a video and the specific interactive behavior categories. This kind of method constructs and analyzes the spatio-temporal interaction graph structure between people in the video, captures the spatio-temporal semantic information in the video frame by using graph network, recurrent neural network and other technologies, so as to recognize and locate the time period and interactive behavior category of the interactive behavior of people in the video.

[0003] At present, the interactive behavior recognition method based on people is mainly based on graph network. For example, in order to model the spatio-temporal correlation and long-term time dynamics of objects at the same time, Ning W, Guangming Z, Hongsheng L and others proposed STGC network, which captures the intra-frame and inter-frame dependency by capturing the spatial and temporal correlation at the same time. In order to further improve the model to capture the context information and complex relationship in the scene, Hong H S, Lee J C, Kumar A and others proposed DT-HOI by introducing extended text description as a supplementary input of visual features, which enhanced the recognition ability of the model for ambiguous actions. However, the existing video-based interactive behavior recognition method mainly starts from deep mining of the spatio-temporal relationship in the video, and improves the understanding ability of the model for interactive video by constructing the spatio-temporal network feature map of a single video, without considering the influence of the distribution of data on the model.

[0004] In addition, the current research on the problem of uneven data distribution is mainly based on image-based interactive behavior recognition, including methods based on training strategies and methods based on combined learning. Among them, the method based on training strategy refers to improving the performance of interactive behavior recognition through data enhancement and other strategies. Fang S, Liu S, Li J, etc. proposed to generate virtual images using the label-to-image generation method to solve the problem of uneven data distribution. The method based on combined learning forms new triplets by recombining human-object pairs and different representations of interaction to alleviate the data distribution problem. For example, Hou Z, Peng X, Qiao Y, etc. decomposed the HOI representation into object and verb specific features to form new interactive samples in the visual feature space, and in addition, the virtual-real fusion chemical experiment platform based on multi-modal perception for real-time feedback disclosed in the Chinese patent application No. CN202411636115.X, in the process of student experiment, the interactive behavior between the student and the experimental equipment is recognized by HOI to determine whether it is aligned with the key points in the preset experimental steps. Although the above methods have achieved good results in some application scenarios, they mainly focus on solving the interactive behavior data distribution problem in the image field, and few people have studied the recognition method based on video to judge the interactive action in the chemical experiment process.

[0005] In the real interactive scene of chemical experiments, there is often a situation of uneven distribution of class sample data, that is, the number of samples of some classes is much more than that of other classes, which leads the model to tend to learn data-rich samples, and the recognition accuracy of common behaviors is high, while the learning ability of data-scarce samples is limited, and the recognition ability of few-sample behaviors is insufficient, so that the model appears deviation in the recognition of intra-class homogeneity and inter-class heterogeneity in the optimization process. For example, in the eight chemical experiment scenes that students must do according to the chemistry curriculum standards of compulsory education in junior high school, some key actions (such as 'oscillation') only appear in a few experiments, and other common actions (such as 'pouring') appear frequently. The accurate recognition of key actions such as 'oscillation' which are data-scarce is very important for the subsequent evaluation of chemical experiment operation ability. Although the existing video-based interactive behavior recognition method has achieved good results, few people have studied the problem of uneven data distribution. Therefore, how to improve the recognition ability of scarce data under the condition of uneven data distribution and narrow the intra-class distance and expand the inter-class distance is a problem to be solved in the task of video-based interactive behavior recognition. SUMMARY

[0006] The technical problem solved by the present application is that the existing human interactive behavior recognition method has unstable recognition performance in the process of chemical experiments, and a memory network enhancement method for experimental interactive behavior recognition and application are provided.

[0007] The present application adopts the following technical solutions:

[0008] The present invention first discloses a memory network enhancement method for experimental interactive behavior recognition, comprising the following steps:

[0009] S1. Obtain a video dataset containing multiple experimental videos, construct a global video graph of the experimental videos through a video-level graph network, and extract the global features of each experimental interaction behavior category in all experimental videos in the video dataset;

[0010] S2. Build a frame-level local interaction graph of the experimental video through the region-level graph network to obtain the local features of a single experimental video.

[0011] S3. Use global features to initialize the memory items of the memory network, project local features into the feature space of the memory items through the attention mechanism, and enhance them by fusing with global features. Finally, use memory embedding loss to expand the inter-class distance of similar experimental interaction behaviors between memory items.

[0012] In the memory network enhancement method for experimental interactive behavior recognition provided by the present invention, further, in step S1, the video-level graph network uses BiRNN to process the time series of all experimental video frame-level features to obtain the visual features of each experimental video, and uses the visual features to instantiate the video nodes of the experimental video, and connects every two video nodes to obtain an initialized global video graph.

[0013] In the memory network enhancement method for experimental interaction behavior recognition provided by the present invention, further, in step S1, the global feature extraction step of each experimental interaction behavior category is as follows:

[0014] First, the video nodes of the global video graph are updated through the graph attention network, and the attention coefficient a between the video nodes is obtained through the attention mechanism. ij ;

[0015] Secondly, the final feature x of a single experimental video is obtained by updating the attention coefficient i ',

[0016] ,

[0017] Among them, x i is the video node feature, W video is the attention mechanism weight matrix, σ is the activation function;

[0018] Finally, after updating the visual features of all experimental videos using the graph attention network, the visual features of the experimental videos are grouped according to the label information of the experimental interaction behavior. The global features of each experimental interaction behavior category are calculated using the following formula:

[0019] ,

[0020] where c k is the global feature of the kth experimental interaction behavior class, N video is the number of all experimental videos, y i is the visual feature of the ith experimental video, and the global feature dataset C of all experimental interaction behavior classes is obtained, C={c k} k=1,...,K , where K represents the number of experimental interaction behavior classes.

[0021] In the memory network enhanced method for experimental interaction behavior recognition provided by the application, further, in step S2, the regional level graph network uses FasterR-CNN or YOLOv8 to locate the instances of operators or experimental equipment on each frame picture of the experimental video, splices the visual features, spatial features and semantic features of all instances, constructs the frame level local interaction graph of the experimental video, and connects the spliced features in the frame and between frames to obtain the local features of a single experimental video.

[0022] In the memory network enhanced method for experimental interaction behavior recognition provided by the application, further, the instance located by the FasterR-CNN has a four-dimensional coordinate frame and a target class,

[0023] Each instance is cropped from the original experimental video frame according to its four-dimensional coordinate frame to obtain a region of interest, and is input into a pre-trained ResNet-50 model to extract visual features F v ,

[0024] The four-dimensional coordinates of each instance are converted into spatial features F s of the corresponding dimension using a multi-layer perception (MLP).

[0025] The semantic features F w of the instance are generated using a learnable word embedding model.

[0026] In the memory network enhanced method for experimental interaction behavior recognition provided by the application, further, the frame level local interaction graph includes a set of operator or experimental equipment nodes in the frame picture and a set of operator or experimental equipment edges in the frame picture, the edge weight of the edge connected by the operator and the experimental equipment is initialized to 1, and the rest is 0, the intra-frame edge connection graph and the inter-frame edge connection graph are analyzed using a graph attention network to obtain the intra-frame edge connection graph attention coefficient and the inter-frame edge connection graph attention coefficient The final local features of a single experimental video are connected by the intra-frame and inter-frame spliced features through the following formula ,

[0027] ,

[0028] wherein W inra and W inter are intra and inter edge weight matrices, H represents the concatenation of visual, spatial and semantic features of instances, and [] represents the concatenation operation.

[0029] In the memory network enhancement method for experimental interactive behavior recognition provided by the application, further, in the step S3, the process of fusion and enhancement of the local features and the global features is as follows:

[0030] S31, constructing a memory network M using the global features of all experimental videos, the memory network having K memory items corresponding to the number of categories of experimental interactive behaviors, each memory item being initialized with the global feature of a corresponding category of experimental interactive behavior;

[0031] S32, inputting the local features of a single experimental video into the memory network, projecting the local features to the feature space of the memory items through an attention mechanism method to obtain new feature representations ;

[0032] S33, enhancing the projected local features by the memory items through the following formula,

[0033]

[0034] wherein, is the local feature projected to the feature space of the memory items, is the enhanced feature of the memory item, c k is the global feature of the kth category of experimental interactive behavior, n j represents the global feature corresponding to the jth memory item of the memory network;

[0035] S34, obtaining the predicted category label of the experimental interactive behavior through a softmax function.

[0036] In the memory network enhancement method for experimental interactive behavior recognition provided by the application, further, in the step S3, the cosine similarity loss L sim and the Euclidean distance loss L euc are combined as the memory embedding loss L mem of the memory network through the following formula;

[0037]

[0038] wherein λ is a hyperparameter;

[0039] The cosine similarity loss L sim is obtained by minimizing the cosine similarity between the experimental interactive behavior features of different memory items through L2 norm.​​

[0040] ,

[0041] wherein, s ij represents the cosine similarity between memory item n i and memory item n j , s is the similarity matrix of all cosine similarities between memory items of the memory network, and N is the number of memory items of the memory network;

[0042] The Euclidean distance between the memory items is calculated to obtain a distance matrix of all Euclidean distances between the memory items of the memory network, and the elements of each row of the distance matrix are sorted in ascending order to obtain D'=(d ij ')∈R N×N , d ij ' represents the Euclidean distance between memory item n i and memory item n j after sorting, and the k2 smallest values of each row of the distance matrix are selected to increase the Euclidean distance between the memory items by the following formula,

[0043] ,

[0044] wherein, d — is the increased Euclidean distance, the Euclidean distance loss L euc =max(0,-d — +γ), and γ is a hyperparameter for adjusting the distance between the memory items.

[0045] The application also discloses an experimental interactive behavior recognition method, wherein the memory network enhanced by the memory network enhancement method of the application is input with an experimental video to be recognized, and a memory item corresponding to an experimental interactive behavior is retrieved from the memory network in combination with the current input video data, and the category of the experimental interactive behavior in the experimental video to be recognized is output.

[0046] Further, the application also discloses an experimental interactive behavior recognition system, comprising:

[0047] an input module for inputting an experimental video to be recognized;

[0048] a memory module for storing the memory network enhanced by the memory network enhancement method of the application;

[0049] a query module for retrieving a memory item corresponding to an experimental interactive behavior from the memory network in combination with the current input video data;

[0050] an output module for outputting the category of the experimental interactive behavior in the experimental video to be recognized.

[0051] The application has the following beneficial effects:

[0052] (1) The application firstly uses a video-level graph network to extract the global features of each experimental interaction category in the experimental video data set, and initializes the memory items of the memory network as supplementary information to reduce the intra-class distance; secondly, a region-level graph network is used to extract the local features of a single experimental video, capture the detailed information of the experimental interaction behavior, and fuse it with the global features in the memory network to enrich the sample features; finally, a memory embedding loss is designed to expand the inter-class distance between similar samples, the global features and local features are obtained by constructing a global video graph and a frame-level local interaction graph, the global features and local features are fused by designing a memory network, the context information of the experimental interaction behavior of the scarce data is supplemented to reduce the intra-class distance, the memory embedding loss is introduced to solve the uneven distribution of the interactive action of the characters in the experimental video, and the accurate recognition of the experimental interaction behavior is realized, thereby improving the robustness of the experimental interaction behavior recognition model.

[0053] (2) The application adopts a global feature and local feature fusion framework, and the existing method mainly focuses on the construction of the spatio-temporal relationship inside a single experimental video, and does not supplement the context information of the rare category interaction behavior from a global perspective; the global features and local features of the interactive behavior of characters in the experimental video are respectively extracted by independent global feature branches (video-level graph network) and local feature branches (region-level graph network); the global feature branch models all experimental videos by using a video-level graph network, excavates the relationship between different video nodes by a graph attention network, updates the node features, and groups the visual features according to the label information of the experimental interaction behavior, so as to obtain the global features of each interactive behavior category, aggregate the sample features of the entire video data set, excavate the internal connection and rule of the video data set, and enrich the experimental interaction behavior feature representation of the model. The local feature branch constructs a frame-level local interaction graph by using a region-level graph network, extracts the local features of a single experimental video, including the visual features, spatial features and semantic features of all instances, and uses a graph attention network to analyze the intra-frame edge connection graph and the inter-frame edge connection graph, so as to capture the overall semantics of the frame-level interactive behavior. The memory items in the memory network are initialized as the global features of each category of experimental interaction behavior, and when the local features of a single experimental video are input, the local visual features are projected into the feature space of the memory items by a multilayer perceptron, the memory items are used to enhance the features, so that the final visual features fuse the global features and local features in the entire data set similar to them, especially the context information of the sample rare interactive behavior is enriched. The two are synergistically enhanced, which makes up for the defect that the local features are difficult to handle similar categories, and effectively copes with the challenge of uneven distribution of experimental interactive behavior sample data in the experimental video.

[0054] (3) To solve the problem of large similarity of part of the interactive behavior semantics and small feature distance between different categories in the identification experiment, the memory network is introduced to store the prototype features of each category of experimental interactive behavior in the data set, and the cosine similarity loss and Euclidean distance loss are combined as the memory embedding loss, the cosine similarity loss is used to minimize the cosine similarity between the memory items, so that each pair of memory items is distinguished in the semantic space; the Euclidean distance loss is used to further expand the distance of the memory items in the Euclidean space, to ensure that each memory item maintains a sufficient distance from other memory items, thereby expanding the inter-class distance of similar experimental interactive behaviors and improving the discrimination ability of the identification model for similar experimental interactive behaviors. Through the memory embedding loss, the inter-class distance between similar experimental interactive behavior samples is expanded, the understanding ability of the model for data scarce experimental interactive behaviors is enhanced, and in the training, resources can be redistributed for data scarce experimental interactive behavior samples and data rich experimental interactive behavior samples, better feature distinction is performed, so that the discrimination ability of the identification model for similar experimental interactive behaviors is significantly improved, and the overall performance of the identification model is optimized.

[0055] In summary, the memory network enhancement method for experimental interactive behavior identification and application provided by the present application realizes accurate identification of experimental interactive behavior categories in experimental videos, and the overall performance of the model is excellent, and can be used for video automatic identification of teaching experiment examination of various science and engineering disciplines.

[0056] The present application will be further described below in conjunction with the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The flowchart of the memory network enhancement method for experimental interactive behavior identification in the embodiment.

[0058] Figure 2 The sample distribution table of the CE-HIRD data set in the embodiment.

[0059] Figure 3 The sample distribution table of the CAD120 data set in the embodiment.

[0060] Figure 4 The sample distribution table of the Something-else data set in the embodiment.

[0061] Figure 5 The confusion matrix diagram of the chemical experimental interactive behavior identification of the baseline model (left) and the present embodiment (right) about the CE-HIRD data set.

[0062] Figure 6 The confusion matrix diagram of the object function availability of the baseline model (left) and the present embodiment (right) about the CE-HIRD data set.

[0063] Figure 7 Baseline model feature visualization chart.

[0064] Figure 8 Baseline model feature visualization chart. DETAILED DESCRIPTION

[0065] EMBODIMENT

[0066] Reference Figure 1 With the processing of chemical experiment interaction behavior in chemical experiment video as an example, the global feature branch and the local feature branch are used in the embodiment to process the chemical experiment interaction behavior features in the chemical experiment video, the memory network for chemical experiment interaction behavior recognition is data enhanced, and the specific steps include the following steps:

[0067] S1, a video data set containing multiple chemical experiment videos is obtained, the global feature branch first constructs a global video graph of the chemical experiment video through a video-level graph network, extracts the global features of each chemical experiment interaction behavior class in all chemical experiment videos in the video data set, and initializes the memory items of the memory network with the global features as supplementary information to reduce the intra-class distance between the chemical experiment interaction behaviors.

[0068] The global feature can provide more comprehensive chemical video data information and context knowledge. By introducing the global feature, more comprehensive data information and context knowledge can be obtained, the intra-class distance between the chemical experiment interaction behavior features caused by the difference between the local features is reduced, and the understanding ability of the data scarce interaction behavior class is improved. Therefore, the global feature branch is introduced in the method, the global features of each class in the data set are extracted by constructing the global video graph of the chemical experiment video. Specifically, in the global feature branch, the video-level graph network of the embodiment processes the time sequence of all chemical experiment frame-level features by using BiRNN, obtains the visual features Y={y i}i=1,...,N video , N video is the number of all chemical experiment videos in the data set, and a global video graph G video =(V video ,E video ) of the chemical experiment video is constructed, wherein V video represents a node set of the chemical experiment video, V video ={x i |i=1,2,...,N video}, and each video node of the chemical experiment video is instantiated by a single visual feature y i , E video represents an edge connecting different video nodes, and each two video nodes are connected to obtain an initialized global video graph with full connectivity.

[0069] The extraction step of the global feature of each chemical experiment interactive behavior category in this step is as follows:

[0070] Firstly, the video node of the global video graph is updated by the graph attention network. The graph attention network (GAT) refers to the weighted summation of the adjacent video node features by the attention mechanism. The weight of the adjacent video node feature depends entirely on the node feature and is independent of the graph structure. The attention coefficient a between the video nodes is obtained by the attention mechanism. ij The specific calculation formula is as follows:

[0071] .

[0072] Where W video is the attention mechanism weight matrix, [x i , x j ] means that the feature vectors of the video nodes x i and x j are spliced and then multiplied by the attention mechanism weight matrix W, LeakyReLU is a nonlinear activation function, and softmax is a normalized activation function used to map the attention coefficient to the range of [0, 1].

[0073] Secondly, the graph attention network updates the final feature x i ’ of a single chemical experiment video through the attention coefficient,

[0074] .

[0075] Where x i is the video node feature, W video is the attention mechanism weight matrix, and σ is the activation function.

[0076] Finally, after updating the visual features of all chemical experiment videos by the graph attention network, the visual features of the chemical experiment videos are grouped according to the label information of the chemical experiment interactive behaviors. The global feature of each chemical experiment interactive behavior category is calculated by the following formula,

[0077] .

[0078] Where c k is the global feature of the kth chemical experiment interactive behavior category, N video is the number of all chemical experiment videos, and y i is the visual feature of the ith chemical experiment video. The global feature dataset C of all chemical experiment interactive behavior categories is obtained, C={c k} k=1,...,Kwhere K represents the number of chemical experiment interaction behavior categories. At this time, the global feature dataset C aggregates the sample features of the entire chemical experiment dataset to mine the internal relationship and rule of the chemical experiment dataset and enrich the feature representation of the model.

[0079] The global feature extraction models all chemical experiment videos using a video-level graph network, mines the relationship between different video nodes through a graph attention network, updates the node features, and groups the visual features according to the label information to calculate the global features of each interaction behavior category, aggregate the sample features of the entire dataset, mine the internal relationship and rule of the dataset, and enrich the feature representation of the model.

[0080] S2, the local feature branch constructs a frame-level local interaction graph of the chemical experiment video through a region-level graph network to obtain the local features of a single chemical experiment video, captures the detailed information of the interaction behavior, and inputs the local features into a memory network to fuse and enhance the global features to enrich the sample features.

[0081] The global feature branch mainly focuses on the features of the entire dataset, while the local feature branch extracts the frame-level features of a single chemical experiment video to capture the detailed information of the interaction behavior, making up for the deficiency of the global features in describing the details of the specific interaction behavior in a single chemical experiment video. In this step, the region-level graph network uses FasterR-CNN or YOLOv8 to locate the instances of operators or chemical experiment equipment on each frame of the chemical experiment video, extracts the visual features, spatial features, and semantic features of all instances for splicing, constructs a frame-level local interaction graph of the chemical experiment video, and connects the spliced features within and between frames to obtain the local features of a single chemical experiment video.

[0082] This embodiment takes FasterR-CNN as an example. The FasterR-CNN locates the instances of operators or chemical experiment equipment with a four-dimensional coordinate frame and a chemical target category. Each instance crops the region of interest from the original chemical experiment video frame according to its four-dimensional coordinate frame and inputs it into a pre-trained ResNet-50 model to extract visual features F v , uses a multi-layer perceptron MLP to convert the four-dimensional coordinates of each instance into spatial features F s of the corresponding dimension, and uses a learnable word embedding model to generate semantic features F w of the instance.

[0083] Specifically, given a chemical experiment video {X m ,cm} m=1,...,M , where X m ={ } t=1,...,TT represents the total number of frames of the video, M represents the number of video nodes of operators or chemical experiment equipment in the experimental video, c m represents the interactive behavior or object function participation label of the corresponding video node, that is, the label information of the chemical experiment interactive behavior in the corresponding video node. Through the FasterR-CNN target detection network, the interactive behavior or object function participation label of the corresponding video node is obtained in each video frame picture The instances of locating the operators or the chemical experiment equipment are as follows:

[0084] .

[0085] Each instance has a four-dimensional coordinate frame and a chemical target class, where T is the chemical target class of the instance, B is the four-dimensional coordinate frame (x, y, W, H) of the instance, x and y are the horizontal and vertical coordinates of the center point of the four-dimensional coordinate frame, W and H are the width and height of the four-dimensional coordinate frame, RPN refers to a region proposal network, which aims to generate a set of candidate regions, ROIPooling refers to region of interest pooling to map each candidate region to a fixed-size feature vector, FC refers to a fully connected layer, CNN ) refers to inputting the t-th frame image of the m-th node into a convolutional neural network CNN. .

[0086] For the chemical experiment visual features in the extracted local features, the embodiment crops the region of interest (Region of Interest, ROI) from the original video frame according to the four-dimensional coordinate frame of the operator or the chemical experiment equipment in each video frame picture, and inputs it into the pre-trained ResNet-50 model to extract the visual feature F v , and the calculation formula is as follows:

[0087] .

[0088] , where represents the t-th video frame picture of the m-th chemical experiment video, Crop refers to cropping the target region corresponding to the four-dimensional coordinates (x, y, W, H) of the frame picture. .

[0089] For the spatial features in the extracted local features, the embodiment uses a multilayer perceptron (Multilayer Perceptron, MLP) to convert the four-dimensional coordinates of the instance into spatial features F s of the corresponding dimension to represent the position information between the operators and the chemical experiment equipment.

[0090] For the semantic features in the extracted local features, the embodiment generates the semantic features F wto represent the specific category information of students or chemical experiment equipment. The word embedding model can adopt a Linear128-BatchNorm-ReLU-Linear256-BatchNorm-ReLU model.

[0091] Next, the above obtained visual feature F v , spatial feature F s and semantic feature F w are concatenated (Concat) by the following formula:

[0092] .

[0093] H is the concatenated feature of the visual feature, spatial feature and semantic feature of the instance.

[0094] Next, a frame-level local interaction graph G local of the chemical experiment video is constructed to explicitly model the human-object interaction relationship, which is defined as G local = (V ho , E ho ), where V ho represents a set of operator or chemical experiment equipment nodes in the frame picture, E ho represents a set of operator or chemical experiment equipment edges in the frame picture, and E ho is composed of intra-frame edges E and inter-frame edges E , the intra-frame edge E is composed of edges between different operators or chemical experiment equipment in the same frame t, and the inter-frame edge E represents edges between the same operator or chemical experiment equipment in two frames t and t+1 that are apart, and the edge weight of the edge connected between the operator and the chemical experiment equipment is initialized to 1, and the rest is 0. The intra-frame edge connection graph and the inter-frame edge connection graph are parsed using a graph attention network to obtain the intra-frame edge connection graph attention coefficient and the inter-frame connection graph attention coefficient , and the final local feature of a single chemical experiment video is obtained by connecting the intra-frame and inter-frame concatenated features by the following formula: , which captures the overall semantics of the frame-level interaction behavior,

[0095] .

[0096] where W inra and W inter are the intra-frame edge weight matrix and the inter-frame edge weight matrix, H represents the concatenated feature of the visual feature, spatial feature and semantic feature of the instance, and [] represents the connection operation.

[0097] The local feature extraction constructs a frame-level local interaction graph of the chemical experiment video through a region-level graph network, extracts local features of a single video, including visual features, spatial features and semantic features of all instances, and parses the intra-frame edge connection graph and the inter-frame edge connection graph using a graph attention network to capture the overall semantics of the frame-level interaction behavior.

[0098] S3, initializing the memory items of the memory network with the global features, inputting the local features into the memory network, projecting the local features to the feature space of the memory items through an attention mechanism, fusing and enhancing the local features with the global features, expanding the inter-class distance of similar chemical experiment interaction behaviors between the memory items through a memory embedding loss, realizing accurate recognition of the interaction behaviors in the chemical experiment video, and further improving the robustness of the model.

[0099] In the step S3, the fusion and enhancement process of the local features and the global features is as follows:

[0100] S31, in order to compensate for the context information of the data-scarce interaction behaviors and sufficiently fuse the global features and the local features of the chemical experiment video, a memory network is constructed using the global features of all chemical experiment videos to enrich the features of the data-scarce interaction behaviors, and the memory network M={c k} k=1,...,K has N memory items corresponding to the number of K chemical experiment interaction behavior categories, and each memory item is initialized with the global feature c k of the corresponding chemical experiment interaction behavior category.

[0101] S32, inputting the local features of a single chemical experiment video into the memory network, when the local features of the single chemical experiment video are inputted , the memory network projects the local features to the feature space of the memory items through an attention mechanism method to obtain new feature representation . In this embodiment, the calculation formula of the multi-layer perception projection of the local features is as follows:

[0102] .

[0103] Wherein, MLP refers to multi-layer perception.

[0104] S33, the memory items enhance the projected local features by the following formula,

[0105] .

[0106] Wherein, is the local feature projected to the feature space of the memory items, is the enhanced feature of the memory items, and c kn is the global feature of the kth chemical experiment interaction behavior category j represents the global feature corresponding to the jth memory item of the memory network, where j is the index of the memory item, and the value range is 1 to N, where N is the total number of memory items in the memory network, that is, the number of interaction behavior categories.

[0107] S34, obtain the predicted label of the chemical experiment interaction behavior category through the softmax function, that is, the result label of the chemical interaction action prediction, corresponding to the finally predicted chemical experiment interaction action category such as "oscillation".

[0108] In the above manner, the final visual feature f integrates the global feature and the local feature similar to it in the entire chemical experiment video dataset. Especially for the sample-scarce interaction behavior, the memory network enriches the context information thereof, and improves the understanding ability of the model to the sample-scarce interaction behavior.

[0109] The memory items in the memory network are initialized as the global features of various interaction behaviors in the chemical experiment video. When the local feature of a single chemical experiment video is input, the local feature of the video is projected into the feature space of the memory item through the multilayer perceptron, and the local feature is enhanced by the memory item, so that the final visual feature f integrates the global feature and the local feature similar to it in the entire video dataset, and especially enriches the context information of the sample-scarce experimental interaction behavior.

[0110] In order to further solve the problem that the semantic similarity between part of the interaction behaviors is large and the feature distance between different categories is small, the embodiment introduces a memory embedding loss to expand the inter-class distance of similar interaction behaviors in the chemical experiment video, and improve the discrimination ability of the model to the similar chemical experiment interaction behaviors. Specifically, the embodiment combines the cosine similarity loss L sim and the Euclidean distance loss L euc as the memory embedding loss L mem of the memory network.

[0111] For the cosine similarity loss L sim , first calculate the cosine similarity between the memory items of similar chemical experiment interaction behaviors, and obtain a similarity matrix s, the calculation formula is as follows:

[0112] .

[0113] Wherein, is the normalized memory item matrix, where each row represents a feature vector of a memory item, represents the transpose of s , s ij represents the memory item n i , and the memory item n jcosine similarity between memory item n sim ,

[0114] .

[0115] where s is the similarity matrix of all cosine similarities between memory items of the memory network, || ||2is the L2 norm calculation, s ij represents the cosine similarity between memory item n i and memory item n j , and N is the number of memory items of the memory network.

[0116] However, it is not enough to distinguish memory items by only minimizing cosine similarity, some memory items still cannot be effectively distinguished from other features, resulting in similar chemical experiment interaction behaviors being difficult to be effectively distinguished. The embodiment further expands the distance between memory items in Euclidean space, so that the model can further distinguish their features, and expand the distance between classes of chemical experiment interaction behaviors. The Euclidean distance between two memory items is calculated by the following formula,

[0117] .

[0118] where d ij represents the Euclidean distance between memory item n i and memory item n j . By calculating the Euclidean distance between memory items, a distance matrix D of all Euclidean distances between memory items of the memory network is obtained, and each row element of the distance matrix is sorted in ascending order to obtain D'=(d ij ')∈R K×K , d ij ' represents the Euclidean distance between memory item n i and memory item n j after sorting, where N represents the total number of memory items in the memory network, i.e. the number of interaction behavior categories. Each memory item corresponds to an interaction behavior category, so the rows and columns of matrix D represent the pairwise Euclidean distances between all memory items, and the sorted matrix D' is still N×N, where the elements of each row are arranged in ascending order. The Euclidean distance between memory items is increased by selecting the k2 smallest values of each row of the distance matrix by the following formula,

[0119] .

[0120] where d — is the increased Euclidean distance, and the Euclidean distance loss L euc =max(0,-d —+ γ), γ is a hyperparameter for adjusting the distance between memory items, k2 is an integer less than N, and the value of k2 needs to be determined by experiment according to the specific situation. If the semantic similarity in the data set is relatively high, a larger value can be selected from the integer less than K, and if the semantic similarity in the data set is relatively low, a smaller integer value is selected. In this embodiment, γ is 0.8 and k2 is 3.

[0121] The cosine similarity loss L sim and the Euclidean distance loss L euc The memory embedding loss L of the memory network is obtained mem，

[0122] .

[0123] where λ is a hyperparameter, and in this embodiment, the model performs best when λ is 0.6. Through the update and iteration of the memory network, the existing memory items represent the prototype features of each type of interaction behavior, and by constraining the feature distance of the memory items, the distance between different chemical experiment interaction behavior features can be effectively limited.

[0124] The memory embedding loss is composed of the cosine similarity loss and the Euclidean distance loss. The cosine similarity loss is used to minimize the cosine similarity between memory items, so that each pair of memory items is distinguished in the semantic space. The Euclidean distance loss is used to further expand the distance of the memory items in the Euclidean space, ensuring that each memory item maintains a sufficient distance from other memory items, thereby expanding the inter-class distance of similar experimental interaction behavior features and improving the model's ability to distinguish similar experimental interaction behaviors.

[0125] The embodiment also discloses a method for identifying chemical experiment interaction behaviors by using the above memory network enhancement method. The memory network enhanced by the above memory network enhancement method is input to the chemical experiment video to be identified, and the memory items corresponding to the chemical experiment interaction behaviors are retrieved from the memory network in combination with the current input video data, and the chemical experiment interaction behavior category in the identified chemical experiment video is output.

[0126] The memory network constructed by the memory network enhancement method of the embodiment is applied to a chemical experiment interaction behavior recognition method, a video-level graph network is used to extract global features of each category in the data set, and the memory items of the memory network are initialized as supplementary information to reduce the intra-class distance; a region-level graph network is used to extract local features of a single video, capture detailed information of the interaction behavior, and fuse the global features in the memory network to enrich the sample features; the independent video-level graph network and the region-level graph network extract global features and local features of the chemical experiment video respectively, and synergize the two to make up for the defect that the local features are difficult to handle similar categories, and effectively cope with the challenge of uneven data distribution. Finally, a memory embedding loss is designed to expand the inter-class distance between similar samples, and a cosine similarity loss and an Euclidean distance loss are combined as the memory embedding loss to constrain the feature distance of the memory items, so that the discrimination ability of the model to similar interaction behaviors is significantly improved, the accurate recognition of the interaction behavior is realized, and the robustness of the model is improved. The operator and the chemical experiment equipment interaction instance in the chemical experiment video can be accurately predicted, and the problem of uneven data distribution is solved, so that the model tends to learn data-rich samples, the learning ability of data-scarce samples is enhanced, the deviation in the recognition process is avoided, and the accuracy of the chemical experiment interaction behavior recognition is improved.

[0127] Correspondingly, the embodiment also provides a system for implementing the above chemical experiment interaction behavior recognition method, which specifically includes: an input module, a memory module, a query module and an output module, wherein the input module is used to input a chemical experiment video to be recognized; the memory module stores a memory network enhanced by the memory network enhancement method; the query module retrieves the memory items of the corresponding chemical experiment interaction behavior from the memory network in combination with the current input video data; and the output module outputs the chemical experiment interaction behavior category retrieved in the chemical experiment video to be recognized.

[0128] The chemical experiment interaction behavior recognition method of the embodiment can be used for video recognition and assessment of students' chemical experiment interaction behavior in chemical teaching experiments, and can also be used for video assessment of other science and engineering teaching experiments, which will not be described here.

[0129] The performance of the embodiment is verified by specific data sets as follows.

[0130] Three data sets were used in this embodiment for experiments, namely CE-HIRD data set, CAD120 data set and Something-else data set. Among them, the CE-HIRD data set is a self-built chemical experiment interactive behavior recognition data set, and the CAD120 and Something-else data sets are public data sets. The data sets selected in this paper are all interactive behavior data sets in real scenarios. After statistical analysis of the labeled categories of the data sets, it is found that the data sets all present obvious uneven data distribution.

[0131] Specifically, as Figure 2As shown, the CE-HIRD dataset is a public dataset of human interaction behaviors in a student chemistry experiment scene, containing 160 long videos that capture 20 human subjects performing 8 experimental activities in a chemistry experiment scene. According to the eight major experiments in the Chemistry Curriculum Standard for Compulsory Education, this paper simulates the real chemistry experiment table scene and recruits 20 middle school students as experimenters to shoot a large-scale chemistry experiment video data of 682 minutes. Due to the large amount of labeling information, a semi-automatic labeling method is used to ensure the quality of the dataset and significantly reduce the time cost. Specifically, this paper first uses the ffmpeg tool to frame the chemistry experiment video at a frame rate of 25FPS, with a resolution of 1920x1080 for each frame, totaling 204600 frames. Second, the target class and position of each frame are labeled, and the specific target class includes beaker, narrow-mouth bottle, test tube, wide-mouth bottle, dropper, alcohol lamp, glass rod, glass sheet, bottle plug, iron stand, tweezers, test tube brush, ph test paper, medicine spoon, mortar, funnel, measuring cylinder, and person, totaling 18 categories of target data. After labeling 3000 frames, a chemistry experiment target detection model is trained using the yolov8 algorithm, and the model is used to identify and locate the student's position, equipment category and position in subsequent video frames. Since the target class index obtained above does not correspond within a video frame, this paper subsequently uses the bytetrack target tracker to track and align the detected target class within the same video frame. In terms of interaction behavior labeling, this paper uses the target position information obtained above to crop the target region frame sequence. If the target region class is a person, the interaction behavior class is labeled, otherwise the object function participation target is labeled. Among them, the interaction behavior class mainly includes pouring, putting down, picking up, standing, shaking, stirring, covering and inverting, totaling 8 categories. The object function participation class has 9 categories, mainly including stationary, pourable, put-downable, pick-upable, standable, shakable, stirrable, coverable and invertible, and the specific labeling file structure is [video_id, seg_frames, label]. Among them, video_id represents the video number, seg_frames represents the starting frame of the behavior sequence corresponding to the video, and label includes the interaction class information of the person and the function participation class information of the equipment.

[0132] As shown in Figure 3 , the CAD120 dataset is a public video dataset recording human interaction behaviors, containing 120 RGB-D long videos that capture 4 human subjects performing 10 activities in daily indoor activities, such as making oatmeal, etc., where each activity consists of a sequence of interaction behavior video clips. This dataset contains 10 types of interaction behaviors and 12 types of object function availability commonly seen in daily life, as shown in Figure 3 .

[0133] As Figure 4 shown, the Something-else dataset is a larger-scale public dataset, containing 112795 videos, covering 174 different interactive behavior categories, and the sample distribution is shown in Figure 4 The dataset is created by expanding the SomethingV2 dataset, adding additional hand and object bounding boxes, aiming to recognize combined actions to identify activities and objects simultaneously.

[0134] For the CE-HIRD chemical dataset and the CAD120 public dataset, the sub-activity F1 value and the affordence F1 value, that is, the interactive behavior F1 value and the object function availability F1 value, are used to evaluate the model effect. The F1 value is the harmonic mean of the precision P and the recall R, which can comprehensively evaluate the accuracy and recall ability of the model. For the Something-else public dataset, the Top-1 and Top-5 accuracy are used to evaluate the performance of the method in the experiment, wherein the Top-1 accuracy indicates that the predicted probability of the highest interactive behavior is the real interactive behavior category, representing correct prediction; the Top-5 accuracy indicates that the real interactive behavior category is contained in the top five predicted interactive behaviors, representing correct prediction.

[0135] The CE-HIRD dataset and the CAD120 dataset are both video datasets of human interaction, each frame containing label information of human interactive behavior and object function participation, while the Something-else dataset does not contain label information of object function participation, so different comparison algorithms are used for the two types of datasets, including GPNN, HierGAT, LIGHTEN, STIGPN, IcH-HOI, STGC and the embodiment of the present application.

[0136] As shown in Table 1, in the experiment of the CE-HIRD dataset, the embodiment enriches the context information of the data-scarce interaction behavior by introducing global features and memory embedding loss, and achieves the highest experimental results in the interaction behavior value (Sub-activity F1) and the object function availability value (Affordence F1). The traditional GPNN is insufficient in modeling the time sequence of interaction information, and the time sequence information in video interaction behavior detection has an important influence on the model effect, so the effect is not good compared with other methods. HierGAT extracts hierarchical temporal features from video data to capture the temporal dynamics of human interaction, and LIGHTEN also considers the spatio-temporal features of the video. Compared with the GPNN method, the interaction behavior value of the two methods in the CE-HIRD dataset is increased by 1.42% and 3.01% respectively, and the object function availability value is increased by 0.35% and 0.04% respectively. STIGPN uses graph networks to mine spatial and temporal evolution to simulate the long-term dynamics of the target, and the effect is greatly improved compared with the previous method. IcH-HOI has time reasoning ability through the context fusioner and the interaction state reasoning module, and achieves an effect comparable to STIGPN. STGC is a further improvement of STIGPN, which captures the spatio-temporal correlation by introducing a spatio-temporal feature enhancement module, and is improved by 1.79% and 1.07% compared with STIGPN. Since existing methods all mine video information from the perspective of local features, they do not consider the problem of uneven data distribution, while the embodiment introduces global features of the dataset to make up for the defect that local features are difficult to handle similar categories, and is improved by 2.29% and 1.03% compared with the current optimal STGC method, achieving the effect of SOTA (State-of-the-Art).

[0137] Table 1. Experimental results table on the CE-HIRD dataset.

[0138] .

[0139] Since the scenes of the CE-HIRD dataset are chemical experiment scenes, the action similarity between classes is greater than that of the CAD120 dataset of daily life scenes, and the embodiment solves the problem of greater inter-class similarity by fusing global and local features. Therefore, in the experiment of the CAD120 dataset, as shown in Table 2, the embodiment improves STGC by 0.55%, and the object function availability value is slightly lower than the STGC method, but in the CE-HIRD dataset, it improves by 2.29% and 1.03% compared with it. Since the CE-HIRD dataset and the CAD120 dataset have the same labeling and evaluation standards, the comparative experiment in the CAD120 dataset in this paper is similar to the CE-HIRD dataset, and the comparison between the embodiment and VHOIP and DT-HOI methods is also added. Among them, VHOIP introduces CLIP prior knowledge, but does not consider the data distribution problem, so the effect is lower than the embodiment method. DT-HOI expands the text description by a large model, but the label information in the dataset is relatively small, and after expansion, more rich text information can be obtained, which improves the effect of the model, and the result is higher than the embodiment method proposed in this paper.

[0140] Table 2. Experimental results table on the CAD120 dataset.

[0141] .

[0142] As shown in Table 3, in experiments on the Something-else dataset, graph network-based methods outperformed methods based on spatiotemporal interaction networks. This is because graph networks can more effectively capture and model the complex spatiotemporal dependencies in video data, while methods based on spatiotemporal interaction networks can capture spatiotemporal information using bounding box coordinates as input. However, methods based on spatiotemporal interaction networks have limitations in extracting spatiotemporal features and modeling complex interactive behaviors when dealing with long-range dependencies and subtle motion changes in videos. With the introduction of I3D, methods based on spatiotemporal interaction networks can capture both spatial and temporal information in videos, providing richer feature representations and significantly improving performance. The I3D-based STIN+OIE+NL method combines a separately trained I3D model with the trained STIN+OIE+NL method, achieving state-of-the-art results in the STIN family. Among graph network-based methods, STGCN and STGC achieve superior results compared to methods without I3D by modeling spatiotemporal relationships in videos using spatiotemporal graph networks. DT-HOI relies on inputting label information into a large model to expand text descriptions, thereby improving the understanding of character interactions. However, the data labels of the Something-else dataset are relatively complex, and the improvement effect on the method is not obvious. Because the method proposed in this article takes into account the problem of uneven data distribution, it improves the accuracy of Top-1 and Top-5 by 2.33% and 1.47% compared to the current state-of-the-art STGC method, achieving the best results and further demonstrating the effectiveness and generalization of the method. In summary, after introducing global features and memory embedding loss, the model enriches the contextual information of data-scarce interactions and, to a certain extent, solves the problem of large similarity between interaction behavior categories.

[0143] Table 3. Experimental results on the Something-else dataset.

[0144] .

[0145] To verify the effectiveness of the global feature branch (GB) and memory embedding loss (MEL) proposed in this embodiment, ablation experiments were performed on the CE-HIRD, CAD120 and Something-else datasets. The experimental results are shown in Table 4, where "GLFFN" represents the complete method proposed in this embodiment, "GLFFN w / o All" represents the removal of the global feature branch and the memory embedding loss module in this embodiment, "GLFFN w / o MEL" represents the removal of the memory embedding loss in this embodiment, and "GLFFN w / o GB" represents the removal of the global feature branch in this embodiment. The local feature branch is the basic framework for realizing the interaction behavior, so the experiment does not include the removal of the local feature branch part.

[0146] After introducing the global feature branch and the memory embedding loss, the network performance is effectively improved. Specifically, compared with the method after removing the global feature branch, the interaction behavior recognition and object function participation of this embodiment on the CE-HIRD dataset are improved by 3.39% and 0.27% respectively. The score is improved by 0.6% and 0.25% on the CAD120 dataset. On the Something-else dataset, the Top-1 accuracy and Top-5 accuracy are improved by 1.33% and 1.35% respectively, proving that the global feature branch can provide more comprehensive data information and contextual knowledge to the model after introducing the knowledge of the entire dataset. The memory embedding loss expands the distance between similar interaction behavior categories, improving the model's ability to distinguish such interaction behaviors. Compared with the method after removing the memory embedding loss, the interaction behavior recognition and object function participation of this embodiment on the CE-HIRD dataset are further improved by 0.69% and 1.83%, and on the CAD120 dataset, they also bring 0.99% and 0.35% improvement, and on the Something-else dataset, they are improved by 0.65% and 2.06%, proving the effectiveness of the memory embedding loss. Finally, when the global feature branch and the memory embedding loss are combined together, the network performance is obviously improved, which shows that there is a synergistic effect between the two components, effectively solving the problem that the limited training data of part of the interaction behaviors leads to the recognition deviation of the intra-class homogeneity and inter-class heterogeneity.

[0147] Table 4. Ablation experiment results table on CE-HIRD, CAD120 and Something-else datasets.

[0148] .

[0149] To further analyze the improvement effect of the method on the data scarce categories, the confusion matrix of the baseline model STIGPN and the method of the embodiment is compared as shown in Figure 5 and Figure 6 In the CE-HIRD dataset, the effect of the method of the embodiment on the recognition of each type of interaction behavior is higher than 70%, and is better than the baseline model algorithm. For the categories of interaction behavior data scarce, "inverted", "oscillation", and "cover", the effect is increased by 18%, 20%, and 17% respectively, and a great improvement is obtained. This is mainly because the baseline model does not effectively pay attention to the data scarce samples in training, resulting in that the recognition result is easily misjudged as other data abundant behaviors, so that the effect is generally low. For example, in the baseline model, the "oscillation" interaction behavior is easily recognized as the "dumping" interaction behavior, because the number of "oscillation" samples is small, when the "oscillation" amplitude is large, it is easy to be misjudged as "dumping". The method proposed in the paper introduces global feature branch and memory embedding loss, so that the recognition performance of "dumping" action is slightly reduced, but the recognition performance of "oscillation" action is greatly improved. After analyzing the object function participation, it is found that the improvement of the data scarce "oscillatable" and "coverable" categories is large, which is increased by 11% and 9% respectively. In summary, this shows that the method of the embodiment can re-allocate resources to data scarce samples and data relatively abundant samples in training, and better distinguish features, and optimize the overall performance.

[0150] To further reveal the influence of the method of the embodiment on the intra-class and inter-class distance, the high-dimensional features of the baseline model and the chemical experiment interaction behavior categories in the memory network of the embodiment are mapped to 2D space by t-SNE, so as to intuitively show the influence of different methods on the feature space distribution as shown in Figure 7 and Figure 8 In Figure 7 , the baseline model is difficult to distinguish the two types of interaction behaviors of "dumping" and "oscillation" circled by the dashed line, and the feature space overlaps. In Figure 8 , the inter-class distance of "dumping" and "oscillation" actions of the embodiment is significantly increased, and the feature space boundary is clearer. In addition, the feature space of "put down" and "cover" and "pick up" and "inverted" becomes clearer in the visualization diagram of the method of the embodiment, and the intra-class distance of "stand" and "stir" categories is also obviously improved. This proves that the embodiment can enrich the context information of the categories, reduce the intra-class distance, and on the other hand, can enhance the difference of similar features and expand the distance between the two types of interaction behaviors, so that the performance of the method is obviously improved.

[0151] Experimental results show that compared with the baseline model, the TOP-1 accuracy of the interactive behavior classification of the embodiment on the three data sets of Something-else, CAD120 and CE-HIRD is increased by 2.73%, 1.05% and 4.08% respectively, which proves the improvement effect of the method on the recognition accuracy of the interactive behavior. And it surpasses the current SOTA method in the Something-else and CE-HIRD data sets. By expanding the inter-class distance between similar samples through memory embedding loss, the understanding ability of the model to data scarce interactive behaviors is enhanced, and in the training, resources can be redistributed between data scarce samples and data abundant samples for better feature distinction, optimizing the overall performance and improving the robustness of the model.

[0152] In this article, the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inner", "outer", "vertical", "horizontal" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of expressing the clarity and convenience of the description of the technical solutions, and therefore cannot be understood as a limitation on the present application.

[0153] In this article, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, in addition to containing the listed elements, other elements not explicitly listed can also be contained.

[0154] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A memory network enhancement method for experimental interactive behavior recognition, characterized by The steps include: S1. Obtain a video dataset containing multiple experimental videos, construct a global video graph of the experimental videos through a video-level graph network, and extract the global features of each experimental interaction behavior category in all experimental videos in the video dataset; S2. Build a frame-level local interaction graph of the experimental video through the region-level graph network to obtain the local features of a single experimental video. S3. Initialize the memory items of the memory network using global features, project local features into the feature space of the memory items through the attention mechanism, and enhance the fusion with global features. And expand the inter-class distance of similar experimental interaction behaviors between memory items through memory embedding loss. Combined with cosine similarity loss L through the following formula sim and Euclidean distance loss L euc Memory embedding loss L as memory network mem ; ， Among them, λ is a hyperparameter; The cosine similarity loss L is obtained by minimizing the cosine similarity between the experimental interaction behavior features of different memory items through the L2 norm. sim , ， Among them, s ij Represents a memory item n i and memory items n j The cosine similarity between them, s is the similarity matrix of all cosine similarities between the memory items of the memory network, and N is the number of memory items in the memory network; Calculate the Euclidean distance between memory items, obtain the distance matrix of all Euclidean distances between memory items of the memory network, and sort each row of the distance matrix in ascending order to obtain D'=(d ij ')∈R N×N , d ij 'Indicates memory items after sorting n i and memory items n j The Euclidean distance between the two items is increased by selecting the k2 minimum values ​​in each row of the distance matrix through the following formula: ， Among them, d — is the increased Euclidean distance, and the Euclidean distance loss L euc =max(0,-d — +γ), γ is a hyperparameter for adjusting the distance between memory items, and k2 is an integer less than N.

2. The memory network enhancement method according to claim 1, wherein: In step S1, the video-level graph network uses BiRNN to process the time series of all experimental video frame-level features to obtain the visual features of each experimental video, and uses the visual features to instantiate the video nodes of the experimental video. Every two video nodes are connected to obtain an initialized global video graph.

3. The memory network enhancement method according to claim 1, wherein: In step S1, the steps for extracting the global features of each experimental interaction behavior category are as follows: First, the video nodes of the global video graph are updated through the graph attention network, and the attention coefficient a between the video nodes is obtained through the attention mechanism. ij ; Secondly, the final feature x of a single experimental video is obtained by updating the attention coefficient i ', ， Among them, x i is the video node feature, W video is the attention mechanism weight matrix, σ is the activation function; Finally, after updating the visual features of all experimental videos using the graph attention network, the visual features of the experimental videos are grouped according to the label information of the experimental interaction behavior. The global features of each experimental interaction behavior category are calculated using the following formula: ， where c k is the global feature of the kth experimental interaction behavior category, N video is the number of all experimental videos, y i is the visual feature of the ith experimental video, and the global feature dataset C of all experimental interactive behavior categories is obtained, C={c k } k=1,...,K , where K represents the number of experimental interaction behavior categories.

4. The memory network enhancement method according to claim 1, wherein: In step S2, the region-level graph network uses Faster R-CNN or YOLOv8 to locate instances of operators or experimental equipment on each frame of the experimental video, extracts visual features, spatial features, and semantic features of all instances for splicing, constructs a frame-level local interaction graph of the experimental video, and connects the spliced ​​features within and between frames to obtain local features of a single experimental video.

5. The memory network enhancement method according to claim 4, characterized in that: The FasterR-CNN localization instance has a four-dimensional coordinate box and a target category. For each instance, the region of interest is cropped from the original experimental video frame according to its four-dimensional coordinate box and input into the pre-trained ResNet-50 model to extract the visual feature F v , Use a multi-layer perceptron MLP to transform the four-dimensional coordinates of each instance into spatial features F of the corresponding dimensions s , Generate the semantic features F of the instance using a learnable word embedding model w .

6. The memory network enhancement method according to claim 4, characterized in that: The frame-level local interaction graph includes a set of operator or experimental equipment nodes in the frame image, and a set of operator or experimental equipment edges in the frame image. For the edges connecting operators and experimental equipment, the edge weights are initialized to 1, and the others are 0. The intra-frame edge connection graph and the inter-frame edge connection graph are parsed using a graph attention network to obtain the intra-frame edge connection graph attention coefficient and attention coefficient of inter-frame connection graph , the local features of the final single experimental video are obtained by connecting the intra-frame and inter-frame splicing features , ， Among them, W inra With W inter are the intra-frame edge weight matrix and the inter-frame edge weight matrix, H represents the concatenation features of the instance’s visual features, spatial features, and semantic features, and [] represents the connection operation.

7. The memory network enhancement method according to claim 1, wherein: In step S3, the process of fusing and enhancing the local features with the global features is as follows: S31. Build a memory network M using the global features of all experimental videos. The memory network has N memory items, corresponding to the number of experimental interaction behavior categories. Each memory item is initialized with the global features of the corresponding experimental interaction behavior category. S32, input the local features of a single experimental video into the memory network, and use the attention mechanism method to integrate the local features Projecting to the feature space of the memory item to obtain a new feature representation ; S33, the memory item enhances the projected local features by the following formula: ， in, is the local feature projected into the memory item feature space, is the feature after memory item enhancement, c k is the global feature of the kth experimental interaction behavior category, n j Represents the global feature corresponding to the j-th memory item of the memory network; S34. Obtain the label of the predicted experimental interaction behavior category through the softmax function.

8. An experimental interactive behavior recognition method, characterized by: Input the experimental video to be identified into the memory network enhanced by the memory network enhancement method according to any one of claims 1 to 7, retrieve the memory items corresponding to the experimental interactive behaviors from the memory network in combination with the currently input video data, and output the experimental interactive behavior category retrieved in the experimental video to be identified.

9. Experimental interactive behavior recognition system, characterized by include: Input module, used to input the experimental video to be identified; A memory module storing a memory network enhanced by the memory network enhancement method according to any one of claims 1 to 7; The query module retrieves the memory items corresponding to the experimental interaction behaviors from the memory network based on the current input video data; The output module outputs the experimental interaction behavior category retrieved from the experimental video to be identified.

Images