Group monkey action recognition method based on spatiotemporal interaction network

Abstract

The present application discloses a method for group monkey action recognition based on a spatiotemporal interaction network, including: building a group monkey action recognition model with three layers, namely, a position and category recognition layer, a feature map processing layer, and an action reasoning layer, wherein the residual network in the position and category recognition layer is connected to the second interface of the feature map processing layer through a first region generation network, the feature map processing layer converts the position and category information of different action subjects in the current frame into a multidimensional tensor, uses the position information of the action subject as an index, selects an area of ​​interest from the first region generation network, and superimposes it with the multidimensional tensor, the action reasoning layer organizes the superposition results into a three-dimensional data group in time sequence, performs a shift operation to generate an action reasoning tensor, and outputs the action category corresponding to the position of the action subject; and uses the monkey group action data to train the group monkey action recognition model until the model converges. The technical solution in the present application solves the problem of group monkey action recognition in cages.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a method for group monkey action recognition based on a spatiotemporal interaction network. Background Art

[0002] In the field of computer video action recognition, after being well trained, deep neural networks can distinguish different categories of actions and show excellent action recognition performance.

[0003] With the rapid development of convolutional neural networks and the release of many large-scale data sets, deep learning-based data processing methods using 2D-CNN or 3D-CNN or both are widely used in the field of video action recognition. Among them, 2D-CNN is mainly used for spatial feature extraction, while 3D-CNN has excellent performance in extracting temporal information through 3D convolution, but the amount of 3D convolution calculation is much higher than that of 2D convolution.

[0004] In human disease diagnosis and drug evaluation, primates (such as rhesus monkeys and crab-eating macaques) are indispensable experimental animals. Drug evaluation on primates is an important step before drug clinical trials. Using artificial intelligence methods to perform behavioral recognition and health monitoring on primates can save a lot of manpower and material resources, which helps provide a basis for drug evaluation.

[0005] However, the objects of existing large-scale behavior recognition models and public data sets are all humans. Such behavior recognition models first train the Faster r-cnn target detection network to obtain the bounding box of humans, then use the 3DCNNslowfast network to extract spatiotemporal information, and obtain RoI (region of interest) features through the RPN (region generation network) network in the Faster r-cnn network, and finally perform action classification through the pooling layer and the fully connected layer.

[0006] Existing behavior recognition models are very poor in identifying monkey actions. First, there is no public dataset with bounding box annotations of groups of monkeys in cages. Second, monkeys' bodies are mostly covered with hair of similar colors, making it difficult to identify the subject of the behavior. Third, monkeys' actions are different from those of humans. Not only is the logic between their actions poor and difficult to understand, but the types of behaviors vary greatly. For example, monkeys can climb and walk on all fours, and some actions (such as jumping) occur very quickly, sometimes in less than a second. The shortest existing dataset is the AVA dataset, which has an accuracy of 1 second and cannot identify actions that last less than a second.

[0007] In addition, existing behavior recognition models usually use 2D CNN and 3D CNN for action reasoning, which results in huge memory usage and increases the hardware cost for implementing group monkey behavior recognition. Summary of the invention

[0008] The purpose of this application is to solve the problem of video or image recognition of the movements of a group of monkeys in captivity.

[0009] The technical solution of the present application is: a method for group monkey action recognition based on a spatiotemporal interaction network is provided, the method comprising: building a group monkey action recognition model, wherein the group monkey action recognition model comprises at least three layers, namely a position and category recognition layer, a feature map processing layer and an action reasoning layer, the residual network ResNet in the position and category recognition layer is connected to the second interface of the feature map processing layer through a first region generation network RPN, the position and category recognition layer is used to identify the position and category information of the action subject in the current frame, the feature map processing layer is used to convert the position and category information of different action subjects in the current frame into a multi-dimensional tensor, and use the action determined by the position and category recognition layer The position information of the subject is used as an index to select the region of interest RoI of the corresponding position from the first region generation network RPN, and the region of interest RoI is superimposed with the multidimensional tensor. The action reasoning layer is used to form a three-dimensional data group according to the time sequence of the superposition results output by the feature map processing layer, and perform up and down bidirectional shift operations on the three-dimensional data group in the time dimension to generate an action reasoning tensor to output the action category corresponding to the position of the action subject determined by the position and category recognition layer; the monkey group action recognition model is trained using the monkey group action data until the monkey group action recognition model converges, wherein the converged monkey group action recognition model is used to perform action recognition on the group of monkeys.

[0010] In any of the above technical solutions, further, the output result of the position and category recognition layer at least includes the position information of the bounding box and the category information of the corresponding action subject, and the region of interest RoI is superimposed on the multidimensional tensor in the feature map processing layer, specifically including: connecting the position information of the bounding box and the category information of the corresponding action subject in series to form input information, inputting it into the multilayer perceptron, and the multilayer perceptron outputs the position and category information of different action subjects in the current frame; according to the set number of action subjects, splicing the output data of the multilayer perceptron to form a multidimensional tensor, and the number of action subjects is the number of columns of the multidimensional tensor; using the position information of the action subject determined by the position and category recognition layer as an index, selecting the region of interest RoI of the corresponding position from the first region generation network RPN; adjusting the dimension of the region of interest RoI to the dimension of the multidimensional tensor; and superimposing the region of interest RoI with the adjusted dimension on the multidimensional tensor by addition.

[0011] In any of the above technical solutions, further, the monkey group action recognition model also includes: a head area recognition layer; the head area recognition layer is provided with a second region generation network RPN2 and a second region of interest pooling RoIpooling2 module connected in sequence, wherein the residual network ResNet in the position and category recognition layer is respectively connected to the input end of the second region generation network RPN2 and the input end of the second region of interest pooling RoI pooling2 module, and the output end of the second region of interest pooling RoI pooling2 module is connected to the classifier softmax in the position and category recognition layer through a fully connected layer.

[0012] In any of the above technical solutions, the method further includes: building a monkey group motion database, in which a plurality of monkey group motion data are stored, and the monkey group motion data are obtained by photographing the monkey group in the cage with a camera, and the monkeys in the monkey group wear collars, and the collars are standard collars of different colors.

[0013] In any of the above technical solutions, further, there are multiple cameras, and the multiple cameras are installed at different positions of the cage, including at least the bottom of the front view surface, the top of the front view surface, the side view surface, and the top view surface of the cage. The multiple cameras are used to obtain the monkey movement data of the monkey group in the cage from different perspectives.

[0014] In any of the above technical solutions, further, a monkey group action database is established, which specifically includes: installing multiple cameras at different positions of the cage, and the multiple cameras are used to obtain video data of the monkey group in the cage from different perspectives; classifying the video data according to the installation position of the camera corresponding to the video data, and filtering according to the number of monkeys in the video data to delete the video data with the number of monkeys less than a threshold; performing frame processing on the filtered video data, and segmenting the framed video data according to a preset segmentation interval, and collecting key frames in the segmented video data, wherein the key frames are 3 frames of images with an interval of 7 frames from each other; according to the defined actions of the group of monkeys, manually annotating the collected key frames, recording them as monkey group action data, and building a monkey group action database.

[0015] In any of the above technical solutions, further, a monkey group motion database is built, which specifically includes: performing a flip expansion operation on the monkey group motion data to expand the monkey group motion data in the monkey group motion database, and the flip expansion operation is a left-right flip expansion operation.

[0016] The beneficial effects of this application are:

[0017] The technical solution in this application distinguishes individual monkeys in a monkey group by wearing standard collars of different colors, and places cameras at different positions in the cage to obtain video data of the monkeys, and then uses classification, screening, and frame annotation to build a monkey group action database suitable for the monkey group. Then, by building a monkey group action database with a three-layer structure of a position and category recognition layer, a feature map processing layer, and an action reasoning layer, the feature map processing layer uses the action subject position information determined by the position and category recognition layer as an index, selects the region of interest RoI of the corresponding position from the first region generation network RPN, and superimposes the region of interest RoI with the output result of the position and category recognition layer to improve the recognition effect of the monkey group action data, which helps to accurately identify the individual monkeys in the data, improve the accuracy of monkey action recognition, extract spatiotemporal information through time shift operations, and occupy very little memory.

[0018] In a preferred implementation of the present application, in order to improve the accuracy of individual monkey identification and fully consider the characteristics of the monkey's head, a branch for generating the head area is set up, which is combined with the position and category recognition layer in the model, and a weighted loss function is used to improve the accuracy and reliability of the monkey's full body bounding box output by the position and category recognition layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0020] Figure 1 is a schematic flow chart of a method for group monkey action recognition based on a spatiotemporal interaction network according to an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a network model according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of using “labelme” to label data according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0024] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0025] like Figure 1 and Figure 2 As shown, this embodiment provides a group monkey action recognition method based on a spatiotemporal interaction network, the method comprising:

[0026] Step 1: Install multiple cameras at different positions of the cage, and the multiple cameras are used to obtain video data of the monkey group in the cage from different perspectives. The multiple cameras are installed at different positions of the cage, including at least the bottom of the front view surface, the top of the front view surface, the side view surface, and the top view surface of the cage, and the multiple cameras are used to obtain the monkey group movement data of the monkey group in the cage from different perspectives.

[0027] Specifically, several cameras are installed at the bottom, top, side, and top view of the front view of the cage through fixing devices, so that the cameras fit the cage to avoid being blocked by the outer railings of the cage. Cameras with different viewing angles are used to capture different actions of the monkeys in the cage, so as to collect video data of the group of monkeys in the cage under the breeding environment as the monkey group action data.

[0028] The feeding trough for monkeys is on the outside of the bottom of the cage. Monkeys will grab food from the bottom trough, but may then eat on the ground or on top of the cage. If the camera is placed farther away from the cage to obtain a wider viewing angle, the obstruction of the cage railing will have a great impact. Therefore, two upper and lower cameras are installed on the front side where more movements are detected.

[0029] In addition, the clarity and occlusion of capturing different actions of the monkey are different at different perspectives. It is also convenient to carry out work in other directions, so cameras are installed at multiple perspectives.

[0030] After the camera is installed, the data collector will leave the breeding room to avoid the monkeys being frightened by human factors and unable to collect the normal movements of the monkeys. The data collector can connect to the corresponding camera through the mobile phone APP to monitor the camera shooting situation in real time to prevent accidents.

[0031] After shooting, the data collector exported the video in the camera to the computer. The default frame rate of the camera is 30FPS, the default length of a single video saved by the camera is 9 minutes, and the size mode used is 1080P (1920×1080).

[0032] Among them, XTU Xiaotu camera can be used, and the specific parameters can be configured according to the actual situation. The age of the monkeys in the cage ranges from 1 to 12 years old, including crab-eating macaques and other monkey species.

[0033] Step 2: Classify the video data according to the installation positions of the cameras corresponding to the video data, and filter the video data according to the number of monkeys in the video data to delete the video data with the number of monkeys less than the threshold;

[0034] Specifically, the collected video data is screened and classified according to the video data collection angle, and the classified video data is framed. In addition, since the monkeys in different cages are of different ages and types, the monkeys are frightened to different degrees and have different recovery times when installing the camera, resulting in uneven quality of the collected video data. Therefore, it is necessary to manually filter and classify the data, and delete the video data with fewer or no monkeys. At the same time, since the action categories contained in the data collected from different angles are also very different, the video data is classified according to the shooting source so that the angle of view of the video data corresponds to the camera one by one.

[0035] Step 3: Frame the filtered video data, and segment the framed video data according to the preset segmentation interval, and use a random sampling method to collect key frames in the segmented video data, where the key frames are 3 frames of images with an interval of 7 frames between each other.

[0036] Specifically, after screening and classification, the classified video data is processed by using the computer ffmpeg command. By default, 1 second of video is cut into 30 frames, corresponding to the camera shooting parameters. In this way, every 30 images represent 1 second of video.

[0037] Taking into account that the length of the video detected in the actual scene may be quite different from the length of the video used when we trained the network, the video is segmented so that the network can automatically process the video length, so that the actual input video length is not much different from that during training. For example, a 30-second video is divided into 8 segments, each of which is about 4 seconds long; a 1-minute video is divided into 16 segments to ensure that each segment is also about 4 seconds long.

[0038] The actual input to the network is each processed video, and the preset segmentation interval is set to 4 seconds. The 4-second duration is the empirical value we obtained after manually observing the duration of different actions in the video data. In this way, the data segmentation operation is completed.

[0039] The object of random sampling is each video after the network automatically segments it. Since the specific time of occurrence of the monkey's actions in each video data is inconsistent, which is related to many uncontrollable factors such as the video itself and the type of action, a random sampling method is adopted to capture more accurate information.

[0040] It is assumed that 3 frames of images with an interval of 7 frames are randomly taken from each segment of video data. Therefore, the key frame span is about 30 frames, which contains random continuous 1 second of spatiotemporal information in the segment of video data as a representative of the overall spatiotemporal information of the segment of video data, so as to facilitate data annotation of the key frames.

[0041] Step 4: According to the defined group monkey movements, the key frames are manually labeled and recorded as monkey group movement data, and a monkey group movement database is built. The monkey group movement database stores multiple monkey group movement data. The monkey group movement data is obtained by photographing the monkey group in the cage with a camera. The monkeys in the monkey group wear collars, which are standard collars of different colors.

[0042] Specifically, after step 3, three key frames of each video data are obtained, and these key frames are labeled, and the category of the key frame is used to represent the category of the video data. In the labeling process, it is necessary not only to identify the category of the action, but also to identify which monkey is the subject of the action. Therefore, multi-label labeling is used when defining the key frame label.

[0043] In the actual breeding environment, different monkeys wear standard collars of different colors to distinguish them. Different action categories also correspond to different codes. Therefore, multi-label two-digit annotation can be used.

[0044] The first digit represents the category of the action subject. There are 5 monkeys in each cage. The collar colors are yellow, green, red, black, and white, which are represented by 5 numbers from 0 to 4. For example, if the action is "grabbing food", the action subject of the action is a certain monkey and the food trough. In this embodiment, the number 5 is used to represent the "food trough" and the number "6" is used to represent the "food". The food and the food trough are uniformly made by the breeding company and are the same from a visual perspective.

[0045] The second digit represents the action category of the action subject, which defines group monkey interaction actions such as "grabbing food, eating, hugging, fighting, chasing", etc., which are represented by 5 digits from 0 to 4.

[0046] In this way, there are 15 labels for the actions between monkeys "hug, fight, chase", 5 labels for the action between monkeys and food "eating", and 5 labels for the action between monkeys and troughs "grabbing food". Food has only one action "eating" and the trough has only one action "grabbing food", for a total of 27 labels.

[0047] For example, if the yellow collar monkey and the green collar monkey are hugging, they are labeled as "02" and "12" respectively. Due to the special situation that different hugging actions may appear in a single frame of video, the position information of the labeled bounding box can be used for filtering and identification later; if the red collar monkey is eating, the monkey is labeled as "21" and the food is labeled as "61". The open source tool "labelme" is used in the actual labeling work. Figure 3 As shown, the bounding boxes are box 301 and box 302 respectively.

[0048] Furthermore, in order to expand the amount of data in the monkey group action database, in the process of building the monkey group action database, the data can also be expanded. The method also includes:

[0049] Step 5: Perform a flip expansion operation on the monkey group motion data to expand the monkey group motion data in the monkey group motion database. The flip expansion operation is a left-right flip expansion operation.

[0050] Specifically, after using "labelme" to annotate the key frames of the video data, you can get the annotation results in json format. You need to extract the location information, label category, timestamp, and corresponding video segment of the bounding box from it. Through data processing operations, the extracted information can be saved in txt format. One txt file corresponds to the annotation result of a key frame, and each line in a txt file represents the relevant information of a bounding box in the key frame.

[0051] At the same time, considering the problem of data category imbalance, the marked data needs to be expanded. Since there is no directional action in the defined actions, the image can be expanded by flipping it left and right. For example, the action of "jumping to the left" will not be mistaken for "jumping to the right". However, the monkey's head should be above the tail under normal circumstances. Therefore, the upside-down flipping operation cannot be used, which will introduce a lot of spatial noise information.

[0052] Step 6: Build a group monkey action recognition model, where Figure 2As shown, the group monkey action recognition model includes at least three layers, which are a position and category recognition layer, a feature map processing layer and an action reasoning layer. The residual network ResNet in the position and category recognition layer is connected to the second interface of the feature map processing layer through the first region generation network RPN. The position and category recognition layer is used to identify the position and category information of the action subject in the current frame. The feature map processing layer is used to convert the position and category information of different action subjects in the current frame into a multidimensional tensor, and use the position information of the action subject determined by the position and category recognition layer as an index to select the region of interest RoI of the corresponding position from the first region generation network RPN, and superimpose the region of interest RoI with the multidimensional tensor. The action reasoning layer is used to form a three-dimensional data group according to the time sequence of the superposition results output by the feature map processing layer, and perform up and down bidirectional shift operations on the three-dimensional data group in the time dimension to generate an action reasoning tensor to output the action category corresponding to the position of the action subject determined by the position and category recognition layer;

[0053] Specifically, the first stage of the model is located at Figure 2 The first line is the position and category recognition layer, which performs target detection on the key frame image. Since this stage targets a single frame image, this work can only process spatial information. This embodiment performs target detection based on the SOTA level Faster R-CNN network, obtains the position and category of the action subject in the single frame image, that is, different monkeys, food and troughs, and outputs the position and category information of the identified subject in the image.

[0054] After the data labeling process in step 5 reaches a certain amount of data, it is used to train the target detection network in the first stage, and then the trained detection network is used to predict the single-frame image. The predicted result is the position information of the bounding box corresponding to the action subject. The position information of the bounding box is manually fine-tuned based on the prediction, and the category information of the bounding box is manually labeled. This can greatly reduce the amount of tasks.

[0055] In a preferred implementation of this embodiment, the monkey group action recognition model also includes: a head area recognition layer; the head area recognition layer is provided with a second region generation network RPN2 and a second region of interest pooling RoIpooling2 module connected in sequence, wherein the residual network ResNet in the position and category recognition layer is respectively connected to the input end of the second region generation network RPN2 and the input end of the second region of interest pooling RoI pooling2 module, and the output end of the second region of interest pooling RoI pooling2 module is connected to the classifier softmax in the position and category recognition layer through a fully connected layer.

[0056] Specifically, through Figure 2 The second region generation network RPN2 and the second region of interest pooling RoIpooling2 module at the top are used to generate the monkey's head region. This is because compared with humans or other animals, the hair on the body parts of the same monkeys often has the same hair color, which cannot be used as the main basis for distinguishing different monkey individuals. The monkey's head region is the main basis for distinguishing different individuals. Therefore, setting a branch for generating the head region can make the feature extraction network pay more attention to the head region.

[0057] In this embodiment, a fully connected layer is connected after the second region of interest pooling RoI pooling2 module to output the bounding box of the head area, and the corresponding loss function is L1.

[0058] like Figure 2 As shown in the second row, the key frame is passed through the residual network ResNet, the first region generation network RPN, the first region of interest pooling RoI pooling module, and the softmax module to obtain the whole body bounding box and the monkey category, that is, which type of collar is worn by the monkey among "yellow, green, red, black and white". The loss function for identifying the whole body bounding box is set to L2, and the classification loss function is set to L3. The total loss function L in the first stage of the network is a The calculation formula is:

[0059] 0.6*(0.3*L1+0.7*L2)+0.4*L3=L a

[0060] Among them, 0.3 and 0.7 are the weights of the head area loss function L1 and the whole body area loss function L2 respectively. The result of weighted summation corresponds to the detection loss, and then the weighted summation with the classification loss L3 is obtained to obtain the total loss function L in the first stage. a , 0.6 and 0.4 are the corresponding weights respectively. The above weights are debugged and optimized according to the characteristics of monkeys in cages.

[0061] Through the first stage of the network, the position and category information of different action subjects in a single-frame image are obtained. It is passed to the calibration module composed of the Kalman filter and the KM algorithm to reduce the occurrence of "frame skipping" in the actual detection results (that is, inaccurate detection of single-frame results). Use the Kalman filter and the KM algorithm to track the action subject. The Kalman filter generates a trajectory based on the position information of the bounding box of the same action subject in the first two key frames to predict the possible position of the corresponding subject in the next key frame, and then uses the km algorithm to match the predicted position of the action subject with the single-frame detection result to reduce the occurrence of "frame skipping". Figure 2 The first line is shown at the end.

[0062] Through the model and calibration module of the first stage, more accurate information about the position and category of the action subject can be obtained. On the one hand, this result is saved and output. On the other hand, it is passed to the second stage of the model, that is, the output of the position and category recognition layer is connected to the first interface of the feature map processing layer.

[0063] Furthermore, the output results of the position and category recognition layer include at least the position information of the bounding box and the category information of the corresponding action subject. The feature map processing layer superimposes the region of interest RoI with the multidimensional tensor, specifically including:

[0064] The position information of the bounding box and the category information of the corresponding action subject are connected in series to form the input information, which is input to the multi-layer perceptron, and the multi-layer perceptron outputs the position and category information of different action subjects in the current frame;

[0065] According to the set number of action subjects, the output data of the multi-layer perceptron is spliced ​​into a multi-dimensional tensor, and the number of action subjects is the number of columns of the multi-dimensional tensor;

[0066] Using the position information of the action subject determined by the position and category recognition layer as an index, a region of interest RoI at the corresponding position is selected from the first region generation network RPN;

[0067] Adjust the dimension of the region of interest RoI to the dimension of the multidimensional tensor;

[0068] The dimensionally adjusted region of interest RoI is superimposed on the multidimensional tensor by addition.

[0069] Specifically, in the second stage of the model, Figure 2 As shown in the second row, the position information of the bounding box and the category information of the corresponding action subject are connected in series as input and input into a multi-layer perceptron. The action and category labels of a single subject are 1*5 vectors, which are extracted by the multi-layer perceptron MLP to extract the high-dimensional information. The activation function is set to the ReLU function, and its dimension is changed from 5 to n, converting it into a 1*n vector.

[0070] At this point, after the decoding process, we have obtained n-dimensional information representing the position and category of the action subject. Since each frame generally contains multiple action subjects, we preset m action subjects and concatenate the output data of the multilayer perceptron into an n*m matrix to form a multidimensional tensor. In this way, we get an n×m multidimensional tensor from a single frame image and arrange the outputs of different frames in chronological order.

[0071] At the same time, according to the position information of the action subject, the position information of the action subject transmitted from the previous stage is used as the index to select the relative coordinates, select the RoI region of interest at the corresponding position from the first region generation network RPN network, and adjust its dimension to the size of n×m (such as by setting the RoI pooling module). After that, the addition method is used to integrate it with the n×m tensor obtained in the previous stage. Compared with the previous original tensor, the tensor obtained after the addition will strengthen its position information in the original image of the monkey group action data, making it more accurate.

[0072] The t n×m tensors in the second stage can be viewed as t×n×m cubes, which serve as the input of the third stage (action reasoning layer), as follows: Figure 2 As shown in the third row. The action inference layer in the third stage will combine the previous prediction results (i.e. the position and category of the action subject output in the first stage) and the temporal information to infer the action category between the action subjects. In the input t×n×m cube, t is the number of key frames in each video segment, and t=3 is preset.

[0073] The action reasoning layer performs up and down bidirectional shift operations on the cube (three-dimensional data group) in the time dimension, moving the channel along the time dimension, such as Figure 2 In the third stage of the network, along the direction of increasing t, the data position of the m-1 layer is kept unchanged, the m-th layer moves upward, and the m+1-th layer moves downward. This will cause the m-th, m+1-th layers to be staggered with the m-1-th layer. Here, the upward convex information of the m-th layer is intercepted and filled into the adjacent concave position of the layer. Similarly, the m+1-th layer intercepts the convex part for information filling, so that the current frame and adjacent frame information can be mixed. Through the above-mentioned up and down bidirectional shift operation, the conventional zero filling method is avoided, the original information is maintained, and the accuracy of motion reasoning is improved.

[0074] Then, the padded three-dimensional data group is average pooled in the time dimension to obtain an n×m motion reasoning tensor, which is split into m n×1 vectors and input into the classifier respectively. The output is the action categories corresponding to the m action subjects, corresponding to the positions of the action subjects determined by the position and category recognition layer.

[0075] Through the above process, the monkey action recognition model will eventually output the position information of the recognized action subject, its own category information and the inferred action category.

[0076] Step 7, using the monkey group motion data to train the monkey group motion recognition model until the monkey group motion recognition model converges, wherein the converged monkey group motion recognition model is used to perform motion recognition on the monkey group.

[0077] The above describes in detail the technical solution of the present application in combination with the accompanying drawings. The present application proposes a group monkey action recognition method based on a spatiotemporal interaction network, including: building a group monkey action recognition model, wherein the group monkey action recognition model includes at least three layers, namely a position and category recognition layer, a feature map processing layer, and an action reasoning layer, the residual network ResNet in the position and category recognition layer is connected to the second interface of the feature map processing layer through the first region generation network RPN, the position and category recognition layer is used to identify the position and category information of the action subject in the current frame, the feature map processing layer is used to convert the position and category information of different action subjects in the current frame into a multi-dimensional tensor, and use the position and category recognition layer to generate a multi-dimensional tensor. The determined position information of the action subject is used as an index to select the region of interest RoI of the corresponding position from the first region generation network RPN, and the region of interest RoI is superimposed with the multidimensional tensor. The action reasoning layer is used to form a three-dimensional data group according to the time sequence of the superposition results output by the feature map processing layer, and the three-dimensional data group is shifted up and down in the time dimension to generate an action reasoning tensor to output the action category corresponding to the position of the action subject determined by the position and category recognition layer; the group monkey action recognition model is trained using the monkey group action data until the group monkey action recognition model converges, wherein the converged group monkey action recognition model is used to perform action recognition on the group monkeys. The technical solution in this application solves the problem of group monkey action video recognition in a captive state.

[0078] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0079] The units in the device of the present application can be combined, divided and deleted according to actual needs.

[0080] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A group monkey action recognition method based on spatiotemporal interaction network, characterized in that: The method comprises: A monkey group action recognition model is constructed, wherein the monkey group action recognition model includes at least three layers, namely, a position and category recognition layer, a feature map processing layer, and an action reasoning layer. The residual network ResNet in the position and category recognition layer is connected to the second interface of the feature map processing layer through a first region generation network RPN. The position and category recognition layer is used to identify the position and category information of the action subject in the current frame. The feature map processing layer is used to convert the position and category information of different action subjects in the current frame into a multidimensional tensor, and use the position information of the action subject determined by the position and category recognition layer as an index to select a region of interest RoI at the corresponding position from the first region generation network RPN, and superimpose the region of interest RoI with the multidimensional tensor, The action inference layer is used to form a three-dimensional data group according to the time sequence of the superposition results output by the feature map processing layer, and perform an up and down bidirectional shift operation on the three-dimensional data group in the time dimension to generate an action inference tensor to output the action category corresponding to the action subject position determined by the position and category recognition layer; The monkey group action recognition model is trained using the monkey group action data until the monkey group action recognition model converges, wherein the converged monkey group action recognition model is used to perform action recognition on the monkey group.

2. The method for group monkey action recognition based on spatiotemporal interaction network as claimed in claim 1, characterized in that: The output result of the position and category recognition layer includes at least the position information of the bounding box and the category information of the corresponding action subject. The feature map processing layer superimposes the region of interest RoI with the multidimensional tensor, specifically including: The position information of the bounding box and the category information of the corresponding action subject are connected in series to form input information, which is input into a multi-layer perceptron, and the multi-layer perceptron outputs the position and category information of different action subjects in the current frame; According to the set number of action subjects, the output data of the multilayer perceptron is spliced ​​into a multidimensional tensor, and the number of action subjects is the number of columns of the multidimensional tensor; Using the position information of the action subject determined by the position and category recognition layer as an index, selecting a region of interest RoI at the corresponding position from the first region generation network RPN; Adjusting the dimension of the region of interest RoI to the dimension of the multidimensional tensor; The region of interest RoI after dimension adjustment is superimposed on the multidimensional tensor in an addition manner.

3. The method for group monkey action recognition based on spatiotemporal interaction network as claimed in claim 1, characterized in that: The group monkey action recognition model also includes: a head area recognition layer; The head region identification layer is provided with a second region generation network RPN2 and a second region of interest pooling RoIpooling2 module connected in sequence, The residual network ResNet in the position and category recognition layer is respectively connected to the input end of the second region generation network RPN2 and the input end of the second region of interest pooling RoI pooling2 module, The output end of the second region of interest pooling RoI pooling2 module is connected to the classifier softmax in the position and category recognition layer through a fully connected layer.

4. The method for group monkey action recognition based on spatiotemporal interaction network as claimed in claim 1, characterized in that: The method further comprises: A monkey group motion database is built, in which a plurality of monkey group motion data are stored. The monkey group motion data are obtained by photographing a monkey group in a cage with a camera. The monkeys in the monkey group wear collars, and the collars are standard collars of different colors.

5. The method for group monkey action recognition based on spatiotemporal interaction network as claimed in claim 4, characterized in that: There are multiple cameras, which are installed at different positions of the cage, including at least the bottom of the front view surface, the top of the front view surface, the side view surface, and the top view surface of the cage. The multiple cameras are used to obtain the monkey movement data of the monkey group in the cage from different viewing angles.

6. The method for group monkey action recognition based on spatiotemporal interaction network as claimed in claim 5, characterized in that: The construction of the monkey group action database specifically includes: Installing a plurality of the cameras at different positions of the cage, the plurality of the cameras being used to obtain video data of the monkey group in the cage at different viewing angles; Classifying the video data according to the installation positions of the cameras corresponding to the video data, and screening the video data according to the number of monkeys in the video data to delete the video data in which the number of monkeys is less than a threshold; Performing frame processing on the screened video data, and performing segment processing on the segmented video data according to a preset segmentation interval, and collecting key frames in the segmented video data, wherein the key frames are 3 frames of images with an interval of 7 frames between each other; According to the defined actions of the monkey group, the collected key frames are manually annotated and recorded as the monkey group action data, and the monkey group action database is built.

7. The method for group monkey action recognition based on spatiotemporal interaction network as claimed in claim 6, characterized in that: The construction of the monkey group action database specifically includes: A flipping and expansion operation is performed on the monkey group motion data to expand the monkey group motion data in the monkey group motion database, and the flipping and expansion operation is a left-right flipping and expansion operation.

Images