3d slow fusion joint detection and classification of group monkey spatiotemporal action localization network

Abstract

The application is a 3D slow fusion combined detection and classification group monkey space-time action positioning network, comprising an input module, a main body module, a neck module and a prediction module; the main body module comprises a plurality of same 2D main body modules in parallel, and the 2D main body modules respectively output features of three scales of 1 / 8, 1 / 16 and 1 / 32 of the original image; the input module has the same number of input ports as the 2D main body module, and each input port respectively inputs the original image of each sampling frame in sequence; the neck module is used for enhancing the semantic expression and positioning ability on multiple scales; and the prediction module is used for generating a prediction result. The application constructs a 3D slow fusion based group monkey space-time action positioning network combined with detection and classification which can effectively utilize multi-frame information and simplifies the training process.

Description

Technical Field

[0001] The present invention relates to the technical field of video image recognition, and in particular to a 3D slow-fusion joint detection and classification group monkey spatiotemporal motion localization network. Background Art

[0002] Because their genetics closely resemble those of humans, monkeys are widely used in scientific research, such as drug evaluation and hybrid monkey research. Recording monkey behavior is crucial for analyzing the positive and negative effects of drugs and distinguishing hybrid monkeys from non-hybrid monkeys. Traditionally, this recording process involves manually observing captive monkeys every few days and recording their general state. However, this process only provides a rough snapshot of the monkeys' brief state during observation and is heavily influenced by the recorder's subjective experience. This also makes it difficult to observe the monkeys' natural behavior. This is particularly true when studying monkey groups, as manual observation can create an unnatural effect on the monkeys.

[0003] The lack of datasets in the field of behavior recognition also hinders its development. Most publicly available datasets are for human behavior recognition, and most, such as UCF101 and Kinetics, identify one action per video segment. This data annotation method, which treats behavior recognition as video classification, cannot meet the requirements of spatiotemporal action localization, which requires identifying the position and behavior of each monkey simultaneously when multiple monkeys appear. The AVA human dataset, based on spatiotemporal action localization, attempts to combine behavior recognition with target detection and action classification. While this two-stage algorithm can combine the classification of previous and next frames, it requires converting the data into the formats required by two models and training two models, making it extremely complex to use. Summary of the Invention

[0004] The present invention provides a 3D slow fusion joint detection and classification group monkey spatiotemporal action localization network, characterized in that the group monkey spatiotemporal action localization network includes an input module, a trunk module, a neck module, and a prediction module;

[0005] The backbone module includes multiple identical 2D backbone modules connected in parallel, and the 2D backbone module includes multiple CBS modules, 2DC3 modules and an SPPF module. The 2D backbone module outputs features at three scales of 1 / 8, 1 / 16 and 1 / 32 of the original image respectively;

[0006] The input module has the same number of input ports as the 2D backbone modules, and the input ports are connected to the corresponding 2D backbone modules; each input port inputs the original image of each sampling frame in sequence;

[0007] The neck module includes an FPN module, a first PAN module, and a second PAN module. The FPN module is connected to the trunk module to obtain features at three scales of 1 / 8, 1 / 16, and 1 / 32 of the original image for enhancing semantic expression at multiple scales. The first PAN module and the second PAN module are connected to the FPN module to transfer shallow positioning information to deep features to enhance multi-scale positioning capabilities.

[0008] The prediction module convolves the three feature maps in one layer, obtains feature maps of three sizes, and generates prediction results.

[0009] Furthermore, the 2D backbone module includes first to fifth CBS modules, first to third 2DC3 modules and an SPPF module;

[0010] The output end of the first CBS module is connected to the second CBS module, the output end of the second CBS module is connected to the first 2DC3 module, the output end of the first 2DC3 module is connected to the third CBS module, the output end of the third CBS module is connected to the second 2DC3 module, the output end of the second 2DC3 module outputs 1 / 8 scale features of the original image and is connected to the fourth CBS module, the output end of the fourth CBS module is connected to the third 2DC3 module, the output end of the third 2DC3 module outputs 1 / 16 scale features of the original image and is connected to the fifth CBS module, the output end of the fifth CBS module is connected to the fourth 2DC3 module, the output end of the fourth 2DC3 module is connected to the SPPF module, and the output end of the SPPF module outputs 1 / 32 scale features.

[0011] Furthermore, the CBS modules each include a 2D convolution unit, a 2D BatchNorm unit and a SiLU unit.

[0012] Furthermore, the 2DC3 module includes a 2DC3 convolution unit, a concat unit and two CBS modules. The 2DC3 convolution unit and one CBS module are connected in parallel and have the same input node and output node. The output node of the 2DC3 convolution unit and the CBS module are connected in parallel to the concat unit, and the concat unit is connected to another CBS module. The 2DC3 convolution unit includes multiple CBS modules connected in series.

[0013] Furthermore, the SPPF module includes two CBS modules, three maxpool units and a concat unit; the first and second maxpool units of the three maxpool units are connected in series and in parallel with the third maxpool unit; the input end of the SPPF module is connected to the CBS module; the CBS module has multiple output ends, which are respectively connected to the first maxpool unit, the third maxpool unit and the concat unit; the output end of the first maxpool unit is connected to the second maxpool unit and the concat unit; the output end of the third maxpool unit is connected to the concat unit; the output end of the second maxpool unit is connected to the concat unit; and the output end of the concat unit is connected to another CBS module.

[0014] Furthermore, the FPN module includes a first 3DCBS unit, a first Upsample unit, a first concat unit, a first 3DC3 convolution unit, a second 3DCBS unit, a second Upsample unit, a second concat unit and a second 3DC3 convolution unit, which are connected in sequence.

[0015] Furthermore, the 3DCBS unit includes a 3D convolution unit, a 3D BatchNorm unit and a SiLU unit.

[0016] Furthermore, the 3DC3 convolution module includes a 3DC3 convolution unit, a concat unit and two 3DCBS units. The 3DC3 convolution unit and one 3DCBS unit are connected in parallel and have the same input node and output node. The output node of the 3DC3 convolution unit and the 3DCBS unit are connected in parallel to the concat unit, and the concat unit is connected to another 3DCBS unit. The 3DC3 convolution unit includes multiple 3DCBS units connected in series.

[0017] Furthermore, the 3DCBS unit, the concat unit, and the 3DC3 convolution module in the first PAN module are connected in sequence; the input end of the 3DCBS unit is connected to the output end of the second 3DC3 convolution unit of the FPN module; the input end of the concat unit is connected to the 3DCBS unit and also connected to the second 3DCBS unit of the FPN module; the output end of the 3DC3 convolution module outputs the first prediction data;

[0018] The 3DCBS unit, concat unit and 3DC3 convolution module in the second AN module are connected in sequence; the input end of the 3DCBS unit is connected to the output end of the second 3DC3 convolution unit of the FPN module; the input end of the concat unit is connected to the 3DCBS unit and also connected to the first 3DCBS unit of the FPN module; the output end of the 3DC3 convolution module outputs the third prediction data.

[0019] Furthermore, the prediction module generates position information, the monkey's confidence, the probability of the monkey's collar color, and the probability of the monkey's action for each frame of the image;

[0020] The total loss of the prediction module is:

[0021] Ltotal=k1*Lbox+k2*Lobject+k3*Lcolor+k4*Laction

[0022] Among them, k1, k2, k3 and k4 are preset hyperparameters, Lbox represents location information, Lobject represents the monkey's confidence, Lcolor represents the color of the monkey's collar, and Laction represents the monkey's action.

[0023] The beneficial effects achieved by the present invention are:

[0024] The present invention is different from the existing two-stage model of detectors and 3D classifiers. It proposes an end-to-end one-stage joint detection and classification group monkey spatiotemporal motion localization model, designs a 3D slow fusion method, and constructs a group monkey spatiotemporal motion localization network based on 3D slow fusion joint detection and classification that can effectively utilize multi-frame information. It simplifies the training process and outperforms the two-stage model in performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the structure of a 3D slow-fusion joint detection and classification group monkey spatiotemporal action localization network. DETAILED DESCRIPTION

[0026] The technical solutions of the present invention are described in more detail below with reference to the accompanying drawings. The present invention includes but is not limited to the following embodiments.

[0027] As attached Figure 1 As shown, the present invention provides a 3D slow fusion joint detection and classification group monkey spatiotemporal action localization network, which includes: an input module, a trunk module, a neck module, and a prediction module.

[0028] The backbone module includes a plurality of identical 2D backbone modules connected in parallel, and the 2D backbone module includes a plurality of CBS modules, a 2DC3 module and an SPPF module.

[0029] The 2D backbone extracts features at three scales of 1 / 8, 1 / 16, and 1 / 32 of the original image from the corresponding input images respectively, and a special 3D backbone is used to achieve weight sharing of multiple 2D backbones.

[0030] Compared to the 2D backbone, this special 3D backbone dilates all 2D convolutions into 3D convolutions. The 3D convolutions maintain the same spatial dimensions as the 2D ones. The kernel's temporal dimension is 1, the stride's temporal dimension is 1, and the padding's temporal dimension is 0. The 2DBatchnorm is replaced with the 3D Batchnorm. The 2D MaxPool is dilated into the 3D MaxPool. The kernel's temporal dimension is 1, the stride's temporal dimension is 1, and the padding's temporal dimension is 0.

[0031] In one embodiment, the 2D backbone module includes first to fifth CBS modules, first to third 2DC3 modules and an SPPF module; the output end of the first CBS module is connected to the second CBS module, the output end of the second CBS module is connected to the first 2DC3 module, the output end of the first 2DC3 module is connected to the third CBS module, the output end of the third CBS module is connected to the second 2DC3 module, the output end of the second 2DC3 module outputs 1 / 8 scale features of the original image and is connected to the fourth CBS module, the output end of the fourth CBS module is connected to the third 2DC3 module, the output end of the third 2DC3 module outputs 1 / 16 scale features of the original image and is connected to the fifth CBS module, the output end of the fifth CBS module is connected to the fourth 2DC3 module, the output end of the fourth 2DC3 module is connected to the SPPF module, and the output end of the SPPF module outputs 1 / 32 scale features.

[0032] The CBS modules all include 2D convolution units, 2D BatchNorm units, and SiLU units.

[0033] The 2DC3 module includes a 2DC3 convolution unit, a concat unit and two CBS modules. The 2DC3 convolution unit and one CBS module are connected in parallel and have the same input nodes and output nodes. The output nodes of the 2DC3 convolution unit and the CBS module are connected in parallel and are connected to the concat unit, and the concat unit is connected to another CBS module. The 2DC3 convolution unit includes multiple CBS modules connected in series.

[0034] The SPPF module includes two CBS modules, three maxpool units, and a concat unit. The first and second maxpool units are connected in series, and the third maxpool unit is connected in parallel. The input of the SPPF module is connected to the CBS module. The CBS module has multiple outputs, connected to the first, third, and concat units. The output of the first maxpool unit is connected to the second maxpool unit and the concat unit. The output of the third maxpool unit is connected to the concat unit. The output of the second maxpool unit is connected to the concat unit. The output of the concat unit is connected to another CBS module.

[0035] The input module has the same number of input ports as the 2D backbone module. Each input port is configured with the key frame as the center according to the length of the input video, and samples are sampled at equal intervals. The sampled images are input into each 2D backbone module in sequence.

[0036] In one embodiment, the data annotation module outputs a minimum feature map that is 1 / 32 of the original image size, so the length and width of the image need to be set to integer multiples of 32. In this experiment, the length and width of the image are set to 320 and 320, respectively.

[0037] The neck module includes an FPN module, a first PAN module, and a second PAN module. The backbone module outputs three different deep feature maps: shallow feature maps with stronger positional information and weaker semantic features, while deep feature maps have stronger semantic features and weaker localization information. The FPN module transfers deep semantic information to shallow feature maps to enhance semantic expression at multiple scales. The PAN module transfers shallow localization information to deep feature maps to enhance multi-scale localization capabilities.

[0038] The FPN module includes a 3DCBS unit, an upsample unit, a concat unit, and a 3DC3 convolution module. The PAN module includes a 3DCBS unit, a concat unit, and a 3DC3 convolution module.

[0039] The 3DCBS unit includes a 3D convolution unit, a 3D BatchNorm unit, and a SiLU unit.

[0040] The 3DC3 convolution module includes a 3DC3 convolution unit, a concat unit and two 3DCBS units. The 3DC3 convolution unit and one 3DCBS unit are connected in parallel and have the same input node and output node. The output node of the 3DC3 convolution unit and the 3DCBS unit are connected in parallel to the concat unit, and the concat unit is connected to another 3DCBS unit. The 3DC3 convolution unit includes multiple 3DCBS units connected in series.

[0041] In the neck module, all 2D convolutions are expanded to 3D convolutions. The kernel of the 3D convolution is expanded to a cube, and the stride is 1 in the time dimension. All 2D Batchnorms are replaced with 3D Batchnorms, and the scale factor in the upsampling is 1 in the time dimension.

[0042] In one embodiment, in the FPN module, the first 3DCBS unit, the first Upsample unit, the first concat unit, the first 3DC3 convolution unit, the second 3DCBS unit, the second Upsample unit, the second concat unit, and the second 3DC3 convolution unit are connected in sequence. The input of the first 3DCBS unit is connected to multiple SPPF modules to obtain multiple 1 / 32 scale features, and the output of the first 3DCBS unit is connected to the second PAN module and the first Upsample unit. The input of the first concat unit is connected to the first Upsample unit and the 2DC3 module to obtain 1 / 16 scale features. The output of the second 3DCBS unit is connected to the first PAN module. The input of the second concat unit is connected to the second Upsample unit and the 2DC3 module to obtain 1 / 8 scale features. The output of the second 3DC3 convolution unit outputs second prediction data and is connected to the first PAN module and the second PAN module.

[0043] The 3DCBS unit, concat unit, and 3DC3 convolution module in the first PAN module are connected in sequence. The input of the 3DCBS unit is connected to the output of the second 3DC3 convolution unit in the FPN module. The input of the concat unit is connected to the 3DCBS unit and also to the second 3DCBS unit in the FPN module. The output of the 3DC3 convolution module outputs the first prediction data.

[0044] The 3DCBS unit, concat unit, and 3DC3 convolution module in the second PAN module are connected in sequence. The 3DCBS unit input is connected to the output of the second 3DC3 convolution unit in the FPN module; the concat unit input is connected to the 3DCBS unit and also to the first 3DCBS unit in the FPN module; and the 3DC3 convolution module outputs the third predicted data.

[0045] The prediction module includes a first prediction data unit, a second prediction data unit and a third prediction data unit, which are respectively used to receive the first prediction data, the second prediction data and the third prediction data. The prediction data units each include a View unit and a 2D convolution unit.

[0046] The prediction module combines the temporal and RGB dimensions of the feature maps into a single dimension and convolves the three feature maps in a single layer, resulting in feature maps of sizes 3x40x40x20, 3x20x20x20, and 3x10x10x20, respectively. The 3 represents the number of object boxes predicted for each feature point, which aligns with the predefined anchor boxes for each feature point. 40x40, 20x20, and 10x10 represent the size of the feature maps. Therefore, the three feature maps predict 4800, 1200, and 300 object boxes, respectively.

[0047] Each object frame is represented by 20 numbers, of which four numbers represent the location information of the cuboid, used to calculate Lbox; one number represents the monkey's confidence, used to calculate the loss caused by the object Lobject; five numbers represent the probability of the monkey's five collar colors, used to calculate the loss caused by the color Lcolor; and 10 numbers represent the probability of the monkey's 10 actions, used to calculate the loss caused by the action Laction. The total loss is calculated as follows:

[0048] Ltotal=k1*Lbox+k2*Lobject+k3*Lcolor+k4*Laction

[0049] Among them, k1, k2, k3 and k4 are preset hyperparameters.

[0050] Experimental results:

[0051] As shown in Table 1, the slow fusion detection and classification network based on YOLOv5m achieved better average accuracy for monkey movements with fewer parameters and computational overhead. When YOLOv5l was used as the base detection network for improvement, the average accuracy for monkey movements was significantly improved. 3D slow fusion also achieved better performance for both the average accuracy of monkey movements and the color of monkey collars.

[0052] Table 1

[0053]

[0054] The present invention is not limited to the above-mentioned specific embodiments. A person skilled in the art can implement the present invention in a variety of other specific embodiments based on the embodiments and the contents disclosed in the drawings. Therefore, any design that adopts the design structure and ideas of the present invention and makes some simple transformations or changes falls within the scope of protection of the present invention.

Claims

1. A 3D slow-fusion joint detection and classification group monkey spatiotemporal action localization network, characterized by: The group monkey spatiotemporal action localization network includes an input module, a trunk module, a neck module, and a prediction module; The backbone module includes multiple identical 2D backbone modules connected in parallel, and the 2D backbone module includes multiple CBS modules, 2DC3 modules and an SPPF module. The 2D backbone module outputs features at three scales of 1 / 8, 1 / 16 and 1 / 32 of the original image respectively; The input module has the same number of input ports as the 2D backbone modules, and the input ports are connected to the corresponding 2D backbone modules; each input port inputs the original image of each sampling frame in sequence; The neck module includes an FPN module, a first PAN module, and a second PAN module. The FPN module is connected to the trunk module to obtain features at three scales of 1 / 8, 1 / 16, and 1 / 32 of the original image for enhancing semantic expression at multiple scales. The first PAN module and the second PAN module are connected to the FPN module to transfer shallow positioning information to deep features to enhance multi-scale positioning capabilities. The prediction module convolves the three feature maps in one layer, obtains feature maps of three sizes, and generates prediction results; In the FPN module, the first 3DCBS unit, the first Upsample unit, the first concat unit, the first 3DC3 convolution unit, the second 3DCBS unit, the second Upsample unit, the second concat unit and the second 3DC3 convolution unit are connected in sequence; the input end of the first 3DCBS unit is connected to multiple SPPF modules to obtain multiple 1 / 32 scale features, and the output end of the first 3DCBS unit is connected to the second PAN module and the first Upsample unit; the input end of the first concat unit is connected to the first Upsample unit and the 2DC3 module to obtain 1 / 16 scale features; the output end of the second 3DCBS unit is connected to the first PAN module; the input end of the second concat unit is connected to the second Upsample unit and the 2DC3 module to obtain 1 / 8 scale features; The output end of the second 3DC3 convolution unit outputs the second prediction data and is connected to the first PAN module and the second PAN module; The 3DCBS unit, concat unit, and 3DC3 convolution module in the first PAN module are connected in sequence; the input end of the 3DCBS unit is connected to the output end of the second 3DC3 convolution unit of the FPN module; the input end of the concat unit is connected to the 3DCBS unit and also to the second 3DCBS unit of the FPN module; the output end of the 3DC3 convolution module outputs the first prediction data; The 3DCBS unit, concat unit and 3DC3 convolution module in the second PAN module are connected in sequence; the input end of the 3DCBS unit is connected to the output end of the second 3DC3 convolution unit of the FPN module; the input end of the concat unit is connected to the 3DCBS unit and also connected to the first 3DCBS unit of the FPN module; the output end of the 3DC3 convolution module outputs the third prediction data.

2. The group monkey spatiotemporal motion localization network according to claim 1 is characterized in that: The 2D backbone module includes first to fifth CBS modules, first to third 2DC3 modules and an SPPF module; The output end of the first CBS module is connected to the second CBS module, the output end of the second CBS module is connected to the first 2DC3 module, the output end of the first 2DC3 module is connected to the third CBS module, the output end of the third CBS module is connected to the second 2DC3 module, the output end of the second 2DC3 module outputs 1 / 8 scale features of the original image and is connected to the fourth CBS module, the output end of the fourth CBS module is connected to the third 2DC3 module, the output end of the third 2DC3 module outputs 1 / 16 scale features of the original image and is connected to the fifth CBS module, the output end of the fifth CBS module is connected to the fourth 2DC3 module, the output end of the fourth 2DC3 module is connected to the SPPF module, and the output end of the SPPF module outputs 1 / 32 scale features.

3. The spatiotemporal motion localization network for monkey groups according to claim 2, characterized in that: The CBS modules each include a 2D convolution unit, a 2D BatchNorm unit, and a SiLU unit.

4. The group monkey spatiotemporal motion localization network according to claim 3 is characterized in that: The 2DC3 module includes a 2DC3 convolution unit, a concat unit and two CBS modules. The 2DC3 convolution unit and one CBS module are connected in parallel and have the same input node and output node. The output node of the 2DC3 convolution unit and the CBS module are connected in parallel to the concat unit, and the concat unit is connected to another CBS module. The 2DC3 convolution unit includes multiple CBS modules connected in series.

5. The group monkey spatiotemporal motion localization network according to claim 3 is characterized in that: The SPPF module includes two CBS modules, three maxpool units and a concat unit; the first and second maxpool units of the three maxpool units are connected in series and in parallel with the third maxpool unit; The input of the SPPF module is connected to the CBS module; The CBS module has multiple output ends, which are respectively connected to the first maxpool unit, the third maxpool unit and the concat unit; the output end of the first maxpool unit is connected to the second maxpool unit and the concat unit; the output end of the third maxpool unit is connected to the concat unit; the output end of the second maxpool unit is connected to the concat unit; and the output end of the concat unit is connected to another CBS module.

6. The monkey group spatiotemporal motion localization network according to claim 1, characterized in that: The FPN module includes a first 3DCBS unit, a first Upsample unit, a first concat unit, a first 3DC3 convolution unit, a second 3DCBS unit, a second Upsample unit, a second concat unit and a second 3DC3 convolution unit, which are connected in sequence.

7. The monkey group spatiotemporal motion localization network according to claim 6, characterized in that: The 3DCBS unit includes a 3D convolution unit, a 3D BatchNorm unit and a SiLU unit.

8. The monkey group spatiotemporal motion localization network according to claim 7, characterized in that: The 3DC3 convolution module includes a 3DC3 convolution unit, a concat unit and two 3DCBS units. The 3DC3 convolution unit and one 3DCBS unit are connected in parallel and have the same input node and output node. The output node of the 3DC3 convolution unit and the 3DCBS unit are connected in parallel to the concat unit, and the concat unit is connected to another 3DCBS unit. The 3DC3 convolution unit includes multiple 3DCBS units connected in series.

9. The spatiotemporal motion localization network for monkey groups according to claim 1, characterized in that: The prediction module generates position information, the monkey's confidence, the probability of the monkey's collar color, and the probability of the monkey's action for each frame of image; The total loss of the prediction module is: Ltotal=k1*Lbox+k2*Lobject+k3*Lcolor+k4*Laction Among them, k1, k2, k3 and k4 are preset hyperparameters, Lbox represents the loss of position information, Lobject represents the loss of the monkey's confidence, Lcolor represents the loss of the monkey's collar color, and Laction represents the loss of the monkey's action.

Images