A method and apparatus for classifying deformable objects based on shallow temporal feature extraction and tree fusion.

Abstract

This invention discloses a method and apparatus for classifying deformable objects based on shallow temporal feature extraction and tree-structured fusion. The method includes acquiring a sequence of color images of a deformable object during its deformation process; using multiple shallow temporal extraction modules with shared parameters to extract features from each frame of the color image by fusing multi-scale optical flow and historical shallow features; employing a tree-structured feature fusion module to recursively fuse the features of all color images according to a tree structure, obtaining a fused feature as the video feature; and using a fully connected layer to classify the video feature, with the category with the highest confidence level being the final classification result for the deformable object. This method and apparatus can achieve high-precision classification of deformable objects using only a sequence of color images acquired from a single viewpoint, without requiring additional viewpoint or type of sensor data or human assistance.

Description

Technical Field

[0001] This invention relates to the field of deformable object classification, specifically to a method and apparatus for deformable object classification based on shallow temporal feature extraction and tree fusion. Background Technology

[0002] Deformable objects are very common in daily life. Compared to rigid bodies, their shapes change with their own state or external forces, rather than remaining essentially unchanged like rigid bodies. This poses a significant challenge to their automatic recognition and manipulation. Deformable objects can be mainly divided into three categories: one-dimensional deformable objects, represented by ropes; two-dimensional deformable objects, represented by clothing and paper; and three-dimensional deformable objects, represented by meat, human tissue, and dough. Manipulation of deformable objects is crucial in many applications, such as folding and flattening clothing, tissue cutting and surgical suturing in medical surgical robots, assisting with dressing and undressing, origami, food cutting, and automatic clothing washing.

[0003] Deformable object classification, as a prerequisite task for deformable object manipulation, plays a crucial role in the accuracy of downstream tasks by correctly predicting the object's type. During deformable object manipulation, it is necessary to identify the object's category based on information collected from various sensors. Because the shape of objects is continuously variable, it is impossible to pre-collect all possible object poses for training the classification algorithm. Furthermore, key features may be occluded during object deformation, making it difficult for existing object classification algorithms based on single-frame images to achieve high accuracy.

[0004] Existing object continuous deformation classification algorithms all use the classification results of a single frame image as the basis, select the category with the largest proportion in the image sequence as the object classification result, or design a voting method to select the category with the highest confidence as the final classification result.

[0005] The main drawback of existing methods is that objects in a single frame may have a large occluded area. Although the acquired features contain some information about the object, key discriminative information cannot be obtained, thus failing to yield correct classification results based on a single frame. If the classification results of most frames are inaccurate, no matter how the voting method is adjusted, a correct object classification result may not be obtained if the classification results of only a single frame are used as the basis.

[0006] Furthermore, existing methods often rely on multiple sensors, such as color image sensors and depth sensors. Depth image sensors are relatively expensive, and using both sensors simultaneously requires pre-setting such as sensor calibration and time alignment, making the process complex. Moreover, due to parallax between multiple sensors, in practical applications, after image alignment between the depth sensor and the color image sensor, pixels at corresponding positions in the color image that the depth sensor could not capture will have no values, affecting subsequent recognition accuracy.

[0007] Patent document CN115731423 A discloses a method for classifying deformable objects, including the following steps: acquiring depth image sequences and video frame sequences of the deformable object during the deformation process; using a deep convolutional network to calculate the classification result of the deformable object in each depth image frame; calculating the optical flow change between adjacent video frames based on the video frame sequence; determining the validity of the optical flow change between adjacent video frames based on the classification results of two adjacent depth images; calculating the cumulative value of optical flow change for each object category based on the validity of the optical flow change; and selecting the object category with the largest cumulative value of optical flow change as the final classification result determined during the deformation process. This technical solution proposes a voting weight calculation method for each frame's classification result, which improves the classification accuracy of deformable objects. However, it still cannot effectively classify objects when the single-frame classification result is poor, and it requires both depth and color image sensors, resulting in high cost.

[0008] Therefore, a deformable object classification method is needed that integrates the features of each frame in an image sequence, and can achieve high-precision and robust classification results using only information collected by a single-view color image sensor. Summary of the Invention

[0009] In view of the above, the purpose of this invention is to provide a method and apparatus for classifying deformable objects based on shallow temporal feature extraction and tree fusion, so as to improve the accuracy of object classification while reducing the number and types of sensors required.

[0010] To achieve the above-mentioned objectives, this invention provides a method for classifying deformable objects based on shallow temporal feature extraction and tree fusion, comprising the following steps:

[0011] Obtain a sequence of color images of a deformable object during the deformation process;

[0012] Multiple shallow temporal extraction modules with shared parameters are used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features;

[0013] A tree-structured feature fusion module recursively fuses the features of all color images according to the tree structure, resulting in a fused feature as the video feature.

[0014] A fully connected layer is used to classify video features, and the category with the highest confidence is taken as the final classification result for deformable objects.

[0015] Preferably, the multiple shallow temporal extraction modules employing parameter sharing extract features from each frame of the color image by fusing multi-scale optical flow and historical shallow features, including:

[0016] A shallow part of the network based on a deep convolutional backbone network is used to extract shallow features of the current frame color image by fusing multi-scale optical flow;

[0017] The shallow features of the current frame color image are fused with the hidden state of the shallow features representing historical frames using a depthwise convolution-based gated recurrent unit to obtain shallow features containing information about the historical motion of deformable objects.

[0018] A deep sub-network based on a deep convolutional backbone network is used to extract deep features as features of the current frame color image based on shallow features containing historical motion information of deformable objects.

[0019] Preferably, the step of extracting shallow features of the current frame color image by fusing multi-scale optical flow from a shallow portion of the deep convolutional backbone network includes:

[0020] At time t, based on the color images I of frame t and frame (t-1) t and I t-1 Calculate the inter-frame dense optical flow variation:

[0021]

[0022] Among them, Flow t It is a matrix of size W×H×2 that stores the orthogonal optical flow values ​​of all pixels on the video frame. W and H are the width and height of the video frame, respectively. OpticalFlow(·) represents the dense optical flow calculation method.

[0023] Optical Flow t With the color image I of frame t t The fused input is obtained through the matrix concatenation operation Concat.

[0024]

[0025] BackboneShallow, a shallow part of a deep convolutional backbone network, is based on the fused input. Extract the initial shallow features x of frame t t :

[0026]

[0027] Optical Flow t Downsampling and combining with shallow features x t The connection yields the final shallow features.

[0028]

[0029] Among them, DS m (Flow t ) is Flow t downsampling The subsequent optical flow, This is the downsampling factor.

[0030] Preferably, the method of using a depthwise convolution-based gated recurrent unit to fuse the shallow features of the current frame color image with the hidden states of the shallow features representing historical frames to obtain shallow features containing historical motion information of deformable objects includes:

[0031] In the depthwise convolution-based gated recurrent unit, the shallow features of the current frame's color image are first considered. and shallow features h representing historical frames t-1 Calculate the input gate r of frame t t And the Forgotten Gate z t :

[0032]

[0033] Among them, h 0 Initialize as an all-zero matrix, G r and G z σ represents the kernel parameters of the convolutional layer, σ is the Sigmoid function, * represents the convolution operation, Concat represents the feature connection operation, and T represents the length of the video sequence.

[0034] Secondly, based on the input gate r t And the Forgotten Gate z t Calculate the shallow feature q in the output of frame t that contains information about the historical motion of the deformable object. t and hidden state h t :

[0035]

[0036] Among them, G q These are the convolution kernel parameters of the convolutional layer. The symbol · represents the pointwise matrix multiplication operation. The output q of the t-th frame... t This refers to shallow features that contain information about the historical motion of deformable objects, and Tanh represents the Tanh activation function.

[0037] Preferably, the step of using a deep sub-network based on a deep convolutional backbone network to extract deep features as features of the current frame color image based on shallow features containing historical motion information of deformable objects includes:

[0038] Extracting deep features q from shallow features t The input is fed into BackboneDeep, a deep partial network based on a deep convolutional backbone, to compute the features F of the color image in frame t. t :

[0039] F t =BackboneDeep(q t ), t=1,2,...,T

[0040] Where T represents the length of the video sequence.

[0041] Preferably, the feature fusion module employing a tree structure recursively fuses the features of all color images according to the number of features in the tree structure to obtain a fused feature as the video feature, including:

[0042] The features of each frame of color image are regarded as leaf nodes of a tree structure. The tree fusion unit with shared parameters is used to fuse the features in pairs until only one fused feature remains as the video feature.

[0043] The minimum depth D of the tree structure is calculated using the following formula:

[0044]

[0045] Among them, symbols This is a rounding up operation, where T is the length of the video sequence;

[0046] The depth of the tree structure is indexed by the {·} symbol at the index position. The bottom layer of the tree structure is:

[0047]

[0048] Where 0 represents the all-zero feature;

[0049] The tree structure is constructed iteratively using the following formula:

[0050]

[0051] Finally, the video features were obtained.

[0052] Preferably, the method of classifying video features using a fully connected layer, wherein the category with the highest confidence is used as the final classification result for deformable objects, includes:

[0053] Video features were analyzed using fully connected (FC) and softmax layers. Classification yields category confidence scores.

[0054]

[0055] The final classification result k is determined using the following formula. * (T):

[0056]

[0057] Where k represents a category of deformable objects, K represents the total number of categories that deformable objects can be classified into, and argmax(·) represents the maximum value to be filtered.

[0058] To achieve the above-mentioned objectives, this invention provides a deformable object classification device based on shallow temporal feature extraction and tree fusion, comprising an acquisition unit, a feature extraction unit, a feature fusion unit, and a classification module.

[0059] The acquisition unit is used to acquire a sequence of color images of a deformable object during the deformation process;

[0060] The feature extraction unit is used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features using multiple shallow temporal extraction modules with shared parameters.

[0061] The feature fusion unit is used to recursively fuse the features of all color images according to the tree structure of the feature fusion module to obtain a fused feature as a video feature.

[0062] The classification unit is used to classify video features using a fully connected layer, with the category with the highest confidence level being the final classification result for deformable objects.

[0063] To achieve the above-mentioned objectives, an embodiment of the present invention provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned deformable object classification method based on shallow temporal feature extraction and tree fusion.

[0064] To achieve the above-mentioned objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when processed and executed, implements the above-mentioned deformable object classification method based on shallow temporal feature extraction and tree fusion.

[0065] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0066] This invention extracts features from each frame of color image using a shallow temporal extraction module with shared parameters. A tree-structured feature fusion module then fuses the features of all image sequences into video features. A fully connected layer is used to classify the video features, with the category with the highest confidence level being the final classification result for deformable objects. This classification method effectively captures the motion state of deformable objects through the shallow temporal feature extraction module. Unlike existing algorithms that rely solely on single-frame classification results, this invention fuses features containing more information using a tree structure to obtain more complete object features, thereby improving the classification accuracy of deformable objects. Furthermore, this method only requires a single-viewpoint color image sequence to achieve high-precision deformable object classification, without the need for additional viewpoints or types of sensor data or human assistance. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of a deformable object classification method based on shallow temporal feature extraction and tree fusion provided in the embodiment;

[0069] Figure 2 This is a flowchart of a deformable object classification method based on shallow temporal feature extraction and tree fusion provided in the embodiment.

[0070] Figure 3 This is a flowchart illustrating the features of each frame of a color image provided in the embodiment;

[0071] Figure 4 This is a schematic diagram of the deformable object classification device based on shallow temporal feature extraction and tree fusion provided in the embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0073] To improve the robustness and accuracy of deformable object classification based on color image sequences acquired from only a single viewpoint, this embodiment provides a method and apparatus for deformable object classification based on shallow temporal feature extraction and tree fusion.

[0074] Figure 1 and Figure 2 These are flowcharts and block diagrams of a deformable object classification method based on shallow temporal feature extraction and tree fusion provided in the embodiments. Figure 1 and Figure 2 As shown in the embodiment, the deformable object classification method based on shallow temporal feature extraction and tree fusion includes the following steps:

[0075] Step 1: Obtain a sequence of color images of the deformable object during the deformation process.

[0076] In this embodiment, the deformable object will deform during the process of being grasped by the robotic arm, and color image data of the deformable object during the deformation process will be collected to form a color image sequence.

[0077] Step 2: Multiple shallow temporal extraction modules with shared parameters are used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features.

[0078] In the embodiments, such as Figure 2 As shown, a shallow temporal extraction module is used to extract features for each frame of the color image. Specifically, as... Figure 3 As shown, from frame 1 to the end of the color image sequence, the following steps are taken to extract image features frame by frame, including:

[0079] Step 2-1 involves using a shallow sub-network based on a deep convolutional backbone to extract shallow features of the current frame's color image by fusing multi-scale optical flow. Specifically, this includes:

[0080] At time t, based on the color images I of frame t and frame (t-1) t and I t-1 Calculate the inter-frame dense optical flow variation:

[0081]

[0082] Wherein, Flowt is a matrix of size W×H×2, storing the orthogonal optical flow values ​​of all pixels in the video frame, where W and H are the width and height of the video frame, respectively. OpticalFlow(·) represents the dense optical flow calculation method, specifically including the Horn-Schunck algorithm (see Horn BKP, Schunk B G. Determining optical flow[J]. Artificial intelligence, 1981, 17(1-3): 185-203.), the LiteFlowNet algorithm (see Hui TW, Tang X, Loy CC. Liteflownet: A lightweight convolutional neural network for optical flowestimation[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8981-8989.), and the PWCNet algorithm (see Sun D, ​​Yang X, Liu MY, et al. Pwc-net: CNNs for optical flow using pyramid, warping, and cost). volume[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2018: 8934-8943.) etc.

[0083] Optical Flow t With the color image I of frame t t The fused input is obtained through the matrix concatenation operation Concat.

[0084]

[0085] BackboneShallow, a shallow part of a deep convolutional backbone network, is based on the fused input. Extract the initial shallow features x of frame t t :

[0086]

[0087] The deep convolutional backbone network includes the ResNet-18 algorithm, the ResNet-50 algorithm, and the ResNet-101 algorithm (see K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016). In this embodiment, the shallow part of the deep convolutional backbone network specifically includes the conv1 layer and conv2_x layer of ResNet-like algorithms.

[0088] Optical Flow t Downsampling and combining with shallow features x t The connection yields the final shallow features.

[0089]

[0090] Among them, DS m (Flow t ) is Flow t downsampling The subsequent optical flow, This represents the downsampling factor. In this example, m is set to 4, meaning that... Scale downsampling.

[0091] Step 2-2: Using a depthwise convolution-based gated recurrent unit, the shallow features of the current frame's color image are fused with the hidden states of the shallow features representing historical frames to obtain shallow features containing information about the historical motion of deformable objects. Specifically, this includes:

[0092] In the depthwise convolution-based gated recurrent unit, the shallow features of the current frame's color image are first considered. and shallow features h representing historical frames t-1 Calculate the input gate r of frame t t And the Forgotten Gate z t :

[0093]

[0094] Among them, h 0 Initialize as an all-zero matrix, G r and G z σ represents the kernel parameters of the convolutional layer, σ is the Sigmoid function, * represents the convolution operation, Concat represents the feature connection operation, and T represents the length of the video sequence.

[0095] Secondly, based on the input gate r t And the Forgotten Gate z t Calculate the shallow feature q in the output of frame t that contains information about the historical motion of the deformable object. t and hidden state h t :

[0096]

[0097] Among them, G q These are the convolution kernel parameters of the convolutional layer. The symbol · represents the pointwise matrix multiplication operation. The output q of the t-th frame... t This refers to shallow features that contain information about the historical motion of deformable objects, and Tanh represents the Tanh activation function.

[0098] Steps 2-3 involve using a deep sub-network based on a deep convolutional backbone network to extract deep features from shallow features containing historical motion information of deformable objects, which are then used as features for the current frame's color image. Specifically, this includes:

[0099] Extracting deep features q from shallow features t The input is fed into BackboneDeep, a deep partial network based on a deep convolutional backbone, to compute the features F of the color image in frame t. t :

[0100] F t =BackboneDeep(q t ), t=1,2,...,T

[0101] Where T represents the length of the video sequence. In this embodiment, the deep part of the deep convolutional backbone network includes the conv3_x, conv4_x, and conv5_x layers of a ResNet-like algorithm.

[0102] The features of each extracted color image frame are stored in memory until the color sequence feature extraction of the video is completed.

[0103] Step 3: Use a tree-structured feature fusion module to recursively fuse the features of all color images according to the number of features in the tree structure, and obtain a fused feature as the video feature.

[0104] In this embodiment, the features of each frame of color image are regarded as leaf nodes of a tree structure, and the tree fusion unit with shared parameters is used to fuse the features in pairs until only one fused feature remains as the video feature.

[0105] The minimum depth D of the tree structure is calculated using the following formula:

[0106]

[0107] Among them, symbols This is a rounding up operation, where T is the length of the video sequence;

[0108] The depth of the tree structure is indexed by the {·} symbol at the index position. The bottom layer of the tree structure is:

[0109]

[0110] Where 0 represents the all-zero feature;

[0111] The tree structure is constructed iteratively using the following formula:

[0112]

[0113] Finally, the video features were obtained.

[0114] Step 4: Use a fully connected layer to classify video features, and use the category with the highest confidence as the final classification result for deformable objects.

[0115] In this embodiment, a fully connected (FC) layer and a softmax layer are used to process video features. Classification yields category confidence scores.

[0116]

[0117] The final classification result k is determined using the following formula. * (T):

[0118]

[0119] Where k represents a category of deformable objects, K represents the total number of categories that deformable objects can be classified into, and argmax(·) represents the maximum value to be filtered.

[0120] Examples illustrating the training strategy of the embodiments, such as Figure 2 As shown, this method uses a two-stage training strategy to train the shallow temporal extraction module and the feature fusion module separately. First, the features extracted from the last frame of the sequence, i.e., frame T... Classification is performed using fully connected layers and softmax layers. The class with the highest confidence is selected as the predicted class, and its cross-entropy loss is calculated with the ground truth class as the loss function. The shallow temporal extraction module shares parameters with the training parameters. After training convergence, the parameters of the shallow temporal extraction module are fixed, and then the fused video features are... Classification is performed using fully connected layers and softmax layers. The class with the highest confidence is selected as the predicted class. The cross-entropy loss is calculated between the predicted class and the ground truth class and used as the parameter of the tree fusion unit that shares the training parameters of the loss function 2 until training converges.

[0121] The deformable object classification method provided in this embodiment effectively captures the motion state of deformable objects through a shallow temporal feature extraction module. The features extracted in each frame contain information from historical frames, resulting in better feature extraction performance. Employing a post-feature fusion approach, rather than the single-frame classification result fusion method of existing algorithms, it can preserve the features of deformable objects and their motion in the image sequence to a greater extent. The fusion unit based on a deep convolutional neural network in the tree structure can adaptively adjust the weight of each frame of the video color image sequence, thereby obtaining video features that better represent the characteristics of the object, thus improving the classification accuracy of deformable objects.

[0122] Based on the same inventive concept, the embodiment also provides a deformable object classification device based on shallow temporal feature extraction and tree fusion, such as... Figure 4 As shown, it includes an acquisition unit, a feature extraction unit, a feature fusion unit, and a classification module;

[0123] The acquisition unit is used to acquire a sequence of color images of a deformable object during the deformation process; the feature extraction unit is used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features through multiple shallow time-series extraction modules with shared parameters; the feature fusion unit is used to recursively fuse the features of all color images according to the tree structure of the feature fusion module to obtain a fused feature as the video feature; and the classification unit is used to classify the video features using a fully connected layer, with the category with the highest confidence being the final classification result of the deformable object.

[0124] It should be noted that the deformable object classification device based on shallow temporal feature extraction and tree fusion provided in the above embodiments should be illustrated by the division of the above functional units when classifying deformable objects. The functions can be assigned to different functional units as needed, i.e., the internal structure of the terminal or server can be divided into different functional units to complete all or part of the functions described above. Furthermore, the deformable object classification device based on shallow temporal feature extraction and tree fusion provided in the above embodiments belongs to the same concept as the deformable object classification method embodiment based on shallow temporal feature extraction and tree fusion. For details of its implementation process, please refer to the deformable object classification method embodiment based on shallow temporal feature extraction and tree fusion, which will not be repeated here.

[0125] The embodiment also provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described deformable object classification method based on shallow temporal feature extraction and tree fusion, including the following steps:

[0126] Step 1: Obtain a sequence of color images of the deformable object during the deformation process;

[0127] Step 2: Multiple shallow temporal extraction modules with shared parameters are used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features;

[0128] Step 3: Use a tree-structured feature fusion module to recursively fuse the features of all color images according to the number of features in the tree structure, and obtain a fused feature as the video feature;

[0129] Step 4: Use a fully connected layer to classify video features, and use the category with the highest confidence as the final classification result for deformable objects.

[0130] The memory can be volatile memory located at the near end, such as RAM, or non-volatile memory, such as ROM, FLASH, floppy disk, hard disk, etc., or it can be a remote storage cloud. The processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP), or field-programmable gate array (FPGA), that is, the steps of the deformable object classification method can be implemented through these processors.

[0131] The embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is processed and executed, it implements the above-described deformable object classification method based on shallow temporal feature extraction and tree fusion, including the following steps:

[0132] Step 1: Obtain a sequence of color images of the deformable object during the deformation process;

[0133] Step 2: Multiple shallow temporal extraction modules with shared parameters are used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features;

[0134] Step 3: Use a tree-structured feature fusion module to recursively fuse the features of all color images according to the number of features in the tree structure, and obtain a fused feature as the video feature;

[0135] Step 4: Use a fully connected layer to classify video features, and use the category with the highest confidence as the final classification result for deformable objects.

[0136] Computer-readable storage media can be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices, etc.

[0137] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for classifying deformable objects based on shallow temporal feature extraction and tree fusion, characterized in that, Includes the following steps: Obtain a sequence of color images of a deformable object during the deformation process; Multiple shallow temporal extraction modules with shared parameters are used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features. This includes: using a shallow part of the deep convolutional backbone network to extract shallow features of the current frame of color image by fusing multi-scale optical flow, specifically including: In the At that time, according to the first Frame and the Color image of a frame and Calculate the inter-frame dense optical flow variation: in, The size is The matrix stores the optical flow values ​​in orthogonal directions for all pixels on the video frame. and These are the width and height of the video frame, respectively. This describes the method for calculating dense optical flow. Optical flow With the Frame color image Through matrix join operation Get fused input : Shallow part of the network using a deep convolutional backbone Based on fusion input Extract the first Initial shallow features of the frame : Optical flow Downsampling and shallow features The connection yields the final shallow features. : in, yes downsampling The subsequent optical flow, This is the downsampling factor; The shallow features of the current frame color image are fused with the hidden state of the shallow features representing historical frames using a depthwise convolution-based gated recurrent unit to obtain shallow features containing information about the historical motion of deformable objects. A deep part of the deep convolutional backbone network is used to extract deep features from shallow features containing historical motion information of deformable objects as features of the current frame color image. A tree-structured feature fusion module is used to recursively fuse the features of all color images according to the tree structure to obtain a fused feature as the video feature. This includes: treating the features of each frame color image as leaf nodes of the tree structure, and using a parameter-shared tree fusion unit to fuse the features pairwise until only one fused feature remains as the video feature. The minimum depth of the tree structure is calculated using the following formula. : Among them, symbols This is a rounding up operation. The length of the video sequence; Use the index position The symbol index tree structure has a depth of 100%. The bottom layer of the tree structure is: in, Features of all zeros; The tree structure is constructed iteratively using the following formula: Finally, the video features were obtained. Fully connected layers are used to classify video features, and the category with the highest confidence is used as the final classification result for deformable objects.

2. The deformable object classification method based on shallow temporal feature extraction and tree fusion according to claim 1, characterized in that, The method uses a depthwise convolution-based gated recurrent unit to fuse the shallow features of the current frame's color image with the hidden states of the shallow features representing historical frames, obtaining shallow features containing information about the historical motion of deformable objects, including: In the depthwise convolution-based gated recurrent unit, the shallow features of the current frame's color image are first considered. and shallow features representing historical frames Calculate the first Frame input gate And the Gate of Oblivion : in, Initialize as a matrix of all zeros. and These are the convolution kernel parameters of the convolutional layer. for Function, symbol For convolution operations, For feature connection operations, T Indicates the length of the video sequence; Secondly, based on the input gate And the Gate of Oblivion Calculate the first Shallow features in frame output containing historical motion information of deformable objects and hidden state : in, These are the convolution kernel parameters of the convolutional layer, with symbols... For the pointwise multiplication operation of matrices, the first... Frame output This refers to shallow features that contain information about the historical motion of deformable objects. express Activation function.

3. The deformable object classification method based on shallow temporal feature extraction and tree fusion according to claim 1, characterized in that, The method of using a deep sub-network based on a deep convolutional backbone network to extract deep features as features of the current frame color image based on shallow features containing historical motion information of deformable objects includes: Extracting deep features from shallow features Input to a deep subnetwork based on a deep convolutional backbone Calculate the first Features of a frame color image : in, T Indicates the length of the video sequence.

4. The deformable object classification method based on shallow temporal feature extraction and tree fusion according to claim 1, characterized in that, The method employs fully connected layers to classify video features, with the category boasting the highest confidence level serving as the final classification result for deformable objects. This includes: Video features were analyzed using fully connected (FC) layers and a softmax layer. Classification yields category confidence scores. : The final classification result is determined using the following formula. : in, k This indicates a category of deformable objects. K This represents the total number of categories that a deformable object can be classified into. This indicates filtering for the maximum value.

5. A deformable object classification device based on shallow temporal feature extraction and tree fusion, characterized in that, It includes an acquisition unit, a feature extraction unit, a feature fusion unit, and a classification unit; The acquisition unit is used to acquire a video frame sequence of a deformable object during the deformation process; The feature extraction unit is used to extract features of each frame of color image by fusing multi-scale optical flow and historical shallow features through multiple shallow temporal extraction modules with shared parameters. This includes: using a shallow part of a deep convolutional backbone network to extract shallow features of the current frame of color image by fusing multi-scale optical flow, specifically including: In the At that time, according to the first Frame and the Color image of a frame and Calculate the inter-frame dense optical flow variation: in, The size is The matrix stores the optical flow values ​​in orthogonal directions for all pixels on the video frame. and These are the width and height of the video frame, respectively. This describes the method for calculating dense optical flow. Optical flow With the Frame color image Through matrix join operation Get fused input : Shallow part of the network using a deep convolutional backbone Based on fusion input Extract the first Initial shallow features of the frame : Optical flow Downsampling and shallow features The connection yields the final shallow features. : in, yes downsampling The subsequent optical flow, This is the downsampling factor; The shallow features of the current frame color image are fused with the hidden state of the shallow features representing historical frames using a depthwise convolution-based gated recurrent unit to obtain shallow features containing information about the historical motion of deformable objects. A deep sub-network based on a deep convolutional backbone network is used to extract deep features as features of the current frame color image based on shallow features containing historical motion information of deformable objects. The feature fusion unit is used to recursively fuse the features of all color images according to the tree structure of the feature fusion module to obtain a fused feature as a video feature. This includes: treating the features of each frame of color image as leaf nodes of a tree structure, and using a parameter-shared tree fusion unit to fuse the features in pairs until only one fused feature remains as a video feature. The minimum depth of the tree structure is calculated using the following formula. : Among them, symbols This is a rounding up operation. The length of the video sequence; Use the index position The symbol index tree structure has a depth of 100%. The bottom layer of the tree structure is: in, Features of all zeros; The tree structure is constructed iteratively using the following formula: Finally, the video features were obtained. ; The classification unit is used to classify video features using a fully connected layer, with the category with the highest confidence level being the final classification result for deformable objects.

6. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the deformable object classification method based on shallow temporal feature extraction and tree fusion as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is processed and executed, it implements the deformable object classification method based on shallow temporal feature extraction and tree fusion as described in any one of claims 1-4.

Images