Method and device suitable for underwater posture recognition

Abstract

The invention discloses a method and device suitable for underwater posture recognition, and relates to the technical field of underwater detection, and the method comprises the steps: S1, inputting an image, and extracting visual features through a feature extraction network; s2, constructing a position embedding, adding according to positions, and sending a result to a transformer encoder E; s3, extracting image features by adopting a pre-trained model CLIP; and S4, decoding the output result of the E, combining the decoded output result with the features extracted by the CLIP to serve as real features, and inputting the real features into a GAN structure. According to the method and device suitable for underwater posture recognition, the strong representation learning ability of a Transform structure is combined, the Transform structure serves as a visual feature extractor, then visual features without gestures are synthesized through a generative adversarial network, a gesture classifier can be trained through visible and non-visible data, and the weakness of supervised learning is overcome.

Description

Technical Field

[0001] The present invention relates to the field of underwater detection technology, and in particular to a method and a device suitable for underwater posture recognition. Background Art

[0002] Human divers face difficulties in collecting data from deep water due to issues such as increased hydrostatic pressure and oxygen. Over the past few decades, many have handed over the task to autonomous underwater robots (AUVs), which can capture underwater images / videos while being well-resistant to the problems faced by human divers. Therefore, they have been widely used in oceanography, naval warfare, information navigation, and ocean scene understanding.

[0003] In many underwater missions, AUVs are accompanied by human divers who, unable to speak, communicate non-verbally through different gestures. However, underwater gesture recognition is a relatively underdeveloped field in computer vision due to the lack of annotated datasets, mainly with the following two challenges: First, traditional gesture recognition methods face difficulties in analyzing underwater images due to their low contrast, blur, color distortion, and blur. Second, existing gesture recognition models are mainly supervised and can only recognize gestures from a predefined set used to train the model. It is obviously impossible to enumerate all gestures and collect thousands of labeled images for every possible gesture that a human diver might use in the wild.

[0004] Therefore, it is urgent to improve this shortcoming. The present invention studies and improves the existing technology and its shortcomings, and provides a method and device suitable for underwater posture recognition. Summary of the invention

[0005] The object of the present invention is to provide a method and a device suitable for underwater gesture recognition to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions: In a first aspect, a method for underwater gesture recognition is provided, comprising the following steps: S1. Input image ,C is the channel, H is the height, W is the width, and R is the real space. Visual features are extracted through the feature extraction network ; S2. Build a position embedding , vector dimensions and Same, dimension is recorded as , Will and Add them by position and send the result to the transformer encoder E (encoder E needs the spatial position of the visual feature as a token when calculating self-attention); S3, use the pre-trained model CLIP to extract image features; After decoding, the output results of S4 and E are combined with the features extracted by CLIP and input into the structure of GAN as real features. The GAN adopts conditional Wasserstein GAN, i.e. WGAN. S5. The trained WGAN is used to generate visual features of the unseen class, and the classifier is trained using the visible class data and the synthesized unseen class data to achieve zero-shot gesture recognition.

[0007] Furthermore, in step S1, the network for extracting the visual features of the image uses a network based on ResNet-50 as the backbone, and ResNet-50 is a configurable option. Selecting other structures as the backbone does not affect the results of this case.

[0008] Furthermore, in step S2, the input of the transformer encoder E is recorded as: The transformer performs self-attention calculations on the input token, and the output is ,The output carries the image context information and is fed into the decoder.

[0009] Furthermore, in step S3, the visual features extracted by CLIP , and visual feature extraction includes mapping projection to achieve Projected to the feature domain space of CLIP, parameters and are the vector dimensions of the model output and are preset constants. In the specific implementation, the parameters and Usually the default value is preset (the configurable value is a multiple of 64, and usually less than 1024), and and The constant value of is determined by the specific implementation of the selected pre-trained CLIP.

[0010] Furthermore, the transformer network part adopts a two-branch cross attention mechanism, and the operations of the two branches are the same, but with different values ​​of Q (query), K (key), and V (value) (QKV is a term related to the calculation of the attention mechanism in the transformer), and the two branches are respectively denoted as L (left) and R (right): And the output of attention adopts layer normalization method CorssAtt is a cross attention operation. For details, please refer to the relevant content of transformer. It is a fixed operation. The subsequent output of the left and right branches is passed through a The convolutional layer is used to add weight adjustment, and finally the GELU activation function is applied as the final output: Among them, W and b are the network parameters of the corresponding convolutional layer, and the output indicates how much mutual attention information is retained; the left and right branch attentions are fused through a forward feedback network (Feedforward network FFN) to form the final output .

[0011] Furthermore, in step S4, regarding the WGAN adversarial generative network part: Visual features as input , the semantic vector a is used as a conditional variable (a part of WGAN, a trainable parameter), the generator G tries to simulate the real distribution, and then generates a , and send it to the adversarial network as the judge D, and Make a comparison; The output of D is a classifier, which outputs true or false. As data is continuously input for training, D can improve the prediction performance through training, and continuously make the generated features close to the real feature distribution. The loss function is defined as follows: Among them, λ is the penalty coefficient, is the feature generated by the network, z is the noise; the generator is G, the discriminator is D, D(G(z,a)) represents the score of the discriminator on the generated feature; E represents the expectation, Indicates distance, usually L2, Euclidean distance; And in order to avoid mode collapse (that is, the results generated by the generator are true, but not diverse enough), another mode-seeking loss function is introduced: in, Different noises are introduced when generating different features, It represents the distance of L1, which is the absolute value; The final loss function optimization goal is: in, It is an adjustable parameter.

[0012] Furthermore, in step S5, regarding the classifier part: A classifier network is constructed. By default, a multi-layer fully connected network is used, and the loss function uses cross entropy. For the prediction of known gestures, the network of this case is input. The input of the classification network is the output of the gesture decoder, and the judgment is made directly according to the classification. Then the features generated by GAN, which belong to the synthesized unseen features and the visual features of the visible class extracted from the trained structure, are used as the input of the classifier to realize the prediction of unseen type samples. The results of the above two types of inputs are the same. The role is to use the generative network to generate many invisible samples based on the visible samples, train the network, and make up for the problem of insufficient samples.

[0013] In a second aspect, a device suitable for underwater gesture recognition is provided, which is applied to the method suitable for underwater gesture recognition as described above. The device includes: a memory, a processor, and computer program instructions stored in the memory and executable on the processor. When the processor executes the computer program instructions, the method suitable for underwater gesture recognition as described above is implemented.

[0014] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method for underwater gesture recognition as described above is implemented.

[0015] The present invention provides a method and device suitable for underwater posture recognition, which has the following beneficial effects: The present invention proposes a method that combines the strong representation learning ability of the Transform structure as a visual feature extractor, and then uses a generative adversarial network to synthesize visual features of unseen gestures, so that we can use data from both visible and unseen data to train a gesture classifier, overcome the weaknesses of supervised learning, and improve the performance of gesture recognition; thereby improving the accuracy of gesture communication between AUVs and human divers in underwater missions, which is conducive to better capture of underwater images / videos by AUVs, and brings more convenience to underwater detection work. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall process of a method suitable for underwater posture recognition according to the present invention; Figure 2 This is a partial flow chart of a transformer network of a method suitable for underwater posture recognition according to the present invention. DETAILED DESCRIPTION

[0017] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0018] like Figure 1-Figure 2As shown, a method suitable for underwater gesture recognition includes the following steps: S1. Input image ,C is the channel, H is the height, W is the width, and R is the real space. Visual features are extracted through the feature extraction network ; The network for extracting visual features of images uses a network based on ResNet-50 as the backbone, and ResNet-50 is a configurable option. Selecting other structures as the backbone does not affect the results of this case; S2. Build a position embedding , vector dimensions and Same, dimension is recorded as , Will and Add them by position and send the result to the transformer encoder E (encoder E needs the spatial position of the visual feature as a token when calculating self-attention); The input of transformer encoder E is recorded as: The transformer performs self-attention calculations on the input token, and the output is ,The output carries the context information of the image and is fed into the decoder; S3. Use the pre-trained model CLIP to extract image features (reference paper Radford, A., Kim, et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning, this case is only used); Visual features extracted by CLIP , and visual feature extraction includes mapping projection to achieve Projected into the feature domain space of CLIP; The transformer network uses a two-branch cross attention mechanism, and the operations of the two branches are the same, but with different values ​​of Q (query), K (key), and V (value) (QKV is a term related to the calculation of the attention mechanism in the transformer). The two branches are denoted as L (left) and R (right): And the output of attention adopts layer normalization method CorssAtt is a cross attention operation. For details, please refer to the relevant content of transformer. It is a fixed operation. The subsequent output of the left and right branches is passed through a The convolutional layer is used to add weight adjustment, and finally the GELU activation function is applied as the final output: Among them, W and b are the network parameters of the corresponding convolutional layer, and the output indicates how much mutual attention information is retained; the left and right branch attentions are fused through a forward feedback network (Feedforward networkFFN) to form the final output ; After decoding, the output results of S4 and E are combined with the features extracted by CLIP as real features and input into the structure of GAN, and GAN adopts conditional Wasserstein GAN, namely WGAN; the main details refer to (Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks). This case only uses the semantic features of the main generated class; About WGAN adversarial generation network: Visual features as input , the semantic vector a is used as a conditional variable (a part of WGAN, a trainable parameter), the generator G tries to simulate the real distribution, and then generates a , and send it to the adversarial network as the judge D, and Make a comparison; The output of D is a classifier, which outputs true or false. As data is continuously input for training, D can improve the prediction performance through training, and continuously make the generated features close to the real feature distribution. The loss function is defined as follows: Among them, λ is the penalty coefficient, is the feature generated by the network, z is the noise; the generator is G, the discriminator is D, D(G(z,a)) represents the score of the discriminator on the generated feature; E represents the expectation, Indicates distance, usually L2, Euclidean distance; And in order to avoid mode collapse (that is, the results generated by the generator are true, but not diverse enough), another mode-seeking loss function is introduced: in, Different noises are introduced when generating different features, It represents the distance of L1, which is the absolute value; The final loss function optimization goal is: in, It is an adjustable parameter; S5. The trained WGAN is used to generate visual features of the unseen class, and the classifier is trained using the visible class data and the synthesized unseen class data to achieve zero-shot gesture recognition. Regarding the classifier part: build a classifier network. By default, a multi-layer fully connected network is used, and the loss function uses cross entropy. For the prediction of known gestures, the network of this case is input. The input of the classification network is the output of the gesture decoder, and the judgment is made directly according to the classification; then the features generated by GAN, which belong to the synthesized unseen features and the visual features of the visible class extracted from the trained structure, are used as the input of the classifier to realize the prediction of unseen type samples; and the results of the above two types of inputs are the same. The role is to use the generative network to generate many invisible samples based on the visible samples, train the network, and make up for the problem of insufficient samples.

[0019] A device suitable for underwater gesture recognition is applied to the method suitable for underwater gesture recognition as described above. The device includes: a memory, a processor, and computer program instructions stored in the memory and executable on the processor. When the processor executes the computer program instructions, the method suitable for underwater gesture recognition as described above is implemented.

[0020] A computer-readable storage medium stores computer-executable instructions, which are used to implement the above-mentioned method for underwater gesture recognition when executed by a processor.

[0021] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A method for underwater gesture recognition, characterized in that: The following steps are involved: S1. Input image ,C is the channel, H is the height, W is the width, and R is the real space. Visual features are extracted through the feature extraction network ; S2. Build a position embedding , vector dimensions and Same, dimension is recorded as , Will and Add by position and send the result to transformer encoder E; S3, use the pre-trained model CLIP to extract image features; After decoding, the output results of S4 and E are combined with the features extracted by CLIP and input into the structure of GAN as real features. The GAN adopts conditional Wasserstein GAN, i.e. WGAN. S5. The trained WGAN is used to generate visual features of the unseen class, and the classifier is trained using the visible class data and the synthesized unseen class data to achieve zero-shot gesture recognition.

2. A method for underwater gesture recognition according to claim 1, characterized in that: In step S1, the network for extracting the visual features of the image uses a network based on ResNet-50 as the backbone, and ResNet-50 is a configurable option.

3. A method for underwater gesture recognition according to claim 1, characterized in that: In step S2, the input of the transformer encoder E is recorded as: The transformer performs self-attention calculations on the input token, and the output is ,The output carries the image context information and is fed into the decoder.

4. A method for underwater gesture recognition according to claim 1, characterized in that: In step S3, the visual features extracted by CLIP , and visual feature extraction includes mapping projection to achieve Projected to the feature domain space of CLIP, parameters and Both are vector dimensions of model output and are preset constants.

5. The method for underwater gesture recognition according to claim 3, characterized in that: The transformer network uses a two-branch cross attention mechanism, and the operations of the two branches are the same, but with different values ​​of Q, K, and V. The two branches are denoted by L and R respectively: And the output of attention adopts layer normalization method CorssAtt is the cross attention operation; The subsequent output of the left and right branches is passed through a The convolutional layer is used to add weight adjustment, and finally the GELU activation function is applied as the final output: Among them, W and b are the network parameters of the corresponding convolutional layer, and the output indicates how much mutual attention information is retained; the left and right branch attentions are fused through a forward feedback network to form the final output .

6. The method for underwater gesture recognition according to claim 1, characterized in that: In step S4, regarding the WGAN adversarial generative network part: Visual features as input , the semantic vector a is used as a conditional variable, the generator G tries to simulate the real distribution, and then generates a , and send it to the adversarial network as the judge D, and Make a comparison; The output of D is a classifier, which outputs true or false. As data is continuously input for training, D can improve the prediction performance through training, and continuously make the generated features close to the real feature distribution. The loss function is defined as follows: Among them, λ is the penalty coefficient, is the feature generated by the network, z is the noise; the generator is G, the discriminator is D, D(G(z,a)) represents the score of the discriminator on the generated feature; E represents the expectation, To express the distance, L2 and Euclidean distance are used; And in order to avoid mode collapse, another mode-seeking loss function is introduced: in, Different noises are introduced when generating different features, It represents the distance of L1, which is the absolute value; The final loss function optimization goal is: in, It is an adjustable parameter.

7. A method for underwater gesture recognition according to claim 1, characterized in that: In step S5, regarding the classifier part: A classifier network is constructed. By default, a multi-layer fully connected network is used, and the loss function uses cross entropy. For the prediction of known gestures, the network of this case is input. The input of the classification network is the output of the gesture decoder, and the judgment is made directly according to the classification. Then the features generated by GAN, which belong to the synthesized unseen features and the visual features of the visible class extracted from the trained structure, are used as the input of the classifier to realize the prediction of unseen type samples. The results of the above two types of inputs are the same.

8. A device for underwater gesture recognition, applied to the method for underwater gesture recognition as claimed in any one of claims 1 to 7, characterized in that: The device comprises: a memory, a processor, and computer program instructions stored in the memory and executable on the processor, wherein when the processor executes the computer program instructions, the method for underwater gesture recognition as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for underwater gesture recognition as described in any one of claims 1 to 7.

Images