A method and device suitable for underwater posture clustering

Abstract

The present invention discloses a method and device for underwater posture clustering, which relates to the technical field of underwater gesture recognition, including S1, learning useful image representation by training a deep network, so that target points are evenly dispersed in d-dimensional space; S2, normalizing the target conversion vector; if they are all evenly distributed, using mixed Gaussian distribution to project the vector onto a sphere; S3, while optimizing the clustering target, the convolutional neural network model learns image representation and classifies the image. The method and device for underwater posture clustering proposes an unsupervised clustering method, constructs a deep neural network in an end-to-end manner, and provides direct clustering assignment of posture images without additional processing; adopts multimodal deep clustering, trains the deep network to align the image embedding with the target points sampled from the Gaussian mixture model distribution, and determines the clustering assignment by associating the mixed components of the image embedding.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater gesture recognition, and in particular to a method and a device suitable for underwater gesture clustering. Background Art

[0002] In many underwater missions, autonomous underwater vehicles (AUVs) are accompanied by human divers who, unable to speak, communicate non-verbally through different gestures. However, underwater gesture recognition is a relatively underdeveloped field in computer vision due to the lack of annotated datasets. There are two main challenges: First, underwater images suffer from low contrast, blur, color distortion, and blur; therefore, traditional gesture recognition methods face problems in analyzing them. Second, existing gesture recognition models are mainly supervised and can only recognize gestures from a predefined set used to train the model. It is obviously impossible to enumerate all gestures and collect thousands of labeled images for every possible gesture that a human diver may use in the wild.

[0003] In addition, there are also certain problems with the scalability and maintainability of underwater gesture recognition algorithms. With the continuous development of underwater operations and the increasing application needs, gesture recognition algorithms need to be continuously updated and upgraded. However, the current underwater gesture recognition algorithms still have certain deficiencies in scalability and maintainability, and it is difficult to meet the needs of future applications.

[0004] Therefore, it is urgent to improve this shortcoming. The present invention studies and improves the existing technology and its shortcomings, and provides a method and device suitable for underwater posture clustering. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device suitable for underwater posture clustering to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, a method for underwater posture clustering is provided, comprising the following steps:

[0008] S1. Learn useful image representations by training a deep network to align its image embeddings with a fixed set of target points, so that the target points are evenly distributed in d-dimensional space;

[0009] Assumptions is a set of images, f θ :X→Z is the mapping operation performed by the deep neural network, f θThe outputs of are all normalized using the L2 normalization method (L2 normalization is a common data preprocessing technique, which is mainly used to scale each feature of the data to the same scale to improve the training efficiency and performance of the model);

[0010] Z is a d-dimensional vector;

[0011] The so-called target point refers to the mapping of a vector, from the embedding of the image, uniformly mapped to the representation of another vector space; assuming is the mapped vector, each image x i , needs to be mapped to a y through a mapping P i (y is the label, y∈y i , the value of y is 1-K, image classification), and the formula of the optimization objective is expressed as:

[0012]

[0013] Among them, y i =t p (i), l represents the Euclidean distance;

[0014] The optimization problem is solved in a random manner by iteratively on randomly sampled mini-batches; the specific operation is: given a mini-batch of images X b (b stands for batch), and then calculate f θ (X b ), then Formula 1 uses the Hungarian algorithm to calculate the number of b P is optimized, and the optimization goal is to minimize the Euclidean distance between the source image embedding and the target conversion vector; the specific implementation is: the training of the parameters of the calculation neural network; through the training parameters, the image embedding can be passed to complete the mapping of the target vector; subsequently, the grouping and clustering operations are performed on the mapped target vector space;

[0015] S2 and the target transformation vector are all normalized. If they are evenly distributed, they are not suitable for unsupervised clustering, and the boundaries between clusters will be blurred. Multimodal distribution needs to be introduced because it will cause separation in the target space. In this case, a mixed Gaussian distribution (GMM) is used to project the normalized vector onto the sphere:

[0016]

[0017] The probability density function of the Gaussian distribution is expressed as the weighted average of K Gaussian distributions, d is the dimension of the embedded vector space, α k is the weight, p k (u) is a normal distribution, or a Gaussian distribution N(μ k ,σ k), where the expectation is μ k , the covariance matrix is ​​σ k ; and assume that each Gaussian distribution is similar formula Indicates L2 normalization;

[0018] Imagex i The cluster assignment c i Use the following formula:

[0019]

[0020] Argmin is a common formula in machine learning and is defined as:

[0021]

[0022] S3. While optimizing the clustering target, the convolutional neural network model learns image representation and classifies images. In this case, AlexNet is used as the basic network of the convolutional neural network, and the loss function is defined as:

[0023]

[0024] Among them, g(X b ) is a flip transformation, which transforms the image X labeled y i Perform several specified flip operations, y is the label, θ is the parameter of the network, F y (·) is the result of flipping the image with the predicted label y (the result predicted by the neural network is a probability distribution);

[0025] The convolutional neural network adopts the AlexNet network structure by default:

[0026] Relu is used as the activation function; a 3×3 maximum pooling operation is added between the 2nd and 3rd convolutional layers;

[0027] The difference from the original AlexNet is that the original AlexNet structure was divided into two parts due to insufficient computing power, and this case merged them into one part;

[0028] The fully connected network after the convolutional neural network adopts a 3-layer structure by default, with 4096 neurons in each layer, and the activation function adopts relu, which can be adjusted according to specific circumstances.

[0029] Furthermore, in step S1, the L2 normalization method scales each feature vector in the data set so that the L2 norm of the scaled feature vector is equal to a predefined value, usually 1. The L2 norm is the Euclidean distance or the modulus of the vector.

[0030] Furthermore, the detailed process of step S2 is described as follows:

[0031] The main parameters can be adjusted as needed:

[0032] Input Image f θ Represents two convolutional neural networks, k represents the number of classifications required, iters represents the number of iterations, epoch represents the round of training, σ represents the variance of the distribution, d represents the dimension of the embedding vector, λ represents the learning rate and is a parameter, g represents the random image flipping operation, and r represents the number of image flipping instances in a batch;

[0033] The loss function Loss is shown in formula 2;

[0034] For each epoch, that is, each round of training:

[0035] Iterate i=1 to iters times:

[0036] Get a small batch of initially classified data from the data;

[0037] Calculate f θ (X b );

[0038] Update P by minimizing formula 1;

[0039] The flip transformation g(X b ), then calculate the loss Loss through formula 1, and update the network parameter gradient according to the loss Loss;

[0040] Update the neural network parameters θ according to the λ learning rate;

[0041] Iterate i=1 to iters times:

[0042] Get a small batch of X from the data b ;

[0043] Flip {0°, 90°, 180°, 270°};

[0044] Input the flipped image into the neural network, and the loss function is Loss;

[0045] Update the neural network parameters θ according to the λ learning rate;

[0046] The above two parts are mainly for training formula 1 and neural network.

[0047] Furthermore, the AlexNet network structure specifically includes: input layer, convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, and the height, width, and number of channels of each layer are represented by (h×w×c), which are: input layer (224×224×3), convolution layer 1 (55×55×96), convolution layer 2 (27×27×256), convolution layer 3 (13×13×384), convolution layer 4 (13×13×384), and convolution layer 5 (13×13×256).

[0048] In a second aspect, a device suitable for underwater posture clustering is provided, which is applied to the method suitable for underwater posture clustering as described above. The device includes: a memory, a processor, and computer program instructions stored on the memory and executable on the processor. When the processor executes the computer program instructions, the method suitable for underwater posture clustering as described above is implemented.

[0049] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method applicable to underwater posture clustering as described above is used.

[0050] The present invention provides a method and device suitable for underwater posture clustering, which has the following beneficial effects:

[0051] The present invention proposes an unsupervised clustering method, which constructs a deep neural network in an end-to-end manner to provide direct clustering assignment of posture images without additional processing; multimodal deep clustering is adopted to train the deep network to align image embeddings with target points sampled from the Gaussian mixture model distribution, and the clustering assignment is determined by associating the mixed components of the image embeddings; at the same time, the deep network is trained to solve the prediction of image rotation, promote better clustering, and is suitable for unstable underwater shooting environments.

[0052] The present invention can accurately classify unknown images by learning the features of known spaces, and can achieve the goal of correctly classifying unknown samples by learning a portion of known gesture samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A logical flow diagram of a method for underwater posture clustering according to the present invention;

[0054] Figure 2 A schematic diagram of a convolutional neural network suitable for underwater posture clustering method according to the present invention. DETAILED DESCRIPTION

[0055] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0056] like Figure 1-Figure 2 As shown, a method suitable for underwater posture clustering includes the following steps:

[0057] S1. Learn useful image representations by training a deep network to align its image embeddings with a fixed set of target points, so that the target points are evenly distributed in d-dimensional space;

[0058] Assumptions is a set of images, f θ :X→Z is the mapping operation performed by the deep neural network, f θ The outputs of are all normalized using the L2 normalization method (L2 normalization is a common data preprocessing technique, which is mainly used to scale each feature of the data to the same scale to improve the training efficiency and performance of the model);

[0059] The L2 normalization method scales each feature vector in the data set so that the L2 norm of the scaled feature vector is equal to a predefined value, usually 1. The L2 norm is the Euclidean distance or the modulus of the vector.

[0060] Z is a d-dimensional vector;

[0061] The so-called target point refers to the mapping of a vector, from the embedding of the image, uniformly mapped to the representation of another vector space; assuming is the mapped vector, each image x i , needs to be mapped to a y through a mapping P i (y is the label, y∈y i , the value of y is 1-K, image classification), and the formula of the optimization objective is expressed as:

[0062]

[0063] Among them, y i =t p (i), l represents the Euclidean distance;

[0064] The optimization problem is solved in a random manner by iteratively on randomly sampled mini-batches; the specific operation is: given a mini-batch of images X b (b stands for batch), and then calculate f θ (X b), then Formula 1 uses the Hungarian method (see https: / / blog.csdn.net / u013384984 / article / details / 90718287 for details) to calculate the number of pixels in this small batch of X. b P is optimized, and the optimization goal is to minimize the Euclidean distance between the source image embedding and the target conversion vector; the specific implementation is: the training of the parameters of the calculation neural network; through the training parameters, the image embedding can be passed to complete the mapping of the target vector; subsequently, the grouping and clustering operations are performed on the mapped target vector space;

[0065] S2 and the target transformation vector are all normalized. If they are evenly distributed, they are not suitable for unsupervised clustering, and the boundaries between clusters will be blurred. Multimodal distribution needs to be introduced because it will cause separation in the target space. In this case, a mixed Gaussian distribution (GMM) is used to project the normalized vector onto the sphere:

[0066]

[0067] The probability density function of the Gaussian distribution is expressed as the weighted average of K Gaussian distributions, d is the dimension of the embedded vector space, α k is the weight, p k (u) is a normal distribution, or a Gaussian distribution N(μ k ,σ k ), where the expectation is μ k , the covariance matrix is ​​σ k ; Here we assume that each Gaussian distribution is similar formula Indicates L2 normalization; (For a general introduction to GMM, refer to https: / / blog.csdn.net / qq_52466006 / article / details / 127186276. This case only uses this method);

[0068] Imagex i The cluster assignment c i Use the following formula:

[0069]

[0070] Argmin is a common formula in machine learning and is defined as:

[0071]

[0072] The detailed process of this case is described as follows:

[0073] The main parameters can be adjusted as needed:

[0074] Input Image f θ Represents two convolutional neural networks, k represents the number of classifications required, iters represents the number of iterations, epoch represents the round of training, σ represents the variance of the distribution, d represents the dimension of the embedding vector, λ represents the learning rate and is a parameter, g represents the random image flipping operation, and r represents the number of image flipping instances in a batch;

[0075] The loss function Loss is shown in formula 2;

[0076] For each epoch, that is, each round of training:

[0077] Iterate i=1 to iters times:

[0078] Get a small batch of initially classified data from the data;

[0079] Calculate f θ (X b );

[0080] Update P by minimizing formula 1;

[0081] The flip transformation g(X b ), then calculate the loss Loss through formula 1, and update the network parameter gradient according to the loss Loss;

[0082] Update the neural network parameters θ according to the λ learning rate;

[0083] Iterate i=1 to iters times:

[0084] Get a small batch of X from the data b ;

[0085] Flip {0°, 90°, 180°, 270°};

[0086] Input the flipped image into the neural network, and the loss function is Loss;

[0087] Update the neural network parameters θ according to the λ learning rate;

[0088] The above two parts are mainly for training formula 1 and neural network;

[0089] Data augmentation is a common technique to improve the performance of machine learning algorithms. Random image transformations such as cropping, flipping, rotation, scaling, and photometric transformations are applied to images in order to expand the dataset with new and unique images. In the unsupervised clustering task in this case, these random transformations provide several instances of the same image that look different but have the same semantics since they contain the same objects;

[0090] Let g represent a random image transformation operation. When training a convolutional neural network, when fitting the original image and the transformed image, it is necessary to find the implicit common semantics between them; the semantics is likely to appear in other images in the dataset belonging to the same class, which prompts the network to map images containing the same object to closer positions to each other with a beneficial effect similar to self-supervision;

[0091] S3. While optimizing the clustering target, the convolutional neural network model learns image representation and classifies images. The success of unsupervised clustering is highly related to the quality of the learned representation. Combining supervised methods can significantly improve the quality of representation obtained in unsupervised learning scenarios. In this case, AlexNet is used as the basic network of the convolutional neural network, and the loss function is defined as:

[0092]

[0093] Among them, g(X b ) is a flip transformation, which transforms the image X labeled y i Perform several specified flip operations, y is the label, θ is the parameter of the network, F y (·) is the result of flipping the image with the predicted label y (the result predicted by the neural network is a probability distribution);

[0094] The "convolutional neural network" in the flowchart of this case uses the AlexNet network structure by default:

[0095] See Figure 2 , (h×w×c) under the number of layers represents the height, width, and number of channels, and the activation function uses relu; a 3×3 maximum pooling operation is added between the 2nd and 3rd convolutional layers;

[0096] The AlexNet network structure specifically includes: input layer, convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, and the height, width, and number of channels of each layer are represented by (h×w×c), which are: input layer (224×224×3), convolution layer 1 (55×55×96), convolution layer 2 (27×27×256), convolution layer 3 (13×13×384), convolution layer 4 (13×13×384), convolution layer 5 (13×13×256);

[0097] The difference from the original AlexNet is that the original AlexNet structure was divided into two parts due to insufficient computing power, and this case merged them into one part;

[0098] The "fully connected network" after the "convolutional neural network" in the flowchart of this case adopts a 3-layer structure by default, with 4096 neurons in each layer, and the activation function uses relu, which can be adjusted according to specific circumstances.

[0099] A device suitable for underwater posture clustering is applied to the method suitable for underwater posture clustering as mentioned above. The device includes: a memory, a processor, and computer program instructions stored in the memory and executable on the processor. When the processor executes the computer program instructions, the method suitable for underwater posture clustering as mentioned above is implemented.

[0100] A computer-readable storage medium stores computer-executable instructions, which are used to implement the above-mentioned method for underwater posture clustering when executed by a processor.

[0101] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A method suitable for underwater posture clustering, characterized in that: The following steps are involved: S1. Learn useful image representations by training a deep network to align its image embeddings with a fixed set of target points, so that the target points are evenly distributed in d-dimensional space; Assumptions is a set of images, f θ :X→Z is the mapping operation performed by the deep neural network, f θ The outputs are all normalized using the L2 normalization method; Z is a d-dimensional vector; The so-called target point refers to the mapping of a vector, from the embedding of the image, uniformly mapped to the representation of another vector space; assuming is the mapped vector, each image x i , needs to be mapped to a y through a mapping P i , the optimization objective formula is expressed as: Among them, y i =t p (i), l represents the Euclidean distance; The optimization problem is solved in a stochastic manner by iteratively on randomly sampled mini-batches; the specific operation is: given a mini-batch of images X b , then calculate f θ (X b ), then Formula 1 uses the Hungarian algorithm to calculate the number of b P is optimized, and the optimization goal is to minimize the Euclidean distance between the source image embedding and the target conversion vector; the specific implementation is: the training of the parameters of the calculation neural network; through the training parameters, the image embedding can be passed to complete the mapping of the target vector; S2 and target conversion vectors are all normalized. If they are evenly distributed, they are not suitable for unsupervised clustering, and the boundaries between clusters will be blurred. Multimodal distribution needs to be introduced, and mixed Gaussian distribution is used to project the normalized vectors onto the sphere: The probability density function of the Gaussian distribution is expressed as the weighted average of K Gaussian distributions, d is the dimension of the embedded vector space, α k is the weight, p k (u) is a normal distribution, or a Gaussian distribution N(μ k ,σ k ), where the expectation is μ k , the covariance matrix is ​​σ k ; and assume that each Gaussian distribution is similar formula Indicates L2 normalization; Imagex i The cluster assignment c i Use the following formula: Argmin is a common formula in machine learning and is defined as: S3. While optimizing the clustering target, the convolutional neural network model learns image representation and classifies images. AlexNet is used as the basic network of the convolutional neural network, and the loss function is defined as: Among them, g(X b ) is a flip transformation, which transforms the image X labeled y i Perform several specified flip operations, y is the label, θ is the parameter of the network, F y (·) is the result of flipping the image with predicted label y; The convolutional neural network adopts the AlexNet network structure by default: Relu is used as the activation function; a 3×3 maximum pooling operation is added between the 2nd and 3rd convolutional layers; The fully connected network after the convolutional neural network adopts a 3-layer structure by default, with 4096 neurons in each layer and relu as the activation function.

2. A method for underwater posture clustering according to claim 1, characterized in that: In step S1, the L2 normalization method scales each feature vector in the data set so that the L2 norm of the scaled feature vector is equal to a predefined value. The L2 norm is the Euclidean distance or the modulus of the vector.

3. A method for underwater posture clustering according to claim 1, characterized in that: The detailed process of step S2 is described as follows: The main parameters can be adjusted as needed: Input Image f θ Represents two convolutional neural networks, k represents the number of classifications required, iters represents the number of iterations, epoch represents the round of training, σ represents the variance of the distribution, d represents the dimension of the embedding vector, λ represents the learning rate and is a parameter, g represents the random image flipping operation, and r represents the number of image flipping instances in a batch; Loss function Loss see formula 2 For each epoch, that is, each round of training: Iterate i=1 to iters times: Get a small batch of initially classified data from the data; Calculate fθ(X b ); Update P by minimizing formula 1; The flip transformation g(X b ), then calculate the loss Loss through formula 1, and update the network parameter gradient according to the loss Loss; Update the neural network parameters θ according to the λ learning rate; Iterate i=1 to iters times: Get a small batch of X from the data b ; Flip {0°, 90°, 180°, 270°}; Input the flipped image into the neural network, and the loss function is Loss; Update the neural network parameters θ according to the λ learning rate; The above two parts are mainly for training formula 1 and neural network.

4. A method for underwater posture clustering according to claim 1, characterized in that: The AlexNet network structure specifically includes: input layer, convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, and the height, width, and number of channels of each layer are represented by (h×w×c), which are: input layer (224×224×3), convolution layer 1 (55×55×96), convolution layer 2 (27×27×256), convolution layer 3 (13×13×384), convolution layer 4 (13×13×384), and convolution layer 5 (13×13×256).

5. A device for underwater posture clustering, applied to the method for underwater posture clustering as claimed in any one of claims 1 to 4, characterized in that: The device comprises: a memory, a processor, and computer program instructions stored in the memory and executable on the processor, wherein when the processor executes the computer program instructions, the method for underwater posture clustering as described in any one of claims 1 to 4 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method for underwater posture clustering as described in any one of claims 1 to 4 above.

Images