A method for detecting underwater pin-to-hole

Abstract

The present invention discloses a method for detecting underwater pin-hole alignment, which specifically includes the following steps: (1) obtaining a video of the pin-hole alignment process and performing image preprocessing; (2) obtaining beam vibration frequency data and performing sound feature extraction; (3) using a pre-trained image recognition model and sound recognition model to obtain semantic-level image features and sound features; (4) using a cross-modal alignment model to achieve alignment of image and sound features; (5) using the aligned features for feature fusion and detecting whether the pin-hole alignment is correct through a deep learning model. The present invention adopts a cross-modal network model that can effectively integrate multiple types of data, provide richer and more comprehensive information, and improve the accuracy of underwater identification. The method can assist management personnel in monitoring the site, strengthen the standardization of project management, improve the overall efficiency of the project, and at the same time reduce cost consumption and the probability of accidents and casualties.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a method for detecting underwater pin-to-hole alignment. Background Art

[0002] With the rapid development of my country's economy and the increasing number of infrastructure construction projects, the application of pin-to-hole matching has become increasingly widespread. Therefore, the control and safety of pin-to-hole matching have become particularly important. Currently, on construction sites, the main method to ensure the safe control of pin-to-hole matching is manual supervision. However, due to the complexity of the construction site environment, it is impossible to ensure that workers are always within sight. Therefore, this method is inefficient and the safety of construction site personnel is difficult to guarantee.

[0003] In some special scenarios, such as underwater operations, the accuracy of the pin-to-hole alignment between grab beams and gates cannot be directly observed. Traditional image recognition technology is affected by the underwater environment and cannot function properly. Therefore, it is necessary to develop high-precision detection methods. To address these issues, further technical improvements are needed. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the existing technology and propose a method for detecting underwater pin-to-holes, which aims to solve the problem of accurate inspection of underwater pin-to-holes, solve the difficulty of accurate inspection in complex underwater scenes, and achieve accurate control of the safe docking of the grab beam and the gate.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for detecting underwater pin-to-hole alignment comprises the following steps:

[0007] (1) Obtaining a video of the pin-to-hole alignment process and performing image preprocessing;

[0008] (2) Obtaining the vibration frequency data of the grab beam and extracting the sound information;

[0009] (3) Using pre-trained image recognition models and sound recognition models to obtain semantic-level image features and sound features;

[0010] (4) Using a cross-modal alignment model to align image and sound features;

[0011] (5) Use the aligned features for feature fusion and use the deep learning model to detect whether the pins and holes are aligned.

[0012] Preferably, the step (1) includes the following sub-steps:

[0013] (1.1) Video frame extraction: The video is split into several frames using the equal-interval sampling method. The relevant formula is:

[0014] F t =V(t)

[0015] Among them, F t represents the t-th frame image, V(t) represents the t-th frame video;

[0016] (1.2) Image scaling and cropping: The extracted video frames are scaled and cropped to meet the size requirements of the model input and reduce the amount of calculation. The relevant formula is:

[0017] F t ′=crop(scale(F t ))

[0018] Among them, F t ' represents the image after scaling and cropping, crop represents the cropping operation, and scale represents the scaling operation;

[0019] (1.3) Image enhancement: Perform brightness enhancement, contrast enhancement, and color space transformation on an image to improve image quality and enhance image features. The relevant formula is:

[0020] F t =enhance(F t ′)

[0021] Among them F t ″ represents the image after image enhancement, enhance represents the image enhancement operation;

[0022] (1.4) Data standardization: The enhanced image is standardized so that each pixel value in the image has the same data range and mean. The standardization operation can make the model more stable and reliable. The relevant formula is:

[0023] F t ″′=normalize(F t ″)

[0024] Among them F t ″′ represents the normalized image, and normalize represents the data normalization operation;

[0025] (1.5) Data enhancement: Data enhancement is performed on the image to increase the data samples and enhance the generalization ability of the model. The relevant formula is:

[0026]

[0027] in Represents the image data after data enhancement, and augment represents the data enhancement operation.

[0028] Preferably, the step (2) includes the following sub-steps:

[0029] (2.1) Sampling and quantization of audio signals: converting continuous sound signals into digital signals;

[0030] (2.2) Pre-emphasis: Perform high-pass filtering on the input signal in advance. The relevant formula is:

[0031] s′(n)=s(n)-αs(n-1)

[0032] Where s(n) represents the original sound signal, s′(n) represents the preprocessed signal, and α is the pre-emphasis coefficient;

[0033] (2.3) Framing: The long-duration sound signal is divided into several short-duration frames. The Hamming window function is used to perform weighted smoothing on each frame. The window function is expressed as:

[0034] f(n)=w(n)s′(n)

[0035] Where f(n) represents the processed signal, w(n) is the Hamming window function;

[0036] (2.4) Fast Fourier Transform (FFT): Perform a fast Fourier transform on each frame of signal to convert the time domain signal into the frequency domain signal;

[0037] (2.5) Feature normalization: The extracted features are normalized so that they have zero mean and unit variance to facilitate subsequent modeling and training.

[0038] Preferably, the step (3) includes the following sub-steps:

[0039] (3.1) Use the Vision Transformer (ViT) pre-trained model to obtain image features: The input image is divided into fixed-size tiles. For each tile, it is compressed into a vector as the input of the model. Positional encoding is embedded in these vectors to help the model identify different parts of the image. These vectors are passed as input to the Transformer encoder, which contains multiple Transformer blocks. Each block consists of multiple self-attention layers and fully connected layers:

[0040] h image =ViT(x image )

[0041]

[0042]

[0043] Qimage =K image =V image =x image

[0044] where h image is the semantic-level image feature output by the ViT model, x image After image preprocessing The image information obtained after division, Multi_Att represents the self-attention module, Norm represents the normalization module, MLP represents the fully connected module, and d represents x image The dimension of , softmax represents the normalization function;

[0045] (3.2) The Conformer model is used to obtain sound features. The input sequence X is processed through a convolutional layer, which helps capture local patterns and interactions between adjacent features. It is then fed into a multi-head self-attention layer, which allows the model to focus on different parts of the input sequence. The relevant formula is as follows:

[0046] h sound =Conformer(X sound )

[0047]

[0048] Conv=ReLU(W1X+b1)W2+b2

[0049]

[0050] Q sound =K sound =V sound =X sound

[0051] where h sound is the semantic-level sound feature output by the Conformer model, X sound To extract the frequency domain signal X(x) of the sound, Conv represents the convolutional layer module, ReLU represents the activation function, W1 and W2 are the weight matrices of the convolutional layer, and b1 and b2 are the bias vectors of the convolutional layer.

[0052] Preferably, the step (4) includes the following sub-steps:

[0053] The step (4) includes the following sub-steps:

[0054] (4.1) The image features h obtained in steps (2) and (3) are image and sound characteristics h sound Mapped to the same shared latent space:

[0055] x i =g(h image θ g ),x j =g(h sound θ g )

[0056] where x i and x i denote the representation vectors of image and sound in the shared latent space, θ g is the parameter of the mapping function g;

[0057] (4.2) Using a training method based on contrast loss function, the correspondence between images and sounds is learned, expressed as:

[0058] L(x i ,x j ,y ij )=(1-y ij )max(0,mD(x i ,x j )) 2 +y ij D(x i ,x j ) 2

[0059] Where L(x i ,x j ,y ij ) represents the loss function between samples, y ij Is a binary variable indicating whether the two data come from the same set of data; the goal of the above model is to learn a joint representation so that similar image and sound data are close to each other in the shared latent space, while dissimilar data are pushed away; specifically, the loss function of the model is the average of the contrastive loss function of all training sample pairs:

[0060]

[0061] Where N is the total number of training sample pairs.

[0062] Preferably, the step (5) includes the following sub-steps:

[0063] (5.1) A multi-head attention mechanism is used for feature fusion. In the downstream task, a classifier is used to detect whether the pins and holes are aligned. The relevant formula is as follows:

[0064] x fusion =Multi_Att(x i ,x j )

[0065] Z=softmax(x fusion W cls +b cls )

[0066] where x fusion is x i and x j One-to-one correspondence with the fused features, Z is the inspection result, W cls and b cls are the weight and bias vectors of the classifier;

[0067] (5.2) Use the cross entropy loss function and the contrast loss function of step (4.2) as the loss function of the entire model to train the model, and fine-tune the ViT model and Conformer model of step (3):

[0068]

[0069] Where L is the loss function of the entire model, is the contrast loss function of step (4.2), and α and β are the loss function weights.

[0070] Compared with the prior art, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0071] (1) The present invention provides a method for underwater pin-to-hole detection. Compared with existing technologies, the present invention adopts a cross-modal network model that can effectively integrate multiple types of data, providing richer and more comprehensive information, thereby improving the accuracy of underwater recognition.

[0072] (2) The present invention provides a method for detecting underwater pin-to-hole alignment, which can assist management personnel in monitoring the site, strengthen the standardization of project management, and improve the overall efficiency of the project. At the same time, it can also reduce cost consumption and the probability of accidents and casualties. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 The present invention is a flowchart of an underwater pin-to-hole detection method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0075] See Figure 1, is a flow chart of a method for detecting underwater pin-hole alignment provided by an embodiment of the present invention; as a preferred embodiment of the present invention, this embodiment provides a method for detecting underwater pin-hole alignment, comprising the following steps:

[0076] (1) Obtaining a video of the pin-to-hole alignment process and performing image preprocessing;

[0077] (2) Obtaining the vibration frequency data of the grab beam and extracting the sound information;

[0078] (3) Using pre-trained image recognition models and sound recognition models to obtain semantic-level image features and sound features;

[0079] (4) Using a cross-modal alignment model to align image and sound features;

[0080] (5) Use the aligned features for feature fusion and use the deep learning model to detect whether the pins and holes are aligned.

[0081] In some preferred embodiments, step (1) includes the following sub-steps:

[0082] (1.1) Video frame extraction: The video is split into several frames using the equal-interval sampling method. The relevant formula is:

[0083] F t =V(t)

[0084] Among them, F t represents the t-th frame image, V(t) represents the t-th frame video;

[0085] (1.2) Image scaling and cropping: The extracted video frames are scaled and cropped to meet the size requirements of the model input and reduce the amount of calculation. The relevant formula is:

[0086] F t ′=crop(scale(F t ))

[0087] Among them, F t ' represents the image after scaling and cropping, crop represents the cropping operation, and scale represents the scaling operation;

[0088] (1.3) Image enhancement: Perform brightness enhancement, contrast enhancement, and color space transformation on an image to improve image quality and enhance image features. The relevant formula is:

[0089] F t =enhance(F t ′)

[0090] Among them F t″ represents the image after image enhancement, enhance represents the image enhancement operation;

[0091] (1.4) Data standardization: The enhanced image is standardized so that each pixel value in the image has the same data range and mean. The standardization operation can make the model more stable and reliable. The relevant formula is:

[0092] F t ″′=normalize(F t ″)

[0093] Among them F t ″′ represents the normalized image, and normalize represents the data normalization operation;

[0094] (1.5) Data enhancement: Data enhancement is performed on the image to increase the data samples and enhance the generalization ability of the model. The relevant formula is:

[0095]

[0096] in Represents the image data after data enhancement, and augment represents the data enhancement operation.

[0097] In some other preferred embodiments, step (2) includes the following sub-steps:

[0098] (2.1) Sampling and quantization of audio signals: converting continuous sound signals into digital signals;

[0099] (2.2) Pre-emphasis: Perform high-pass filtering on the input signal in advance. The relevant formula is:

[0100] s′(n)=s(n)-αs(n-1)

[0101] Where s(n) represents the original sound signal, s′(n) represents the preprocessed signal, and α is the pre-emphasis coefficient;

[0102] (2.3) Framing: The long-duration sound signal is divided into several short-duration frames. The Hamming window function is used to perform weighted smoothing on each frame. The window function is expressed as:

[0103] f(n)=w(n)s′(n)

[0104] Where f(n) represents the processed signal, w(n) is the Hamming window function;

[0105] (2.4) Fast Fourier Transform (FFT): Perform a fast Fourier transform on each frame of signal to convert the time domain signal into the frequency domain signal;

[0106] (2.5) Feature normalization: The extracted features are normalized so that they have zero mean and unit variance to facilitate subsequent modeling and training.

[0107] In some preferred embodiments, step (3) includes the following sub-steps:

[0108] (3.1) Use the Vision Transformer (ViT) pre-trained model to obtain image features: The input image is divided into fixed-size tiles. For each tile, it is compressed into a vector as the input of the model. Positional encoding is embedded in these vectors to help the model identify different parts of the image. These vectors are passed as input to the Transformer encoder, which contains multiple Transformer blocks. Each block consists of multiple self-attention layers and fully connected layers:

[0109] h image =ViT(x image )

[0110]

[0111]

[0112] Q image =K image =V image =x image

[0113] where h image is the semantic-level image feature output by the ViT model, x image After image preprocessing The image information obtained after division, Multi_Att represents the self-attention module, Norm represents the normalization module, MLP represents the fully connected module, and d represents x image The dimension of , softmax represents the normalization function;

[0114] (3.2) The Conformer model is used to obtain sound features. The input sequence X is processed through a convolutional layer, which helps capture local patterns and interactions between adjacent features. It is then fed into a multi-head self-attention layer, which allows the model to focus on different parts of the input sequence. The relevant formula is as follows:

[0115] h sound =Conformer(X sound )

[0116]

[0117] Conv=ReLU(W1X+b1)W2+b2

[0118]

[0119] Q sound =K sound =V sound =X sound

[0120] where h sound is the semantic-level sound feature output by the Conformer model, X sound To extract the frequency domain signal X(k) of the sound, Conv represents the convolutional layer module, ReLU represents the activation function, W1 and W2 are the weight matrices of the convolutional layer, and b1 and b2 are the bias vectors of the convolutional layer.

[0121] In some other preferred embodiments, in the underwater pin-to-hole detection method, step (4) includes the following sub-steps:

[0122] The step (4) includes the following sub-steps:

[0123] (4.1) The image features h obtained in steps (2) and (3) are image and sound characteristics h sound Mapped to the same shared latent space:

[0124] x i =g(h image θ g ),x j =g(h sound θ g )

[0125] where x i and x j denote the representation vectors of image and sound in the shared latent space, θ g is the parameter of the mapping function g;

[0126] (4.2) Using a training method based on contrast loss function, the correspondence between images and sounds is learned, expressed as:

[0127] L(x i ,x j ,y ij )=(1-y ij )max(0,mD(x i ,x j )) 2 +y ij D(x i ,x j ) 2

[0128] Where L(x i ,x j ,y ij ) represents the loss function between samples, y ij Is a binary variable indicating whether the two data come from the same set of data; the goal of the above model is to learn a joint representation so that similar image and sound data are close to each other in the shared latent space, while dissimilar data are pushed away; specifically, the loss function of the model is the average of the contrastive loss function of all training sample pairs:

[0129]

[0130] Where N is the total number of training sample pairs.

[0131] In some preferred embodiments, based on the above implementation, in the underwater pin-to-hole detection method, step (5) includes the following sub-steps:

[0132] (5.1) A multi-head attention mechanism is used for feature fusion. In the downstream task, a classifier is used to detect whether the pins and holes are aligned. The relevant formula is as follows:

[0133] x fusion =Multi_Att(x i ,x j )

[0134] Z=softmax(x fusion W cls +b cls )

[0135] where x fusion is x i and x j One-to-one correspondence with the fused features, Z is the inspection result, W cls and b cls are the weight and bias vectors of the classifier;

[0136] (5.2) Use the cross entropy loss function and the contrast loss function of step (4.2) as the loss function of the entire model to train the model, and fine-tune the ViT model and Conformer model of step (3):

[0137]

[0138] Where L is the loss function of the entire model, is the contrast loss function of step (4.2), and α and β are the loss function weights.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for detecting underwater pin-to-hole alignment, characterized in that: The steps include: (1) Obtaining a video of the pin-to-hole alignment process and performing image preprocessing; (2) Obtaining the vibration frequency data of the grab beam and extracting the sound information; (3) Using pre-trained image recognition models and sound recognition models to obtain semantic-level image features and sound features; (4) Using a cross-modal alignment model to align image and sound features; (5) Using the aligned features for feature fusion, the deep learning model is used to detect whether the pins and holes are aligned; The step (4) includes the following sub-steps: (4.1) The image features h obtained in steps (2) and (3) are image and sound characteristics h sound Mapped to the same shared latent space: x i =g(h image ;θ g ),x j =g(h sound ;θ g ) where x i and x j denote the representation vectors of image and sound in the shared latent space, θ g is the parameter of the mapping function g; (4.2) Using a training method based on contrast loss function, the correspondence between images and sounds is learned, expressed as: L(x i ,x j ,y ij )=(1-y ij )max(0,mD(x i ,x j )) 2 +y ij D(x i ,x j ) 2 Where L(x i ,x j ,y ij ) represents the loss function between samples, y ij Is a binary variable indicating whether the two data come from the same set of data; the goal of the above model is to learn a joint representation so that similar image and sound data are close to each other in the shared latent space, while dissimilar data are pushed away; specifically, the loss function of the model is the average of the contrastive loss function of all training sample pairs: Where N is the total number of training sample pairs.

2. The underwater pin-to-hole detection method according to claim 1, characterized in that: The step (1) includes the following sub-steps: (1.1) Video frame extraction: The video is split into several frames using the equal-interval sampling method. The relevant formula is: F t =V(t) Among them, F t represents the t-th frame image, V(t) represents the t-th frame video; (1.2) Image scaling and cropping: The extracted video frames are scaled and cropped to meet the size requirements of the model input and reduce the amount of calculation. The relevant formula is: F′ t =crop(scale(F t )) Among them, F′ t Represents an image after scaling and cropping, crop represents the cropping operation, and scale represents the scaling operation; (1.3) Image enhancement: Perform brightness enhancement, contrast enhancement, and color space transformation on an image to improve image quality and enhance image features. The relevant formula is: F″ t =enhance(F′ t ) where F t Represents the image after image enhancement, and enhance represents the image enhancement operation; (1.4) Data standardization: The enhanced image is standardized so that each pixel value in the image has the same data range and mean. The standardization operation can make the model more stable and reliable. The relevant formula is: F″′ t =normalize(F″ t ) where F″′ t Represents the standardized image, and normalize represents the data normalization operation; (1.5) Data enhancement: Data enhancement is performed on the image to increase the data samples and enhance the generalization ability of the model. The relevant formula is: in Represents the image data after data enhancement, and augment represents the data enhancement operation.

3. The underwater pin-to-hole detection method according to claim 1, characterized in that: The step (2) includes the following sub-steps: (2.1) Sampling and quantization of audio signals: converting continuous sound signals into digital signals; (2.2) Pre-emphasis: Perform high-pass filtering on the input signal in advance. The relevant formula is: s′(n)=s(n)-αs(n-1) Where s(n) represents the original sound signal, s′(n) represents the preprocessed signal, and α is the pre-emphasis coefficient; (2.3) Framing: The long-duration sound signal is divided into several short-duration frames. The Hamming window function is used to perform weighted smoothing on each frame. The window function is expressed as: f(n)=w(n)s′(n) Where f(n) represents the processed signal, w(n) is the Hamming window function; (2.4) Fast Fourier Transform (FFT): Perform a fast Fourier transform on each frame of signal to convert the time domain signal into the frequency domain signal; (2.5) Feature normalization: The extracted features are normalized so that they have zero mean and unit variance to facilitate subsequent modeling and training.

4. The underwater pin-to-hole detection method according to claim 1, characterized in that: The step (3) includes the following sub-steps: (3.1) Use the Vision Transformer (ViT) pre-trained model to obtain image features: divide the input image into fixed-size tiles, compress each tile into a vector as the input of the model, embed positional encoding on these vectors to help the model identify different parts of the image, and pass these vectors as input to the Transformer encoder, which contains multiple Transformer blocks, each of which consists of multiple self-attention layers and fully connected layers. composition: h image =ViT(x image ) Q image =K image =V image =x image where h image is the semantic-level image feature output by the ViT model, x image After image preprocessing The image information obtained after division, Multi_Att represents the self-attention module, Norm represents the normalization module, MLP represents the fully connected module, and d represents x image The dimension of , softmax represents the normalization function; (3.2) Using the Conformer model to obtain sound features; The input sequence X is processed through a convolutional layer, which helps capture local patterns and interactions between adjacent features, and then fed into a multi-head self-attention layer, which allows the model to focus on different parts of the input sequence. The relevant formula is as follows: h sound =Conformer(X sound ) Conv=ReLU(W1X+b1)W2+b2 Q sound =K sound =V sound =X sound where h sound is the semantic-level sound feature output by the Conformer model, X sound To extract the frequency domain signal X(k) of the sound, Conv represents the convolutional layer module, ReLU represents the activation function, W1 and W2 are the weight matrices of the convolutional layer, and b1 and b2 are the bias vectors of the convolutional layer.

5. The underwater pin-to-hole detection method according to claim 1, characterized in that: The step (5) includes the following sub-steps: (5.1) A multi-head attention mechanism is used for feature fusion. In the downstream task, a classifier is used to detect whether the pins and holes are aligned. The relevant formula is as follows: x fusion =Multi_Att(x i ,x j ) z =softmax(x fusion W cls +b cls ) where x fusion is x i and x j One-to-one correspondence with the fused features, Z is the inspection result, W cls and b cls are the weight and bias vectors of the classifier; (5.2) Use the cross entropy loss function and the contrast loss function of step (4.2) as the loss function of the entire model to train the model, and fine-tune the ViT model and Conformer model of step (3): Where L is the loss function of the entire model, is the contrast loss function of step (4.2), and α and β are the loss function weights.

Images