Underwater acousto-optic pairing data set preparation method, electronic equipment and medium

Abstract

The invention discloses an underwater acousto-optic pairing data set preparation method, electronic equipment and a medium. The underwater acousto-optic pairing data set preparation method comprises the following steps: configuring an underwater vehicle ROV for shooting optical images, a shot model and an underwater environment; respectively adjusting the angles, postures and distances of the ROV and the shot model so as to collect the video of the shot model; intercepting a video to obtain an original optical image set; preprocessing the original optical image set to obtain an optical data set; obtaining a sonar data set; taking the optical data set as a content image, and taking the sonar data set as a style image; inputting the content image and the first style image into a style migration network to obtain a style migration intermediate image; inputting the style migration intermediate image and the second style image into a style migration network to obtain a final style migration image, namely sonar data matched with the optical data; and marking all the optical data and the sonar data paired with the optical data to obtain an acousto-optic pairing data set.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision technology, and in particular relates to a method for preparing an underwater sound and light pairing data set, an electronic device, and a medium. Background Art

[0002] With the leap in computer hardware performance and the continuous innovation of algorithm theory, computer vision technology has undergone a transformation from its infancy to vigorous development. It has now been widely used in autonomous driving, medical image analysis, virtual reality, industrial automation, security systems and other aspects, covering multiple task types such as classification, recognition, and tracking, profoundly changing human production and lifestyle.

[0003] However, in order to train an accurate and fast network model, in addition to efficient algorithms, high-quality training data is also indispensable, especially in specific mission scenarios, where data set resources are scarce and quality is difficult to guarantee. In the detection scenario of underwater targets, the data set used will have the following problems:

[0004] 1. High-quality datasets are scarce.

[0005] 2. The optical image quality in underwater environment is poor. Only slightly clear images can be captured at a distance close to the camera. Once the distance is too far, the image will be blurred.

[0006] 3. There is no paired dataset of multiple signal sources. The attenuation rate of acoustic signals in underwater environments is much lower than that of electromagnetic waves. However, sonar equipment is expensive and the working environment is demanding. The difficulty of producing sonar datasets is much greater than that of optical datasets. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present application provides a method, electronic device, and medium for preparing an underwater sound and light pairing data set.

[0008] In a first aspect, an embodiment of the present invention provides a method for preparing an underwater sound-light pairing dataset, the method comprising:

[0009] An underwater vehicle ROV, a photographed model and an underwater environment configured to take optical images;

[0010] Adjusting the angle, attitude and distance between the underwater vehicle ROV and the photographed model to collect video of the photographed model;

[0011] The video is intercepted at unit frame intervals to obtain an original optical image set; the original optical image set is preprocessed to obtain an optical data set;

[0012] Obtain a sonar data set; use the optical data set as a content image and the sonar data set as a style image; input the content image and the first style image into a style transfer network to obtain a style transfer intermediate image; input the style transfer intermediate image and the second style image into the style transfer network to obtain a final style transfer image, i.e., sonar data matching the optical data; traverse all optical data to obtain sonar data paired with them;

[0013] All optical data and their paired sonar data are annotated to obtain an optical-acoustic paired dataset.

[0014] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned method for preparing an underwater sound and light pairing data set.

[0015] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for preparing an underwater sound-light pairing dataset.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned method for preparing an underwater sound and light pairing dataset when executed by a processor.

[0017] The beneficial effects of the present invention are as follows: the present invention provides a method for preparing an underwater sound-light pairing dataset, comprising acquiring an optical dataset and performing style transfer processing to obtain sonar data matching each optical data, thereby obtaining a high-quality sound-light pairing dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 is a flow chart showing preparation of an underwater sound-light pairing data set according to an exemplary embodiment;

[0020] Figure 2 is a working principle diagram of a style transfer network according to an exemplary embodiment;

[0021] Figure 3 is a flow chart of sonar data set preparation according to an exemplary embodiment;

[0022] Figure 4 is a display diagram of various types of sound-light pairing data according to an exemplary embodiment;

[0023] Figure 5 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0026] like Figure 1 As shown, an embodiment of the present invention provides a method for preparing an underwater sound-light pairing dataset, the method comprising the following steps:

[0027] Step S1, configuring an underwater vehicle ROV for taking optical images, a model to be photographed, and an underwater environment.

[0028] Specifically, first, prepare the underwater vehicle ROV for taking optical images. The underwater vehicle ROV, zero buoyancy cable, repeater, laptop computer and remote control handle are coupled in sequence to ensure that the laptop screen can correctly display the image of the underwater vehicle ROV camera and that the remote control handle can control the underwater vehicle ROV. Then, handle control calibration, underwater vehicle ROV depth sensor calibration, underwater vehicle ROV attitude sensor calibration, underwater vehicle ROV air tightness verification and remaining power check are performed to ensure that the underwater vehicle ROV can be accurately controlled and safely operated in an underwater environment.

[0029] Secondly, prepare underwater models. In this example, in order to ensure the richness of target species and obtain a more generalized data set, prepare underwater models of multiple categories and multiple individuals in each category, including five categories: fish, coral rocks, sunken ships, frogmen, and underwater robots. Each category has more than five individuals with different colors, shapes, and postures, and finally there are 25 different model individuals.

[0030] Finally, prepare the underwater shooting environment. In order to reduce the interference of external factors, water is added to the indoor landscape pool to reach the minimum diving depth of the underwater vehicle ROV, completing the preparation of the underwater environment.

[0031] Step S2, respectively adjusting the angle, posture and distance of the underwater vehicle ROV and the photographed model to collect the video of the photographed model.

[0032] Specifically, a handle is used to control the movement of the ROV in six directions (front and back, left and right, depth and shallowness) and four degrees of freedom (clockwise rotation in the horizontal plane, counterclockwise rotation in the horizontal plane, clockwise and counterclockwise rotation in the vertical plane, and counterclockwise rotation in the vertical plane). Two transparent thin fishing lines are used to tie the head and tail of the model to be filmed to control its movement in six directions and four degrees of freedom, and capture motion videos of the model at different distances, postures, and angles.

[0033] Step S3, intercepting the video at unit frame intervals to obtain an original optical image set; screening, integrating, shuffling, renaming and sorting the original optical image set to obtain an optical data set.

[0034] Specifically, the obtained series of model motion videos are intercepted at intervals of 5 frames to ensure that the images of the model at various distances, postures, and viewing angles can be retained. After processing all the motion videos, the original optical image set is obtained, and then the original optical image set is subjected to a series of processing.

[0035] First, a filtering operation is performed to remove blurred images caused by the model moving too fast, images that are difficult to recognize due to dim light, and incomplete images that move out of the camera range.

[0036] All the images are then integrated and shuffled. Because the model motion video is shot in units of models, the final captured images are also clustered in units of models. To ensure that the distribution of each model image is close to that in the real scene, and to facilitate the division of training and test sets for downstream detection and recognition tasks, all the images are integrated together and randomly shuffled.

[0037] Finally, the shuffled data sets are renamed in ascending order and divided into training sets and test sets in a ratio of 7:3 to obtain the final optical data set.

[0038] Step S4, obtaining a sonar data set; using the optical data set as a content image and the sonar data set as a style image; inputting the content image and the first style image into a style transfer network to obtain a style transfer intermediate image; inputting the style transfer intermediate image and the second style image into a style transfer network to obtain a final style transfer image, i.e., sonar data matching the optical data; traversing all optical data to obtain sonar data paired therewith.

[0039] Among them, Figure 2 and Figure 3As shown in the figure, the style transfer network includes: feature encoding network, self-attention module, multi-scale style fusion module, multi-stroke style transfer module, and feature decoding network.

[0040] The feature encoding network is used to extract features of the content image and the style image to obtain a content feature map and a style feature map respectively.

[0041] The self-attention module is used to extract the self-attention map corresponding to the content feature map, and the expression is as follows:

[0042]

[0043] In the formula, represents the convolution operation, α ij represents the weight coefficient between region i and region j, f c Represents the feature map of the content image, Θ h represents the convolution kernel, and the flat(·) function is used to flatten the multi-dimensional feature map into a one-dimensional vector.

[0044] The multi-scale style fusion module is used to fuse the content feature map and the style feature map at different scales to obtain a content feature map that fuses multiple stroke features. The expression is as follows:

[0045]

[0046] In the formula, and They are content feature maps f c and style feature map f s The structural feature image is obtained after whitening transformation, Γ is the multi-scale mapping function; β k In order to control the control coefficient of the brush stroke size, the style structure feature map is mapped into feature images of multiple brush stroke sizes through multi-scale mapping; F ss is a style transfer function, used to transfer the obtained multi-stroke features to the content structure feature map; It is a content feature map that integrates multiple stroke features.

[0047] The multi-stroke style transfer module is used to guide the content feature map that integrates the multi-stroke features through the self-attention map to obtain a stylized feature map, which is expressed as follows:

[0048]

[0049] In the formula, is the weight of the k-th stroke feature obtained after filtering the sub-attention map, is the final stylized feature map.

[0050] The feature decoding network is used to decode the stylized feature map to obtain a style transfer image. Figure 4 Shown are diagrams showing the sound-light pairing data for each category.

[0051] Furthermore, the sonar dataset adopts the public sonar datasets SCTD (Sonar Common Target Detection Datasets) and SeabedObjects, and selects some images with typical side-scan sonar features as style images.

[0052] Step S5: annotate all the optical data and the sonar data paired therewith to obtain an acoustic-optical paired data set.

[0053] Specifically, all images are organized into four files, namely, optical training set, optical test set, acoustic training set and acoustic test set. The four files are labeled separately using the dataset annotation software labelme. Each image corresponds to a annotation file. Finally, all the annotation files are batch converted to obtain a annotation file in coco format.

[0054] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned underwater sound and light pairing data set preparation method. Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability in which the underwater sound and light pairing data set preparation method provided by the embodiment of the present invention is located. Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0055] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the underwater sound and light pairing data set preparation method as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0056] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0057] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for preparing an underwater sound and light pairing dataset, characterized in that: The method comprises: An underwater vehicle ROV, a photographed model and an underwater environment configured to take optical images; Adjusting the angle, attitude and distance between the underwater vehicle ROV and the photographed model to collect video of the photographed model; The video is intercepted at unit frame intervals to obtain an original optical image set; the original optical image set is preprocessed to obtain an optical data set; Obtain a sonar data set; use the optical data set as a content image and the sonar data set as a style image; input the content image and the first style image into a style transfer network to obtain a style transfer intermediate image; input the style transfer intermediate image and the second style image into the style transfer network to obtain a final style transfer image, i.e., sonar data matching the optical data; traverse all optical data to obtain sonar data paired with them; All optical data and their paired sonar data are annotated to obtain an optical-acoustic paired dataset.

2. The method for preparing an underwater sound and light paired data set according to claim 1, characterized in that: The depth of the underwater environment is greater than the minimum diving depth of the underwater vehicle ROV.

3. The method for preparing an underwater sound and light paired data set according to claim 1, characterized in that: The photographed models include several categories, and each category includes several individuals of different colors, shapes, and postures.

4. The method for preparing an underwater sound and light paired data set according to claim 3, characterized in that: The categories include fish, coral reefs, shipwrecks, frogmen and underwater robots.

5. The method for preparing an underwater sound and light paired data set according to claim 1, characterized in that: The process of adjusting the angle, attitude and distance between the underwater vehicle ROV and the photographed model to collect the video of the photographed model includes: Control the movement of the underwater vehicle ROV in six directions (front and back, left and right, depth) and four degrees of freedom (clockwise rotation in the horizontal plane, counterclockwise rotation in the horizontal plane, clockwise and counterclockwise rotation in the vertical plane, counterclockwise rotation in the vertical plane), and collect motion videos of the model being photographed at different distances, postures, and angles.

6. The method for preparing an underwater sound and light paired data set according to claim 1, characterized in that: The style transfer network includes: feature encoding network, self-attention module, multi-scale style fusion module, multi-stroke style transfer module, and feature decoding network; The feature encoding network is used to extract features of the content image and the style image to obtain a content feature map and a style feature map respectively; The self-attention module is used to extract the self-attention map corresponding to the content feature map, and the expression is as follows: In the formula, represents the convolution operation, α ij represents the weight coefficient between region i and region j, f c Represents the feature map of the content image, Θ h represents the convolution kernel, and the flat(·) function is used to flatten the multidimensional feature map into a one-dimensional vector; The multi-scale style fusion module is used to fuse the content feature map and the style feature map at different scales to obtain a content feature map that fuses multiple stroke features. The expression is as follows: In the formula, and They are content feature maps f c and style feature map f s The structural feature image is obtained after whitening transformation, Γ is the multi-scale mapping function; β k In order to control the control coefficient of the brush stroke size, the style structure feature map is mapped into feature images of multiple brush stroke sizes through multi-scale mapping; F ss is a style transfer function, used to transfer the obtained multi-stroke features to the content structure feature map; It is a content feature map that fuses multiple stroke features; The multi-stroke style transfer module is used to guide the content feature map that integrates the multi-stroke features through the self-attention map to obtain a stylized feature map, which is expressed as follows: In the formula, is the weight of the k-th stroke feature obtained after filtering the sub-attention map, is the final stylized feature map; The feature decoding network is used to decode the stylized feature map to obtain a style transfer image.

7. The method for preparing an underwater sound and light paired data set according to claim 1, characterized in that: The process of labeling all optical data and paired sonar data to obtain the acoustic-optical paired data set includes: Divide all optical data and paired sonar data into an optical training set, an optical test set, an acoustic training set and an acoustic test set; Use the dataset annotation software labelme to annotate the optical training set, optical test set, acoustic training set, and acoustic test set, respectively. Each image corresponds to a annotation file. Batch convert all annotation files to get a annotation file in coco format.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the underwater sound and light pairing data set preparation method described in any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for preparing an underwater sound-light pairing dataset according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for preparing an underwater sound-light pairing dataset according to any one of claims 1 to 7 is implemented.

Images