A forward-looking sonar image underwater target online identification method and device

Abstract

This invention discloses an online method and device for underwater target recognition using forward-looking sonar images. The method involves inputting a collected forward-looking sonar image dataset into a pre-trained YOLOv3 deep learning detection model for fine-tuning. The model file, weight file, and configuration file are then converted using an ATC tool to obtain an offline model, which is then deployed to an embedded system. The data stream acquired online by the forward-looking sonar device is converted to generate sonar images. The embedded system then uses the detection model to detect targets in the images, and the detection results are packaged and transmitted to the display and control software. This makes the results more intuitive and reduces the difficulty of target recognition in sonar images. It assists operators in judging targets on the display and control system, avoiding missed and false detections, reducing operator error rates, and solving the problem of limited scalability due to the difficulty of manual identification.

Description

Technical Field

[0001] This invention belongs to the fields of underwater vision technology and artificial intelligence, and specifically relates to a method and device for online recognition of underwater targets in forward-looking sonar images. Background Technology

[0002] With the increasing demand for underwater exploration, underwater target identification has become a hot topic in recent years. Underwater target identification is a core technology in unmanned underwater exploration. Compared to image acquisition methods on land and in the air, underwater image acquisition is more difficult and suffers from drawbacks such as high cost and poor quality. The quantity and quality of sonar data available for analysis and research are often insufficient. Therefore, improving the accuracy and speed of underwater target identification and reducing the complexity of the identification algorithm are crucial for current underwater target identification efforts.

[0003] Underwater target recognition generally includes feature extraction and target classification. After years of development, various underwater target recognition algorithms based on sonar images have been proposed. Traditional target recognition algorithms typically compare underwater targets with pixel grayscale values ​​in sonar images by setting thresholds. However, these algorithms fail to fully consider structural features, leading to problems such as poor robustness and low recognition rates. Traditional signal processing methods are convenient for feature selection, but expert supervision is required to ensure the effectiveness of feature selection. The development of sonar systems and technologies has provided richer information features for underwater target detection and recognition, especially for targets with small geometric shapes such as mines. However, traditional signal processing and machine learning algorithms have a high probability of missing detections for these insignificant targets, while deep learning algorithms can achieve higher recognition rates through deep neural networks. Therefore, researchers are attempting to replace manual feature extraction or traditional signal processing methods with intelligent systems such as neural networks and deep learning algorithms to track, detect, and classify underwater acoustic signals. However, in the practical application of neural network algorithms, there is currently a lack of devices that use embedded AI chips to perform online recognition of sonar images, especially those based on domestically produced embedded AI chips. Against the backdrop of building a maritime strategic power, and with the continuous development of deep learning algorithms in underwater target recognition, it is necessary to design an online underwater target recognition method and equipment based on domestically produced AI chips. Summary of the Invention

[0004] This invention addresses the problem of underwater target identification heavily relying on manual intervention in the prior art. It provides an online method and device for underwater target identification using forward-looking sonar images. The method involves inputting a collected forward-looking sonar image dataset into a pre-trained YOLOv3 deep learning detection model for fine-tuning. The model file, weight file, and configuration file are then converted using an ATC tool to obtain an offline model, which is deployed to an embedded system. The data stream acquired online by the forward-looking sonar device is converted to generate sonar images. The embedded system then uses the detection model to detect targets in the images and packages the detection results to send back to the display and control software. This makes the results more intuitive and reduces the difficulty of target identification in sonar images. It assists operators in judging targets on the display and control screen, avoiding missed and false detections, reducing operator error rates, and solving the problem of limited scalability due to the difficulty of manual identification.

[0005] To solve the technical problem, the technical solution of the present invention is as follows:

[0006] A method for online identification of underwater targets in forward-looking sonar images, the method comprising:

[0007] The host computer controls the start of the sonar identification device and determines the start mode of the sonar identification device; the sonar identification device includes two working modes: online operation and playback operation; both use a network to communicate with the host computer.

[0008] When the device is online, it receives beam data from the host computer, decodes the data and stores it in image format; performs preprocessing on the image data; performs target recognition processing on the preprocessed image data; transmits the processing result back to the host computer and stores the result; determines whether the host computer controls the device to stop running; if not, jumps to the step of receiving beam data from the host computer; if so, ends.

[0009] When the device is playing back, it receives playback data, decodes the playback data, and stores it in image format; it performs preprocessing operations on the image data; it performs target recognition processing on the preprocessed image data; it stores the processing result in video format; it determines whether the host computer has controlled the device to stop running or whether the data has been read completely; if not, it continuously acquires image data; if yes, it terminates.

[0010] Furthermore, in both online and playback modes, the preprocessing of the aforementioned image data specifically includes: considering the high noise level in the received raw beam data, first using a denoising algorithm to denoise the data, and then transforming the image to a size suitable for the convolutional neural network through scaling.

[0011] Furthermore, in both online and playback modes, before performing target recognition processing on the preprocessed image data, an offline model is obtained. First, the image data is processed by flipping, rotating, scaling, cropping, and translating to augment the dataset. The augmented dataset is then labeled using annotation software. The sonar image data is trained using the Darknet53 framework to obtain a weight file. The model file is then modified, and the AscendTensor Compiler tool is used to convert the model file and weight file to obtain the offline model. The offline model is then implanted onto the SD3403 development board.

[0012] Furthermore, the dataset annotation specifically involves: acquiring underwater target images and scaling all images to a size suitable for convolutional neural networks; and using annotation software to annotate the images, with the annotation information including target category labels.

[0013] Furthermore, in the online working mode, the device directly receives the beam data collected by the sonar online, decodes each ping beam data collected by the sonar to obtain image data, stores the data in image format on the solid-state drive, sends the obtained recognition results back to the host computer, and stores each ping result in image format on the solid-state drive.

[0014] Furthermore, in the playback mode, the device receives a .dat format data file, decodes the data and reads out the first frame of sonar data, and stores the sonar data in image format on the solid-state drive; it also stores the processing result of each ping in video format on the solid-state drive.

[0015] A forward-looking sonar image underwater target online identification device, the device is connected to a host computer via a network cable, the device is used to execute the forward-looking sonar image underwater target online identification method described above, the device includes: SD3403 development board, power interface, power switch, system power-on indicator, HDMI interface, serial port, network port, SD card slot, USB interface and cooling fan;

[0016] The SD3403 development board incorporates a pre-trained YOLO-v3 deep learning detection model. The power interface and power switch provide stable power to the device. The indicator light shows that the device system has started running. The HDMI interface is used for subsequent display. The network port is used for data transmission, receiving control commands, and debugging. The SD card slot is set to start and expand the device's storage capacity. The USB interface and serial port are used for serial port debugging.

[0017] Compared with the prior art, the advantages of the present invention are as follows:

[0018] This invention presents an online underwater target identification method and device based on a domestically produced AI chip. Compared to traditional sonar image recognition methods, this invention uses a convolutional neural network to identify targets in sonar images, achieving a degree of unmanned detection and assisting sonar operators in target identification. Returning the target category and location results to a host computer makes the results more intuitive and reduces the difficulty of target identification in sonar images. It also reduces operator error rates and solves the problem of limited scalability due to the difficulty of manual identification. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the device of the present invention;

[0020] Figure 2 This is a flowchart of the workflow of the present invention;

[0021] Figure 3 Flowchart for offline model conversion;

[0022] Figure 4 This is a diagram illustrating the connection with the host computer.

[0023] Figure 5 This is a schematic diagram showing the results. Detailed Implementation

[0024] The specific implementation of the present invention is described below with reference to embodiments:

[0025] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0026] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0027] Example 1

[0028] In the field of active sonar image target recognition, an increasing number of convolutional neural network algorithms are being applied to sonar images. These algorithms are trained and verified using sonar images, achieving good results. However, research on online sonar image recognition is still insufficient. Furthermore, due to the relatively short development history of domestic embedded AI, development is quite difficult. Therefore, devices using domestically produced AI chips to support network models for online sonar image recognition are extremely rare. Online sonar image recognition involves collecting data online via sonar, transmitting the data to an embedded AI chip for processing, and then packaging the results for display. Figure 1-5 As shown, this invention will implement the embedded AI chip, complete the interaction with the host computer, and the sonar image recognition and processing.

[0029] The schematic diagram of the device interface of this invention is as follows: Figure 1 As shown, there is one power supply and power switch to provide stable power to the device. There is one indicator light to show that the system is running. There is one HDMI port for subsequent display. There is one Ethernet port for data transmission, receiving control commands, and debugging. There is one SD card slot, which allows for SD card booting and expansion of device storage capacity. There is one USB port. There is one serial port for serial debugging.

[0030] This invention has two working modes: online working mode and playback working mode.

[0031] Furthermore, the specific process of the online work mode is as follows:

[0032] Step 1: Receive beam data from the host computer, decode the data and store it in image format on a solid-state drive.

[0033] Step 2: Perform preprocessing operations on the image data.

[0034] Step 3: Perform target recognition processing on the preprocessed image data.

[0035] Step 4: Send the processing results back to the host computer and save the results as an image in a solid-state drive.

[0036] The device playback workflow is as follows:

[0037] Step 1: Receive playback data, decode the playback data, and store it in image format.

[0038] Step 2: Perform preprocessing operations on the image data.

[0039] Step 3: Perform target recognition processing on the preprocessed image data.

[0040] Step 4: Save the processing result as a video.

[0041] Step 5: Loop through and retrieve the data for the next frame.

[0042] Combination Figure 2 The online working mode and the playback working mode follow the same steps two and three.

[0043] In the online working mode, step one involves directly receiving the beam data acquired by the sonar online. Each ping beam data acquired by the sonar is decoded to obtain image data, which is then stored in image format on a solid-state drive.

[0044] In the online working mode, step four involves sending the obtained location identification results back to the host computer. Each ping result is also stored as an image on a solid-state drive.

[0045] In the playback mode step one, the .dat format data file is received, the data is decoded and the first frame of sonar data is read out, and the sonar data is stored in the solid-state drive in image format.

[0046] In step four of the playback mode, the processing result of each ping is stored in video format and then stored as a video on a solid-state drive.

[0047] In step three, an offline model needs to be obtained before object detection. First, prepare the sonar image dataset. The dataset is enriched by processing it through flipping, rotating, scaling, cropping, and translating. Annotation software is used to annotate the augmented dataset. Second, the Darknet53 framework is used to train the dataset, obtaining the trained weight file. The ATC tool is obtained, environment variables are set, and the AIPP file is configured. The model file (*.prototxt) is modified, and the ATC (Ascend Tensor Compiler) tool is used to convert the model file (*.prototxt), weight file (*.caffemodel), and other files into an offline model, which is then deployed to the development board.

[0048] The above programs were all compiled using a cross-compilation environment. Therefore, a cross-compilation environment for this embedded chip needs to be installed on Linux for program development and compilation. Since the OpenCV library is used, an OpenCV environment needs to be installed and the OpenCV dependency configured in the embedded environment. The Ascend-cann-toolkit development kit, which includes development aids and related development interfaces, also needs to be configured.

[0049] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

[0050] Many other changes and modifications can be made without departing from the concept and scope of this invention. It should be understood that this invention is not limited to the specific embodiments, and the scope of this invention is defined by the appended claims.

Claims

1. A method for online identification of underwater targets from forward-looking sonar images, characterized in that, The method, based on a pre-trained YOLOv3 deep learning detection model, includes: S1: The host computer controls the sonar identification device to start and determines the device's working mode as either online or playback mode based on the control command; the sonar identification device communicates with the host computer via a network. S2: In online working mode: receive sonar beam data from the host computer in real time, decode the beam data and store it in image format; perform denoising and scale transformation image preprocessing on the decoded image data; input the preprocessed image into an offline target detection model deployed on an embedded AI chip for real-time target recognition; send the recognition result back to the host computer and store the recognition image of each ping in a solid-state drive; repeat the above steps until the host computer issues a termination command; S3: In playback mode: Receive playback data files in .dat format, decode the playback data and store it in image format; perform denoising and scale transformation preprocessing on the decoded image; input the preprocessed image into the offline model for target recognition; store the target recognition result of each ping in video format, and process the next frame in a loop until the data is read out or the host computer issues an end command; S4: The offline target detection model is a model trained by the Darknet53 framework after performing flipping, rotation, scaling, cropping and translation enhancement processing on the sonar image dataset. The model is converted to om format by the Ascend Tensor Compiler tool and deployed to the SD3403 development board to realize real-time inference of sonar images.

2. The forward-looking sonar image underwater target online identification device according to claim 1, characterized in that, In both online and playback modes, the preprocessing of the aforementioned image data includes: first, using a denoising algorithm to denoise the data, and then transforming the image to a size suitable for a convolutional neural network through scaling.

3. The forward-looking sonar image underwater target online identification device according to claim 1, characterized in that, In both online and playback modes, before performing target recognition processing on the preprocessed image data, an offline model is obtained. First, the image data is processed by flipping, rotating, scaling, cropping, and translating to augment the dataset. The augmented dataset is then labeled using annotation software. The sonar image data is trained using the Darknet53 framework to obtain a weight file. The model file is then modified, and the Ascend Tensor Compiler tool is used to convert the model file and weight file to obtain the offline model. The offline model is then implanted onto the SD3403 development board.

4. The forward-looking sonar image underwater target online identification device according to claim 3, characterized in that, The specific process of dataset annotation is as follows: acquiring underwater target images and scaling all images to a size suitable for convolutional neural networks; using annotation software to annotate the images, with annotation information including target category labels.

5. The forward-looking sonar image underwater target online identification device according to claim 1, characterized in that, In online operating mode, the device directly receives beam data from the sonar online acquisition site, decodes each ping beam data from the sonar acquisition site to obtain image data, stores the data in image format on the solid-state drive, and sends the obtained recognition results back to the host computer. Each ping result is also stored in image format on the solid-state drive.

6. The forward-looking sonar image underwater target online identification device according to claim 1, characterized in that, In the playback mode, the device receives .dat format data files, decodes the data and reads out the first frame of sonar data, and stores the sonar data in image format on the solid-state drive. The processing result of each ping is stored in video format and then stored as a video on a solid-state drive.

7. An online underwater target identification device based on forward-looking sonar images, wherein the device is connected to a host computer via a network cable and to the sonar via a signal connection, characterized in that... The device is used to perform the online underwater target identification method for forward-looking sonar images as described in claims 1-6. The device includes: an SD3403 development board, a power interface, a power switch, a system power-on indicator, an HDMI interface, a serial port, a network port, an SD card slot, a USB interface, and a cooling fan. The SD3403 development board incorporates a pre-trained YOLO-v3 deep learning detection model. The power interface and power switch provide stable power to the device. The indicator light shows that the device system has started running. The HDMI interface is used for subsequent display. The network port is used for data transmission, receiving control commands, and debugging. The SD card slot is set to start and expand the device's storage capacity. The USB interface and serial port are used for serial port debugging.

Images