A forward-looking sonar video target detection method and device based on the domestic Ascend AI processor

Abstract

A sonar video target detection method and device based on a domestic Ascend AI processor belongs to the field of marine acoustic detection technology. The method includes the following steps: S1, building an edge intelligent hardware platform based on the Atlas200I A2 acceleration module; S2 designing a forward-looking sonar video target detection network, introducing a motion excitation module to enhance inter-frame motion features, and designing a self-attention feature aggregation module to aggregate temporal context features; S3, using a two-stage training method to train the sonar video target detection network, saving the trained network weights and structures as ONNX reasoning framework graph files, and using an asymmetric quantization method to quantize the model weight process; S4, using the Ascend CANN heterogeneous reasoning framework to deploy the video target detection network. The present invention overcomes the challenge of poor detection of motion blur and target concealment in forward-looking sonar videos by single-frame detectors, improves detection accuracy, and promotes the application and development of domestic AI processors in forward-looking sonar detection technology.

Description

Technical Field

[0001] The present invention relates to the field of marine acoustic detection technology, and in particular to a forward-looking sonar video target detection method and device based on a domestically produced Ascend AI processor. Background Art

[0002] With the development of marine acoustic application technology, forward-looking sonar imaging technology has the advantages of long detection distance, strong penetration ability, and applicability to turbid waters compared with other underwater detection technologies such as light, electricity, and magnetism. It has been widely used in underwater rescue, underwater archaeology, energy exploration, route planning, terrain exploration, anti-frogman, submarine cable laying and maintenance, etc. However, in high-performance sonar imaging systems, independent innovation of core technologies such as CPU processors is the key to ensuring national marine security and sustainable development. The Ascend Atlas200I A2 acceleration module is a high-performance domestically produced AI intelligent computing module that integrates the HiSilicon Ascend310B AI processor, including a 4-core Taishan CPU and Da Vinci AI Core, which can provide up to 20Tops (INT8) of AI computing power. Its performance and compatibility provide strong support for the realization of edge AI target detection of forward-looking sonar videos on domestic platforms.

[0003] At present, in the field of underwater target detection, the detection of underwater targets in sonar images is still mainly based on manual mode judgment, which mainly relies on the operator's long-term accumulated experience to identify the targets on the sonar image. In the operation scenes with harsh environment, long time span and wide detection area, it is very likely that the operator's continuous operation level is limited, lack of experience and visual fatigue will lead to the omission of targets, misjudgment and suspension of operations, which will affect the progress and efficiency of the operation. A large number of research works have confirmed that deep learning is very effective in natural image target detection tasks, and many natural image processing methods have been used to solve the sonar target detection problem.

[0004] However, unlike natural images, forward-looking sonar images have a lot of background noise interference, motion blur, and target concealment, making it difficult for general single-frame detectors to achieve good results in forward-looking sonar image target detection tasks. Continuous multi-frame image sequences contain rich motion and spatial features. Therefore, how to mine the spatiotemporal information of multi-frame sonar images and improve detection accuracy has become a very important task in forward-looking sonar image target detection. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a sonar video target detection method and device based on the domestically produced Ascend AI processor.

[0006] The present invention provides a forward-looking sonar video target detection method based on the domestically produced Ascend AI processor, thereby solving the problems of target motion blur and target occlusion in the forward-looking sonar image in the prior art, improving the accuracy of forward-looking sonar image target detection, and promoting the application and development of the domestically produced AI processor in the forward-looking sonar detection technology.

[0007] The first aspect of the present invention provides a forward-looking sonar video target detection method based on a domestic Ascend AI processor, and the specific process is as follows:

[0008] S1. Build an edge intelligent hardware system based on the domestic Ascend AI processor, which consists of a main control chip, a debugging interface, an HDMI interface, a Gigabit Ethernet interface, a USB3.0 interface, an SDIO interface, and a WIFI module;

[0009] S2. Design of a deep neural network for forward-looking sonar video target detection, where the input to the model is a continuous sequence of images , where n is the sequence length. The backbone network in the model is ME-CSPDarkNet53 with motion excitation. The features extracted by the backbone network are fed to the inspection head after feature fusion by the PAFPN module to obtain the basic features of prediction. The feature screening module is further used to screen the features F. The screened features are linearly mapped and input into the self-attention feature aggregation module to aggregate the classification and regression features at different scales for final classification.

[0010] S3. Use the two-stage training method to train the sonar video target detection network, save the trained network weights and structure as an ONNX framework model file, and use the asymmetric quantization method to quantize the weights of the ONNX model;

[0011] S4. Use the Ascend CANN heterogeneous inference framework to deploy the sonar video target detection network designed in step S2 to the edge intelligent hardware system built in step S1.

[0012] Preferably, step S2 comprises:

[0013] S21. Design the ME-CSPDarkNet53 backbone network structure of the forward-looking sonar video target detection network;

[0014] S22. Design a feature filtering module for forward-looking sonar video target detection network;

[0015] S23. Design of a self-attention feature aggregation module for forward-looking sonar video object detection network.

[0016] Step S21 specifically includes: introducing a motion excitation module into the backbone network to excite the motion sensitive information between frames to enhance the motion information. This method does not require training a high-cost optical flow network. , where N represents the batch size, T represents the time step size, whose value is equal to n, C represents the feature channel, H represents the feature height, and W represents the feature width. First, 1*1 convolution is used to reduce the number of channels to improve efficiency. The formula is as follows:

[0017] (1)

[0018] in represents the feature after channel reduction, * represents the convolution operation, and r is the scaling factor.

[0019] The motion features at time t can be approximately represented as two adjacent frames. and The difference between , so the motion characteristics at time t can be expressed as:

[0020] (2)

[0021] in , the motion feature at the end of the time step is recorded as 0, that is , and record all motion features as , and then use global average pooling to summarize the spatial information:

[0022] (3)

[0023] A 1*1 convolutional layer is used to expand the number of channels of the motion feature to the original channel dimension C, and the Sigmoid(·) function is used to obtain the motion attention weight A as follows:

[0024] (4)

[0025] Finally, residual connection is used to enhance motion information while retaining scene information. The specific formula is as follows:

[0026] (5)

[0027] in is the output of the motion excitation module, Denotes Hadamard Product.

[0028] Step S22 specifically includes: selecting high-quality features from the feature map output by the prediction head, using the TopK algorithm, sorting from high to low according to the confidence score, and selecting the top k prediction features. Using the Non-Maximum Suppression (NMS) algorithm, by continuously retrieving the detection box with the highest classification confidence, using the intersection-over-union ratio to represent the connection between two bounding boxes, a fixed number of predictions are selected to reduce redundancy. The classification and regression features of the filtered image are defined as:

[0029]

[0030]

[0031] Where n is the input image sequence number, is the feature dimension of the region of interest (RoI).

[0032] Step S23 specifically includes: the recognition process usually observes information of a certain time series, combines spatial information and time motion information, obtains recognition results, and propagates confident results to cases with insufficient confidence. As a key part of Transformer, the multi-head self-attention mechanism is suitable for this purpose, which enhances the ability of long-distance modeling.

[0033] The filtered classification and regression features with high confidence are used as queries, and the definition is:

[0034]

[0035]

[0036] Where LP(·) is the linear projection operator. The dimension is projected into d dimensions.

[0037] Calculate the self-attention weights of the category and regression features, the formula is:

[0038] (6)

[0039] in The dimension of each feature in Q and K is divided by Can make It conforms to the N(0,1) distribution.

[0040] Since the main purpose is to improve the classification results, and will share the same matrix . Further Combined with the result of formula (6), the formula is as follows:

[0041] (7)

[0042] in Represents the concatenation operation, and then The final classification is performed after entering the average pooling layer.

[0043] Preferably, step S3 specifically includes:

[0044] S31. Using a two-stage training method to train a sonar video object detection network;

[0045] S32. Use asymmetric quantization method to quantize network weights.

[0046] Step S31 specifically includes: in the first stage, the network structure including the ME-CSPDarkNet53 backbone network, the PAFPN module and the detection head is trained using the forward-looking sonar data training set, and the feature screening module and the self-attention feature aggregation module are not involved in the training;

[0047] Load the first stage training weights to initialize the network, freeze the weights of the ME-CSPDarkNet53 backbone network, PAFPN module, and detection head module, and do not participate in back propagation. In this stage, fine-tune the network weights of the self-attention feature aggregation module;

[0048] Step S32 specifically includes: quantizing the original 32-bit network weight to 8 bits in an asymmetric quantization manner; the conversion formula between the original high-precision data and the quantized unsigned int8 data is:

[0049] (8)

[0050] Where scale is a float32 floating point number. is an unsigned int8 fixed-point number, offset is an int8 fixed-point number; the range of the data to be quantized is , the calculation method of scale and offset is as follows:

[0051] (9)

[0052] (10)

[0053] The original high-precision data is calculated to convert the unsigned int8 data, as shown in the following formula:

[0054] (11)

[0055] Preferably, step S4 specifically includes: using the tensor compiler ATC to parse the model under the ONNX framework into an intermediate IR Graph, and then converting it into an OM offline model adapted to the Ascend AI processor after further operations of graph preparation, graph splitting, graph optimization, and graph compilation, and finally loading the model file through the AscendCL interface to realize the reasoning process.

[0056] The second aspect of the present invention relates to a forward-looking sonar video target detection device based on a domestically produced Ascend AI processor, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a forward-looking sonar video target detection method based on a domestically produced Ascend AI processor of the present invention.

[0057] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a forward-looking sonar video target detection method based on a domestically produced Ascend AI processor of the present invention is implemented.

[0058] The working principle of the present invention is:

[0059] In order to improve the performance of the forward-looking sonar image target detection network in handling motion blur, target concealment and other situations, the present invention proposes a new backbone network with motion excitation to fully tap the potential motion features between frames. In addition, the present invention also proposes a self-attention feature aggregation module to fully utilize the rich feature information of the temporal context of the image sequence to alleviate the lack of confidence, motion blur, and hidden target recognition missed detection and false detection, so as to achieve more accurate target detection. The present invention uses the domestically produced Ascend Atlas200 AI platform to deploy a forward-looking sonar video target detector to achieve real-time offline processing and analysis of data.

[0060] The present invention has the following beneficial benefits:

[0061] 1. Building a forward-looking sonar video target detection intelligent hardware system based on the domestically produced Ascend Atlas200I A2 not only improves the flexibility and performance of deep neural network deployment, but also promotes the application and development of domestically produced AI processors in forward-looking sonar detection technology.

[0062] 2. Aiming at the problem of false detection and missed detection of motion blur, target concealment and other situations in forward-looking sonar images by traditional single-frame detectors, a video target detector which utilizes the temporal context of image sequences is proposed. The method mines the inter-frame motion features and aggregates the rich features of the image temporal context, which has good target detection accuracy in forward-looking sonar images. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flow chart of the method of the present invention.

[0064] Figure 2 It is a schematic diagram of the edge intelligent hardware system structure of the method of the present invention.

[0065] Figure 3 It is a video target detection model architecture diagram of the method of the present invention.

[0066] Figure 4 It is a schematic diagram of the motion excitation module of the method of the present invention.

[0067] Figure 5 It is a network model deployment flow chart of the method of the present invention.

[0068] Figure 6 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0069] In order to better understand the above technical solution, the technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0070] Example 1

[0071] like Figure 1-Figure 5 As shown, this embodiment relates to a sonar video target detection method based on a domestic Ascend AI processor, and the implementation process specifically includes the following steps:

[0072] S1. Build an edge intelligent hardware system based on the domestic Ascend AI processor, which consists of a main control chip, debugging interface, HDMI interface, Gigabit Ethernet interface, USB3.0 interface, SDIO module and WIFI module.

[0073] The main control chip is the Ascend Atlas200I A2 acceleration module, which integrates the HiSilicon Ascend310B AI processor, including a 4-core Taishan CPU and Da Vinci AI Core, and can provide a maximum of 20Tops (INT8) of AI computing power, which can provide computing power support for edge deep learning network deployment.

[0074] The debugging interface is connected to the RS232 interface from the main control chip to provide real-time log information. You can log in to the Linux terminal through this interface to debug the system.

[0075] The Gigabit Ethernet interface is connected to the PHY chip through the RGMII and MDIO interfaces of the main control chip, providing Gigabit network communication to the outside and can be connected to the lower computer system.

[0076] The USB3.0 interface provides two interfaces: USB OTG Type-C and USB Host Type-A. The OTG Type-C interface can be used to log in to the Linux terminal, and the USB Host Type-A interface can be used to connect peripherals such as U disk, keyboard, mouse, etc.

[0077] The SDIO interface is used to connect an external TF card, which stores system resources and data resources.

[0078] The WIFI module can be used to connect to an external wireless network to transmit data.

[0079] S2. Design of a deep neural network for forward-looking sonar video target detection, where the input to the model is a continuous sequence of images , where n is the sequence length. The backbone network in the model is ME-CSPDarkNet53 with motion excitation. The features extracted by the backbone network are fed to the inspection head after feature fusion by the PAFPN module to obtain the basic features of prediction. The feature screening module is further used to screen the features F. The screened features are linearly mapped and input into the self-attention feature aggregation module to aggregate the classification and regression features at different scales for final classification.

[0080] S21. By introducing a motion excitation module into the backbone network, motion information is enhanced by stimulating the motion-sensitive information between frames. This method does not require training a high-cost optical flow network. For a given input feature , where N represents the batch size, T represents the time step size, whose value is equal to n, C represents the feature channel, H represents the feature height, and W represents the feature width. First, 1*1 convolution is used to reduce the number of channels to improve efficiency. The formula is as follows:

[0081] (1)

[0082] in represents the feature after channel reduction, * represents the convolution operation, and r is the scaling factor.

[0083] The motion features at time t can be approximately represented as two adjacent frames. and The difference between , so the motion characteristics at time t can be expressed as:

[0084] (2)

[0085] in , the motion feature at the end of the time step is recorded as 0, that is , and record all motion features as , and then global average pooling is used to summarize the spatial information:

[0086] (3)

[0087] A 1*1 convolutional layer is used to expand the number of channels of the motion feature to the original channel dimension C, and the motion attention weight A is obtained using the Sigmoid(·) function. The formula is as follows:

[0088] (4)

[0089] Finally, residual connection is used to enhance motion information while retaining scene information. The specific formula is as follows:

[0090] (5)

[0091] in is the output of the motion excitation module, Denotes Hadamard Product.

[0092] S22. Filter high-quality features from the feature map output by the prediction head, use the TopK algorithm, sort from high to low according to the confidence score, and select the top k prediction features. Use the Non-Maximum Suppression (NMS) algorithm to continuously retrieve the detection box with the highest classification confidence, use the intersection-over-union ratio to represent the connection between two bounding boxes, and select a fixed number of predictions a to reduce redundancy. Define the classification and regression features of the filtered image as:

[0093]

[0094]

[0095] Where n is the input image sequence number, is the feature dimension of the region of interest (RoI).

[0096] S23. The recognition process usually observes information in a certain time series, combines spatial information and temporal motion information, obtains recognition results, and propagates confident results to cases with insufficient confidence. As a key part of Transformer, the multi-head self-attention mechanism is suitable for this purpose, which enhances the ability of long-distance modeling.

[0097] The filtered classification and regression features with high confidence are used as queries, and the definition is:

[0098]

[0099]

[0100] Where LP(·) is the linear projection operator. The dimension is projected into d dimensions.

[0101] Calculate the self-attention weights of the category and regression features, the formula is:

[0102] (6)

[0103] in The dimension of each feature in Q and K is divided by Can make It conforms to the N(0,1) distribution.

[0104] Since the main purpose is to improve the classification results, and will share the same matrix . Further Combined with the result of formula (6), the formula is as follows:

[0105] (7)

[0106] in Represents the concatenation operation, and then The final classification is performed after entering the average pooling layer.

[0107] S3. Use the two-stage training method to train the sonar video target detection network, save the trained network weights and structure as an ONNX framework model file, and use the asymmetric quantization method to quantize the weight process of the ONNX model.

[0108] S31. In the first stage, the forward-looking sonar data training set is used to train the network structure including the ME-CSPDarkNet53 backbone network, PAFPN module and detection head part. The feature screening module and self-attention module are not involved in the training.

[0109] Load the first stage training weights to initialize the network, freeze the weights of the ME-CSPDarkNet53 backbone network, PAFPN module, and detection head module, and do not participate in back propagation. In this stage, fine-tune the network weights of the self-attention module.

[0110] S32, the original network weight is of float32 type, occupying 32 bits. Compressing the network model size can reduce storage overhead and bandwidth requirements at the expense of a small amount of precision, achieving faster computing speed and lower power consumption. Therefore, the original 32-bit network weight is quantized to 8 bits in an asymmetric way.

[0111] The conversion formula between the original high-precision data and the quantized unsigned int8 data is:

[0112] (8)

[0113] Where scale is a float32 floating point number. is an unsigned int8 fixed-point number, and offset is an int8 fixed-point number. The range of the data to be quantized is , the calculation method of scale and offset is as follows:

[0114] (9)

[0115] (10)

[0116] The original high-precision data is calculated to convert the unsigned int8 data, as shown in the following formula:

[0117] (11)

[0118] S4. Use the Ascend CANN heterogeneous inference framework to deploy the sonar video target detection network designed in step S2 to the edge intelligent hardware system built in step S1.

[0119] CANN (Compute Architecture for Neural Networks) is a heterogeneous computing architecture launched by Ascend for AI scenarios. It supports multiple AI frameworks upward and serves AI processors and programming downward. It plays a key role in connecting the upper and lower levels and is a key platform for improving the computing efficiency of Ascend AI processors.

[0120] The Ascend Tensor Compiler (ATC) is used to parse the model under the ONNX framework into an intermediate IR Graph. After a series of operations such as graph preparation, graph splitting, graph optimization, and graph compilation, it is converted into an OM offline model adapted to the Ascend AI processor. Finally, the model file is loaded through the AscendCL (Ascend Computing Language) interface to implement the reasoning process.

[0121] The present invention overcomes the challenge of poor detection of motion blur and target concealment in forward-looking sonar videos by single-frame detectors, improves detection accuracy, and promotes the application and development of domestically produced AI processors in forward-looking sonar detection technology.

[0122] Example 2

[0123] like Figure 6This embodiment relates to a sonar video target detection device based on a domestic Ascend AI processor, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a sonar video target detection method based on a domestic Ascend AI processor of Example 1.

[0124] At the hardware level, the device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0125] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0126] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0127] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0128] For the convenience of description, the above device is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0129] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] Example 3

[0131] The present embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a sonar video target detection method based on a domestically produced Ascend AI processor of Embodiment 1 is implemented.

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0135] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0136] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0137] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0138] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0139] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0141] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A forward-looking sonar video target detection method based on the domestic Ascend AI processor, characterized by: The following steps are included: S1. Build an edge intelligent hardware system based on the domestic Ascend AI processor, which consists of a main control chip, a debugging interface, an HDMI interface, a Gigabit Ethernet interface, a USB3.0 interface, an SDIO interface, and a WIFI module; S2. Design a forward-looking sonar video target detection network of deep neural network type, wherein the input of the forward-looking sonar video target detection network is a continuous image sequence ,in n The length of the sequence is 1, and the backbone network is ME-CSPDarkNet53 with motion excitation. The features extracted by the backbone network are fed into the inspection head after feature fusion by the PAFPN module to obtain the basic features of the prediction, and the feature screening module is further used to screen the features. , after linear mapping of the filtered features, input them into the self-attention feature aggregation module, and aggregate the classification and regression features at different scales for final classification; including: S21. Design the ME-CSPDarkNet53 backbone network structure of the forward-looking sonar video target detection network; S22. Design a feature filtering module for forward-looking sonar video target detection network; S23. Design a self-attention feature aggregation module for forward-looking sonar video object detection network; specifically include: The recognition process usually observes information in a certain time series, combines spatial information and temporal motion information, obtains recognition results, and propagates confident results to cases with insufficient confidence. As a key part of Transformer, the multi-head self-attention mechanism is suitable for this purpose, which enhances the ability of long-distance modeling; The filtered classification and regression features with high confidence are used as queries, and the definition is: , where LP(·) is the linear projection operator, Dimensions projected onto d dimension; Calculate the self-attention weights of the category and regression features, the formula is: in for Q and K The dimension of each feature in is divided by Can make Conforms to N(0,1) distribution; and will share the same matrix , to improve the classification results; Combined with the result of formula (6), the formula is as follows: ,in Represents the concatenation operation, and then The final classification is performed after the average pooling layer is input; S3. Use the two-stage training method to train the sonar video target detection network, save the trained network weights and structure as an ONNX framework model file, and use the asymmetric quantization method to quantize the weights of the ONNX model; S4. Use the Ascend CANN heterogeneous inference framework to deploy the forward-looking sonar video target detection network designed in step S2 to the edge intelligent hardware system built in step S1.

2. The forward-looking sonar video target detection method based on the domestic Ascend AI processor as claimed in claim 1, characterized in that: Step S21 specifically includes: The motion excitation module is introduced into the backbone network to stimulate the motion sensitive information between frames and enhance the motion information. ,in N represents the batch size, T Represents the time step size, its value is equal to n , C represents the feature channel, H represents the feature height, W Represents the feature width. First, use 1*1 convolution to reduce the number of channels to improve efficiency. The formula is as follows: ,in represents the features after channel reduction, * represents the convolution operation, r is the scaling factor; for The motion features of the moment are approximately represented by two adjacent frames and The difference between The motion characteristics of the moment can be expressed as: ,in , the motion feature at the end of the time step is recorded as 0, that is = 0, and record all motion features as ], and then use global average pooling to summarize the spatial information: , using a 1*1 convolutional layer to expand the number of channels of motion features to the original channel dimension , use the Sigmoid(·) function to obtain the motion attention weight , the formula is as follows: Finally, residual connection is used to enhance motion information while retaining scene information. The specific formula is as follows: ,in is the output of the motion excitation module, It represents Hadamard Product.

3. The forward-looking sonar video target detection method based on the domestic Ascend AI processor as claimed in claim 1, characterized in that: Step S22 specifically includes: Select high-quality features from the feature map output by the prediction head, use the TopK algorithm, sort them from high to low according to the confidence score, and select the top features. k prediction features; use the non-maximum suppression (NMS) algorithm to continuously retrieve the detection box with the highest classification confidence, use the intersection-over-union ratio to represent the connection between two bounding boxes, and select a fixed number of a prediction to reduce redundancy; the classification and regression features of the filtered image are defined as: ,in n is the input image sequence number, is the feature dimension of the region of interest RoI.

4. The forward-looking sonar video target detection method based on the domestic Ascend AI processor as claimed in claim 1, characterized in that: Step S3 includes: S31. Using a two-stage training method to train a sonar video object detection network; S32. Use asymmetric quantization method to quantize network weights.

5. The forward-looking sonar video target detection method based on the domestic Ascend AI processor as claimed in claim 4, characterized in that: Step S31 specifically includes: In the first stage, the forward-looking sonar data training set is used to train the network structure including the ME-CSPDarkNet53 backbone network, PAFPN module and detection head part. The feature screening module and self-attention module are not involved in the training; Load the first stage training weights to initialize the network, freeze the weights of the ME-CSPDarkNet53 backbone network, PAFPN module, and detection head module, and do not participate in back propagation. In this stage, fine-tune the network weights of the self-attention module.

6. The forward-looking sonar video target detection method based on the domestic Ascend AI processor as claimed in claim 4, characterized in that: Step S32 specifically includes: The original 32-bit network weights are quantized to 8 bits in an asymmetric way; the conversion formula between the original high-precision data and the quantized unsigned int8 data is: ,in scale is a float32 floating point number, is an unsigned int8 fixed-point number. offset It is an int8 fixed-point number; The range of data to be quantified is ,but scale and offset The calculation method is as follows: , the original high-precision data is calculated to obtain the conversion of unsigned int8 data, which is shown in the following formula: .

7. The forward-looking sonar video target detection method based on the domestic Ascend AI processor as claimed in claim 1, characterized in that: Step S4 specifically includes: using the tensor compiler ATC to parse the model under the ONNX framework into an intermediate IRGraph, and then converting it into an OM offline model adapted to the Ascend AI processor after further operations of graph preparation, graph splitting, graph optimization, and graph compilation. Finally, the model file is loaded through the AscendCL interface to implement the reasoning process.

8. A forward-looking sonar video target detection device based on the domestic Ascend AI processor, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a forward-looking sonar video target detection method based on a domestically produced Ascend AI processor as described in any one of claims 1 to 7.

Images