A shallow sea underwater small sample target detection system and method

Abstract

The present application discloses a shallow water underwater small sample target detection system and method, which can detect underwater obstacle threat targets through passive imaging mode. The system comprises a wide frequency vibration electromechanical cooperative image stabilization platform, a micro polarization imaging system, a real-time sea fog enhancement processing device, an edge neural network computing unit and an underwater feature degradation target small sample detection technology deployed thereon. The system can solve the problems of platform jitter, water mist degradation and insufficient samples in the process of detecting shallow water obstacle threat targets through passive optical imaging means. The system improves the imaging clarity from the front-end imaging device and the back-end processing algorithm, and contains an intelligent detection module and a computing unit, which can realize real-time detection and rendering output of underwater obstacle threat targets at a low cost.

Description

Technical Field

[0001] The present invention relates to the fields of optoelectronic platform design and image target detection, and in particular to a shallow sea underwater small sample target detection system and method. Background Art

[0002] In shallow-water operations, non-cooperative forces often use natural or man-made obstacles, such as reefs, fishery equipment, and railroad forts, both on the surface and underwater, to slow down our progress in beaching and landing operations. These obstacles are numerous and concealed by mist and the shallow waters, posing a significant threat to ship navigation and landing operations. Only by fully understanding the location and types of these obstacles can we formulate a sound and safe plan for landing and beaching operations.

[0003] Current shallow-water optoelectronic reconnaissance equipment is mainly deployed on large manned ships. It uses passive optical equipment such as visible light and infrared to identify and warn of possible threats on the sea and in the air. However, there have been no reports on optoelectronic systems that detect and identify underwater targets based on passive optical equipment.

[0004] In addition, limited by the complexity and particularity of the near-shore combat environment, existing near-shore optoelectronic detection equipment cannot overcome the following unfavorable factors: load platform vibration caused by waves, bearing friction, and forward resistance, discontinuous imaging, and motion blur; and unclear imaging caused by image degradation factors such as water mist, water surface, and bad weather. At the same time, limited by the intelligence level of current optoelectronic detection equipment, some unmanned platforms still require manual identification of obstacle threat factors and remote control propulsion parameters, and cannot realize intelligent identification of obstacle threat factors at the edge. Summary of the Invention

[0005] The technical problem addressed by this invention is to overcome the shortcomings of existing technologies by providing a system and method for detecting small underwater targets in shallow water (fewer than 100 labeled boxes for a single target class). This system achieves high detection accuracy with low sample size, using a training method based on classification reinforcement to reduce the neural network's reliance on massive training data, even in sensitive scenarios where acquisition costs are high and imaging equipment is specialized, and where obtaining large amounts of raw data and labeled samples is difficult.

[0006] The technical solution of the present invention: A shallow water underwater small sample target detection system, including a micro-polarization imaging CMOS layer with multi-angle imaging characteristics, an FPGA image processing layer, and an edge end neural network computing unit;

[0007] The micro-polarization imaging CMOS layer with multi-angle imaging characteristics acquires array image data of multiple images at different incident angles through a single imaging;

[0008] The FPGA image processing layer leverages the speed advantage of hardware circuits to perform preliminary defogging and enhancement processing on the image using a real-time fog penetration enhancement processing algorithm, obtaining array image data with higher contrast. This higher-contrast array image data is then transmitted to the edge neural network computing unit via the interface layer. The imaging device, consisting of the polarization imaging CMOS layer and the FPGA image processing layer, collects fog-enhanced array image data from the mission scene and annotates small sample targets in the array image data to form a data set.

[0009] The contrastive learning method is then used to train the classification-enhanced small-sample underwater feature degradation target detection network, obtaining a weight file for the classification-enhanced small-sample underwater feature degradation target detection algorithm. The weight file is then deployed in the edge neural network computing unit. During training, the network input is two images before and after the transformation, and the network structure is a parallel branch with weight sharing. During the training process, the consistency of the classification results of the two-way branch before and after the transformation is enhanced by constraints to train the network and enhance the discrimination ability of the network classification branch.

[0010] The edge neural network computing unit runs the classification-enhanced small-sample underwater feature degradation target detection algorithm in real time with a typical power consumption of less than or equal to 70W: During operation, the edge neural network computing unit receives the array image data after fog penetration enhancement processing by the FPGA image processing layer, and calculates the classification-enhanced small-sample underwater feature degradation target detection weight file deployed on it to obtain the target detection result corresponding to each array image input, draws the calculated target detection result on the array input image, and outputs the image containing the detection box result to the external display.

[0011] Furthermore, the micro-polarization imaging part is implemented as follows:

[0012] (1) Based on the substrate size limitation, the PCB design of the micro-polarization CMOS layer, FPGA image processing layer, power supply and interface layer is completed. The micro-polarization layer realizes 0°, 45°, 90°, and 135° micro-polarization array imaging, and the amount of single imaging data exceeds that of ordinary cameras of the same size. The FPGA image processing layer is equipped with a DDR memory chip to achieve buffering of polarization array imaging, and is equipped with an FPGA chip to utilize the fast processing characteristics of its hardware circuit to perform microsecond-level low-latency enhancement processing on the array image data output by the micro-polarization CMOS layer.

[0013] (2) The micro-polarization CMOS layer, FPGA image processing layer, power supply and interface layer are arranged in order from front to back, and the three-layer circuit board of the micro-polarization CMOS layer, FPGA image processing layer, power supply and interface layer are flexibly interconnected through a rigid-flexible FPC soft cable to suppress the internal symbiotic noise of the three-layer circuit board during signal interaction and improve the signal-to-noise ratio of the original photoelectric signal conversion.

[0014] Compared with active detection imaging methods, it is lower in cost and more concealed. Compared with traditional optical cameras, it can effectively suppress the influence of scattered atmospheric light and stray light on the water surface.

[0015] Furthermore, the network training process of the small sample underwater feature degradation target detection algorithm based on classification enhancement is as follows:

[0016] (1) First, the labeled data of shallow water underwater obstacles are obtained, and the original data are translated, rotated, and blurred data enhancement operations are performed. The array image data before and after the transformation form a set of image data pairs before and after the transformation, and the projection transformation matrix T before and after the data enhancement transformation is calculated;

[0017] (2) Comparative learning is performed on the data pairs before and after the transformation. The small sample underwater feature degradation target detection algorithm based on classification enhancement contains two upper and lower branches with parameter sharing. The upper branch calculates the position proposal (region proposal, which is the four coordinate points describing an object in the image) and the projection transformation matrix T of the original image through forward reasoning operation. After multiplying the four coordinate points in the position proposal by the projection transformation matrix T in sequence, the region corresponding to the feature extracted from the transformed image by the lower branch is obtained.

[0018] (3) In the upper and lower branches, the corresponding areas in the image features before and after the transformation are proposed according to the position, and the features of the corresponding areas are cropped from the depth feature maps of the two sets of image data before and after the transformation. The classification results are calculated based on the cropped areas respectively. The difference in classification results is minimized by measuring the loss, so as to achieve the purpose of enhancing the classification effect and obtain the weight file of the small sample underwater feature degradation target detection algorithm based on classification enhancement.

[0019] The advantages of the present invention compared with the prior art are:

[0020] (1) Reasonable Design: To achieve the goal of improving image quality, we focus on two aspects: vibration isolation and suppression at the physical level and defogging and enhancement at the algorithm level. This avoids the occurrence of the phenomenon that the algorithm cannot save due to excessive deviation at the physical level. At the same time, we make full use of the flexible characteristics of the algorithm and combine it with physical means to improve the upper limit of imaging quality.

[0021] (2) Cost control: Clear imaging and detection of underwater targets can be achieved by using passive optical imaging equipment, without the need for expensive and complex active imaging equipment such as lidar. The cost is controllable and easy to promote.

[0022] (3) High intelligence: the small sample underwater feature degradation target detection algorithm based on classification enhancement and the edge neural network computing unit are integrated into the device, which can realize real-time and automatic analysis of the acquired data stream without the need for back-end personnel to monitor and interpret it, and has a high level of intelligence.

[0023] (4) Low dependence on data volume. The small-sample underwater feature degradation target detection algorithm based on classification enhancement can achieve generalized detection of other targets of the same type when the number of labeled samples of a single target class is less than 100. This avoids the heavy dependence of common detection algorithms on data volume and is more suitable for the objective conditions of sensitive target types and constant data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall structural diagram of the system of the present invention;

[0025] Figure 2 for Figure 1 Implementation block diagram;

[0026] Figure 3 Comparison of image quality before and after image stabilization;

[0027] Figure 4 This is the design diagram of the micro-polarization imaging system;

[0028] Figure 5 Comparison of imaging effects of the micro-polarization imaging system at angles of 0° and 90°;

[0029] Figure 6 This is the water image before dehazing enhancement;

[0030] Figure 7 This is the enhanced effect of defogging;

[0031] Figure 8 It is a small sample underwater feature degradation target detection algorithm based on classification enhancement. DETAILED DESCRIPTION

[0032] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0033] like Figure 1 As shown, the shallow water small sample target detection system of the present invention includes a broadband vibration electromechanical coordinated image stabilization platform, a micro-polarization imaging system, a real-time sea fog enhancement processing device, an edge neural network computing unit, and an underwater feature degraded target small sample detection algorithm deployed thereon; Figure 2The connection relationship between the various parts involved in the system is shown in the figure: the overall system can be divided into two major parts: optoelectronic processing and broadband vibration electromechanical collaborative stabilization platform. The optoelectronic processing part is fixed to the broadband vibration electromechanical collaborative stabilization platform by bolts; the optoelectronic processing part can be further divided into hardware circuit layer and software algorithm layer. The external light signal is processed layer by layer in the hardware circuit layer to obtain the final display output result; the algorithm of the software algorithm layer actually runs in the hardware circuit. Specifically: the real-time fog penetration enhancement processing algorithm runs in the FPGA processing layer, and the underwater feature degradation small sample target detection algorithm runs in the edge neural network computing unit.

[0034] The specific implementation of each part is as follows:

[0035] (1) Broadband vibration electromechanical coordinated image stabilization platform: used to isolate and suppress vibrations of the detection system carrier caused by factors such as waves, transmission shaft vibrations, and motion resistance, thereby reducing image tailing and motion blur caused by vibrations at the source. Furthermore, there are no specific requirements for the design method and working principle of the image stabilization fixed platform, and only the following requirements are made for its performance indicators:

[0036] Maximum pitch angle ≥±40°;

[0037] Maximum roll angle ≥±40°;

[0038] Maximum yaw angle ≥ ±180;

[0039] Vibration isolation and suppression frequency range 2Hz to 80Hz;

[0040] Vibration attenuation rate is better than -20dB at 10Hz;

[0041] After installing the bottom stabilization fixed platform, the image blur effect is significantly suppressed. Figure 3 shown. Figure 3 In the middle, the left side is without the bottom stabilization fixed platform, directly Figure 1 The center image shows the image quality when the optoelectronic processing unit is fixed to the ship platform. The right image shows the image quality after a broadband vibration electromechanical synergistic image stabilization platform is installed between the optoelectronic processing unit and the ship platform. Figure 3 The left and right parts in the middle are images of the same scene. The difference in blur between the left and right parts proves the necessity of using a broadband vibration electromechanical collaborative stabilization platform.

[0042] (2) Micro-polarization imaging system, used to suppress the influence of scattered atmospheric light and stray light on the water surface, improve observation contrast, and clearly image shallow underwater targets through passive imaging; the micro-polarization imaging system uses Sony IMX250MZR to realize the conversion of external environmental light signals into array imaging data. The sensor adopts pixel quadrupole polarization filtering technology and can perform polarization imaging of four angles (0°, 45°, 90° and 135°) for a single pixel. In addition, the following requirements are made for the integrated design of imaging chips, image analysis and processing layers, and transmission interfaces: Use a rigid-flexible FPC connection method to flexibly interconnect the CMOS photoelectric sensor chip, image analysis and processing layer, and transmission interface components. Its structure is as follows: Figure 4 As shown. Figure 4 The ambient light signal is converted into a television signal by the micro-polarization CMOS layer. The original image is then enhanced by the processing layer, and finally transmitted to the edge neural network computing unit via the power supply and interface layers. The integrated design used within the micro-polarization imaging system is well-suited to the mission-constrained platform, effectively suppressing internal detector symbiotic noise and improving the signal-to-noise ratio of the original photoelectric signal conversion.

[0043] Figure 5 The following image shows the array image data obtained by the micro-polarization imaging system after a single photoelectric conversion of the same scene: Figure 5 Left: When the polarization angle is 0 degrees in the current scene, it can better reflect the state of the underwater target; Figure 5 Right: A 90-degree polarization angle effectively reflects the state of surface targets. Using a micro-polarization imaging CMOS chip helps comprehensively capture scene information in shallow waters, improving the recall rate of shallow water obstacle threats.

[0044] (3) A real-time sea fog enhancement processing device performs real-time defogging and enhancement processing on the image acquired by the micro-polarization system through the dark channel prior defogging algorithm deployed on the FPGA side. The detailed implementation process is as follows:

[0045] The imaging model of foggy images is expressed as:

[0046] I(x)=J(x)t(x)+A(1-t(x))

[0047] Where I(x) is the collected foggy image, J(x) is the expected fog-free image, A is the atmospheric light value, t(x) is the transmittance, and x is used to refer to each pixel in the image.

[0048] The dark channel of an image is calculated as:

[0049] P dark =min y∈Ω(x) (min c∈(r,g,b) Pc (y))

[0050] Where P represents an image, P dark represents the dark channel of the current image, Ω(x) represents a square area with a side length of 9 (or other values, unlimited length) centered on pixel x, and c is used to traverse the three color channels R (red), G (green), and B (blue) of the image.

[0051] Based on the prior information that the dark channel of a fog-free image is approximately 0, the calculation formula for the transmittance t(x) is:

[0052]

[0053] Map the top 0.1% of pixel values ​​in the dark channel of the foggy image I(x) to the foggy image I(x). Select the pixel with the highest value in this region in I(x) as the atmospheric light A. Here, y is a temporary variable used to traverse all pixels within the square region Ω(x) centered on pixel x. The formula contains two mins: the former is used to find the minimum point within the region, and the latter is used to find the minimum point in the channel (R, G, B) dimensions.

[0054] The final calculated expected fog-free image J(x) is:

[0055]

[0056] The image dehazing algorithm is deployed on the processing layer chip to ensure the clarity of the image data output by the interface layer. Figure 6 The image results without dehazing are shown; Figure 7 The effect of defogging is shown, which shows that the present invention can effectively eliminate the influence of image degradation factors such as water mist on image clarity in shallow sea environments.

[0057] (4) Small sample detection algorithm for underwater feature degraded targets, which uses classification reinforcement based on contrastive learning for training, and effectively detects other unlabeled targets of the same type through effective learning of small sample labeled data sets (the number of single-class target samples is less than 100); Small sample underwater feature degraded target detection algorithm based on classification reinforcement: collect original images containing targets such as reefs, rail forts, simulated mines, and fishery production equipment through real-scene collection, simulated scenes, etc., and enhance the original images before labeling the above-mentioned types of targets.

[0058] Then, the labeled data is enhanced and trained using contrastive learning. The principle is as follows: Figure 8As shown. In the figure, Backbone is the backbone network, which is used to extract deep features from the image; FPN is the feature pyramid, which extracts feature maps of different scales from the backbone network and combines them into a pyramid form; RPN is the region proposal network, which proposes the coordinates of the region that may be the foreground target based on the feature pyramid; Roi Pooling and ROI Feat Extrator crop the corresponding region from the aforementioned feature map according to the coordinate position of the region proposal, and scale it to a uniform size for use in subsequent processing; Boxclassifier obtains the classification result vector of the region based on the cropped image features. The original annotated image is rotated, translated, blurred and other transformations are performed on the original annotated image, and the position of the transformed ROI is calculated according to the parameter settings in the transformation process. The feature map corresponding to the transformed image is directly obtained from the FPN based on the calculated position. The classification result obtained based on the feature map is measured by distance with the classification result before transformation. The distance measurement loss is reduced by reversely updating the network parameters to achieve the purpose of enhancing the classification branch. The distance measurement L used in the process is calculated as follows:

[0059]

[0060]

[0061]

[0062] Where N is the batch size during training, which is manually set during training; τ is a normalized hyperparameter with an empirical value of 0.1; Z is the vector output by the classification branch of the detection network, whose length is equal to the number of categories in the training sample; μ and ν are both references used to express the calculation method and have no practical meaning. In expression l, i and j are subscript values, which are used to refer to the vector z output by the classification branch of the detection network respectively. i With z j ; k in the expression L is the subscript value, traversing every integer from 1 to N; this distance metric calculation method realizes the conversion of the target classification result from a vector to a scalar. After back propagation based on the scalar L, the consistency constraint of the classification result of the same target before and after the transformation can be achieved, which can effectively strengthen the classification branch of the network.

[0063] (5) Edge neural network computing unit: used to run the trained detection model in real time with low power consumption and render the detection results. The edge neural network computing unit uses the ARM+NPU computing architecture. ARM devices contain a variety of microcontrollers that can execute complex control logic, which performs overall task distribution and scheduling. NPU devices are used to perform parallel computing, effectively speeding up the processing speed of neural networks. Based on the floating-point conversion tool chain in Huawei Ascend CANN series, the trained model file is low-bit quantized, which greatly reduces the amount of computation required for network forward processing while maintaining the original detection accuracy. The quantized model file is run based on the forward processing tool chain of Huawei Ascend CANN series to process the detection results, and the bounding box in the form of coordinate points is converted into a layer superimposed on the original image and output for display.

[0064] The above embodiments are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. A shallow water underwater small sample target detection system, characterized by: It includes a micro-polarization imaging CMOS layer with multi-angle imaging characteristics, an FPGA image processing layer, and an edge neural network computing unit. The micro-polarization imaging CMOS layer with multi-angle imaging characteristics acquires array image data of multiple images at different incident angles through a single imaging. The FPGA image processing layer leverages the speed advantage of hardware circuits to perform preliminary defogging and enhancement processing on the image using a real-time fog penetration enhancement processing algorithm, obtaining array image data with higher contrast. This higher-contrast array image data is then transmitted to the edge neural network computing unit via the interface layer. The imaging device, consisting of the polarization imaging CMOS layer and the FPGA image processing layer, collects fog-enhanced array image data from the mission scene and annotates small sample targets in the array image data to form a data set. Then, the contrastive learning method is used to train the classification-enhanced small-sample underwater feature degradation target detection network to obtain the classification-enhanced small-sample underwater feature degradation target detection algorithm weight file, and the weight file is deployed in the edge neural network computing unit; During training, the network input is two images before and after the transformation, and the network structure is a parallel branch with weight sharing; During the training process, the network is trained by constraining the consistency of the classification results of the two-way branches before and after the transformation to enhance the discrimination ability of the network classification branch; The edge neural network computing unit runs the classification-enhanced small-sample underwater feature degradation target detection algorithm in real time with a typical power consumption of less than or equal to 70W: During operation, the edge neural network computing unit receives the array image data after fog penetration enhancement processing by the FPGA image processing layer, and calculates the classification-enhanced small-sample underwater feature degradation target detection weight file deployed on it to obtain the target detection result corresponding to each array image input, draws the calculated target detection result on the array input image, and outputs the image containing the detection box result to the external display.

2. The shallow sea underwater small sample target detection system according to claim 1, characterized in that: The micro-polarization imaging part is implemented as follows: (1) Based on the substrate size limitation, the PCB design of the micro-polarization CMOS layer, FPGA image processing layer, power supply and interface layer is completed. The micro-polarization layer realizes 0°, 45°, 90°, and 135° micro-polarization array imaging, and the amount of single imaging data exceeds that of ordinary cameras of the same size. The FPGA image processing layer is equipped with a DDR memory chip to achieve buffering of polarization array imaging, and is equipped with an FPGA chip to utilize the fast processing characteristics of its hardware circuit to perform microsecond-level low-latency enhancement processing on the array image data output by the micro-polarization CMOS layer. (2) Arrange the micro-polarization CMOS layer, FPGA image processing layer, power supply and interface layer in order from front to back, and flexibly interconnect the three-layer circuit board of the micro-polarization CMOS layer, FPGA image processing layer, power supply and interface layer through a rigid-flexible FPC soft cable to suppress the internal symbiotic noise of the three-layer circuit board during signal interaction and improve the signal-to-noise ratio of the original photoelectric signal conversion; Compared with active detection imaging methods, it is lower in cost and more concealed. Compared with traditional optical cameras, it can effectively suppress the influence of scattered atmospheric light and stray light on the water surface.

3. The shallow sea underwater small sample target detection system according to claim 1, characterized in that: The network training process of the small sample underwater feature degradation target detection algorithm based on classification enhancement is as follows: (1) First, the labeled data of shallow water underwater obstacles are obtained, and the original data are translated, rotated, and blurred data enhancement operations are performed. The array image data before and after the transformation form a set of image data pairs before and after the transformation, and the projection transformation matrix T before and after the data enhancement transformation is calculated; (2) Comparative learning is performed on the data pairs before and after the transformation. The classification-enhanced small sample underwater feature degradation target detection algorithm contains two parameter-sharing upper and lower branches. The upper branch calculates the position proposal and projection transformation matrix T of the original image through forward reasoning operation. After multiplying the four coordinate points in the position proposal by the projection transformation matrix T in sequence, the corresponding area of ​​the position proposal in the features extracted from the transformed image by the lower branch is obtained. (3) In the upper and lower branches, the corresponding areas in the image features before and after the transformation are proposed according to the position, and the features of the corresponding areas are cropped from the depth feature maps of the two sets of image data before and after the transformation. The classification results are calculated based on the cropped areas respectively. The difference in classification results is minimized by measuring the loss, so as to achieve the purpose of enhancing the classification effect and obtain the weight file of the small sample underwater feature degradation target detection algorithm based on classification enhancement.

Images