Small target ship sign detection method, system and device for weak light environment

Abstract

The invention provides a small target ship sign detection method, system and device for a weak light environment, and the method comprises the steps: firstly obtaining a weak light image of a small target ship sign, extracting feature representations of different scales, carrying out the pixel-by-pixel enhancement operation, and combining the enhanced features with an original image; performing feature extraction on the restored image to obtain multi-layer features of different scales; secondly, fusing high-dimensional features and low-dimensional features of feature extraction, performing channel number recovery and layer-by-layer progressive processing on the merged features, and performing feature fusion and channel number recovery processing on the obtained features in a layer-by-layer progressive decreasing manner to obtain multi-layer feature representation; and finally, independently processing and calculating the features of each layer, and carrying out non-maximum suppression on the results to obtain a unified final small target ship license plate detection result. According to the method, through combination of image representation enhancement and ship sign detection, the small target ship sign can be accurately detected in a weak light environment, and the method has important practical significance in the aspects of safety monitoring, ship sign identification and the like in the ship field.

Description

Technical Field

[0001] The present invention relates to the fields of natural scene weak-light image enhancement technology and target detection technology, and in particular to a small target ship plate detection method, system and device for a weak-light environment. Background Art

[0002] With the growth of global trade, domestic and international trade transportation has become more frequent. Among them, water transportation has developed rapidly due to its unique advantages and has become an indispensable and important part of trade transportation.

[0003] Intelligent traffic management systems have significantly improved the efficiency and safety of land transportation. On the contrary, in water transportation, especially in busy large ports and complex waters, traffic management has progressed relatively slowly and relies more on manual participation in management and recording. This approach is not only inefficient, but also prone to errors, and wastes human and material resources. In the field of waterway traffic management, there are significant differences between the detection and recognition of ship license plates and the detection and recognition of vehicle license plates in land transportation. Vehicle license plates are easier to detect due to their uniformly prescribed style, font and fixed installation position. In contrast, the position of ship license plates on the hull is not fixed, and the font style and number of characters vary greatly, which brings considerable challenges to the detection work. Existing detection algorithms mainly focus on the global features of the target, such as shape, texture, etc., but do not make full use of fine-grained features such as some subtle strokes and textures, which makes the detection accuracy of some small or low-resolution ship license plate targets low. In addition, most of the existing detection algorithms rely on the appearance features of the target, and the detection performance will be significantly reduced in weak light environments such as insufficient light and shadows. This is very common in practical applications and needs to be further solved. Summary of the invention

[0004] The present invention aims to address the deficiencies of the prior art and proposes a method, system and device for detecting small target ship license plates in a weak light environment.

[0005] The object of the present invention is achieved through the following technical scheme: The present invention proposes a small target ship license plate detection method for a weak light environment, the method comprising the following steps:

[0006] (1) Image restoration: Obtain a low-light image of a small target ship plate, extract feature representations at different scales, perform pixel-by-pixel enhancement, and combine the enhanced features with the original image;

[0007] (2) Feature extraction: The image restored in step (1) is divided into a series of non-overlapping blocks, and feature extraction is performed on each block to obtain multi-layer features of different scales;

[0008] (3) Multi-granularity feature interaction: The high-dimensional features extracted from the features are merged with the low-dimensional features, the number of channels of the merged features are restored, and the features are processed layer by layer. The obtained features are then fused and the number of channels is restored again to obtain a multi-layer feature representation.

[0009] (4) License plate detection: Each layer of features is processed and calculated separately, and non-maximum suppression is performed on these results to obtain a unified final result of small target license plate detection.

[0010] Furthermore, in step (1), the pixel-by-pixel enhancement operation is implemented using a feature enhancement network, which consists of a plurality of serially connected pixel-by-pixel enhancement modules, each of which is composed of a series of "point-by-point convolution-depthwise separable convolution-point-by-point convolution" structures; and layer scaling technology is used to promote model convergence by multiplying the feature map with a series of scaling factors.

[0011] Furthermore, in step (2), feature extraction is performed by sending each block into the embedding layer one by one, and mapping the pixels in the block into a high-dimensional feature vector. During the feature extraction process, each small block will be processed by a multi-layer transposed attention mechanism and a fully connected layer, and through a hierarchical structure, the image will be divided into multiple stages according to different resolutions. Each lower resolution stage performs a self-attention operation on its corresponding higher resolution stage to achieve information transfer and feature fusion between different levels.

[0012] Furthermore, in step (3), a layer-by-layer upsampling operation is performed on the high-dimensional features obtained by feature extraction to achieve the same scale as the low-dimensional features.

[0013] Furthermore, in step (4), the multi-layer features are decoupled, each feature layer is processed separately, the results of each layer are calculated separately, and then the results of the features at different scales are restored to the unified input image scale.

[0014] Furthermore, the overall process uses Focal-EIoU loss as the loss function. The specific formula of the loss function is as follows:

[0015] L Focal-EIoU =IoU γ L EIoU

[0016]

[0017] Among them, γ is a hyperparameter used to control the curvature of the curve, c w and c h are the width and height of the minimum bounding box covering two boxes, and ρ(·) represents the Euclidean distance.

[0018] In a second aspect, the present invention further provides a small target ship license plate detection system for a weak light environment, the system comprising:

[0019] Image restoration module, used to obtain low-light images of small target ship plates, extract feature representations of different scales, perform pixel-by-pixel enhancement operations, and combine the enhanced features with the original image;

[0020] The feature extraction module is used to divide the restored image into a series of non-overlapping blocks, and perform feature extraction on each block to obtain multi-layer features of different scales;

[0021] The multi-granularity feature interaction module is used to fuse the high-dimensional features and low-dimensional features extracted from the features, restore the number of channels of the merged features, and process them layer by layer. The obtained features are then fused and the number of channels is restored again to obtain a multi-layer feature representation.

[0022] The license plate detection module is used to process and calculate each layer of features separately, perform non-maximum suppression on these results, and obtain a unified final result of small target license plate detection.

[0023] In the third aspect, the present invention also provides a small target license plate detection device for low-light environments, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the small target license plate detection method for low-light environments is implemented.

[0024] In a fourth aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method for detecting small target ship license plates in a weak light environment is implemented.

[0025] In a fifth aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the small target ship license plate detection method for a weak light environment.

[0026] The beneficial effect of the present invention is that we propose a new target detection framework specifically for small target ship plates in low-light environments, which enables training of a ship plate detection model with more accurate detection effects. Our detection framework uses a multi-granularity feature interaction module to interactively calculate features between various levels, so that more detailed features are used, which helps the model focus more on small-scale targets during training. Thus, more accurate detection results are obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 The present invention is a flow chart of a method for detecting small target ship license plates in a weak light environment.

[0029] Figure 2 It is a schematic diagram showing the detailed calculation process of the enhancement module in the present invention.

[0030] Figure 3 Schematic diagram of multi-granularity feature interaction in the present invention.

[0031] Figure 4 The present invention is a structural diagram of a small target ship license plate detection device for a weak light environment. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0034] Based on the current target detection technology, the present invention focuses on the method of detecting small target license plates in a weak light environment. In response to this problem, the research methods in the field of target detection are borrowed, and a small target license plate detection method facing a weak light environment is proposed, which is used to effectively detect small target license plates in a weak light environment. The design in this model includes two key modules, which are intended to enhance image representation and improve the processing capabilities of small target license plates. First, the present invention designs an image representation enhancement module. The present invention believes that when effectively detecting small target license plates in a weak light environment, the first step is to enhance the brightness of the weak light image. By enhancing the brightness through illumination correction, the visibility of the image can be improved, making the small target license plates easier to detect. At the same time, the present invention also needs to ensure that the enhanced image can better show the characteristics of the small target license plates in a weak light environment. This can be achieved by adopting appropriate illumination enhancement technology and feature enhancement methods, so that the details of the license plates are clearer and more recognizable, and the accuracy of subsequent processing is improved.

[0035] Secondly, the present invention designs a license plate detection module, which adopts a Transformer-based feature extraction network with a multi-layer structure. This network structure can better model the characteristics of small target license plates; it emphasizes the mining of fine-grained features, which include information such as details and changes of small targets. Through the design of a multi-layer structure, it can more accurately capture and extract richer and more expressive fine-grained features of small target license plates, thereby improving the understanding and distinction of small target license plates. Therefore, in the algorithm design, the present invention should pay more attention to the extraction and utilization of these fine-grained features. A feature extraction network with strong expressive ability can be used, combined with methods such as attention mechanism, to improve the perception and understanding of fine-grained features.

[0036] In addition, these fine-grained features should be fully utilized during model training. By focusing on marking the details and changes of small target ship license plates in the training data and designing appropriate loss functions and training strategies, the model's ability to recognize small targets can be enhanced. At the same time, some data enhancement techniques, such as rotation, scaling, and occlusion, are introduced to increase the diversity of training data and further improve the robustness and generalization ability of the model.

[0037] In the license plate detection module, a multi-granularity feature interaction module is also designed to effectively fuse feature representations of different granularities. This module can comprehensively utilize the different granularity features output by the feature extraction module to fully capture the global and local information of the license plate. Through the interaction and fusion of features, the model can better understand the structure and contextual relationship of small target license plates, and improve the accuracy and robustness of detection.

[0038] The present invention proposes a small target ship license plate detection method in a weak light environment, such as Figure 1 As shown, the specific steps include:

[0039] (1) Image restoration

[0040] (1-1) First, obtain a low-light image containing a small target ship sign Generate feature images using different convolutional networks on I and Get feature representations of different scales. H and W represent the height and width of the image. Based on multi-level spatial feature representation, it can provide rich information for the subsequent feature enhancement network.

[0041] (1-2) The feature enhancement network consists of multiple serially connected pixel-by-pixel enhancement modules, such as Figure 2As shown in the figure, each pixel-by-pixel enhancement module is composed of a series of "point-wise convolution (PWConv) - depth-wise separable convolution (DWConv) - point-by-point convolution (PWConv)" structures. This result can enhance the local detail performance of the target, while removing some noise in the image and improving the image quality. At the same time, layer scaling technology is also used to promote better convergence of the model by multiplying the feature map with a series of scaling factors k1 and k2. Based on the feature enhancement network, a multi-scale feature representation from the feature enhancement network is obtained. and Small scale feature F b contains finer details and boundaries, larger scale features F a Capture more abstract information like shapes and patterns.

[0042] (1-3) F a and F b First, the convolutional network is used to restore the image to the same latitude as the original image, and then the residual connection is used to combine the original image. This mechanism helps to retain the original feature information and provides support for the preservation of image details. Through such a design, not only can the representation ability of the feature be enhanced, but also the integrity of the feature can be maintained during the enhancement process, providing high-quality feature information for subsequent detection processing tasks.

[0043] (2) Feature extraction:

[0044] (2-1) The feature extraction network uses the Swin Transformer network. It receives the image output from the representation enhancement network as its input. It first divides the input image into a series of non-overlapping small blocks, and sends these small blocks one by one to the embedding layer. The role of the embedding layer is to map the pixels in the block to a high-dimensional feature vector. During the feature extraction process, each small block will be processed by a multi-layer transposed attention mechanism and a fully connected layer to achieve information exchange and feature integration between blocks, ensuring that each block can obtain information from other blocks. In addition, it also has a hierarchical structure, which divides the image into multiple stages from low to high according to different resolutions. Each stage will perform autonomous attention operations on the corresponding next resolution stage to achieve information transmission and feature fusion between different levels.

[0045] (3) Multi-granularity feature interaction:

[0046] (3-1) The output of the feature extraction network is four layers of features at different scales. Figure 3As shown in the figure, C2F and conv are both wrappers for the convolution layer and the regularization layer, which perform layer-by-layer upsampling operations on high-dimensional features to keep the same scale as the low-dimensional features. They are then fused with the next layer of low-dimensional features, and the number of channels of the merged features is restored. This process is repeated layer by layer.

[0047] (3-2) The obtained features at each level are passed through a convolution layer and a regularization layer to obtain small-scale features, which are then gradually reduced and feature fusion and channel number recovery are performed again. A new feature matrix containing richer feature descriptions is constructed. This new feature matrix can provide more comprehensive information for model training.

[0048] Through the above two-stage processing, feature maps of different scales can be effectively integrated to obtain a new feature matrix containing more information. And by restoring the number of channels, the representation of the feature map is enhanced, so that the details of small targets, such as texture, morphology and other features, can be captured more accurately. These more detailed features are crucial to improving the effect of small target license plate detection.

[0049] (4) Vessel license plate detection:

[0050] The detection head receives the above multi-layer feature representation and performs decoupling processing on the multi-layer features, that is, processing each feature layer separately and calculating the results of each layer separately. Then, the results of features at different scales are restored to the unified input image scale, and these results are post-processed such as NMS to obtain a unified final result of small target ship license plate detection.

[0051] (5) Loss function training:

[0052] (5-1) Given the prediction result box B and the target box B gt , b and b gt Represents the center point of the two boxes. Focal-EIoU loss is used as the loss function, and the calculation formula is:

[0053] L Focal-EIoU =IoU γ L EIoU

[0054] It consists of EIoU Loss and FocalL1 loss, where γ is a hyperparameter used to control the curvature of the curve. EIoU-Loss further decomposes the influencing factors of the aspect ratio and independently calculates the length and width of the target box and the prediction box. This part of the loss function consists of three main parts: overlapping area loss, center point distance loss, and aspect ratio loss. Among them, the aspect ratio loss accelerates the convergence of the model by directly minimizing the difference in width and height between the target box and the prediction box. The specific formula of the loss function is as follows:

[0055]

[0056] where c w and c h is the width and height of the smallest bounding box covering the two boxes, ρ(b,b gt )=||bb gt || 2 Represents the Euclidean distance. This loss function decomposes the aspect ratio loss into the difference between the width and height of the predicted box and the width and height of the minimum closed box, which not only speeds up the model convergence but also improves the accuracy of the regression task.

[0057] During the training process, the model was trained with the Adam optimizer for 100 cycles. The initial learning rate was 0.0001, and the batch size was set to 16. Mosaic operation was also used, and it was turned off in the last 10 epochs. Mosaic operation is a data enhancement operation suitable for target detection. The main idea is to stitch four pictures into one picture as a training sample. Mosaic is suitable for detection, and the coordinates of the target box should also be changed accordingly when stitching.

[0058] Corresponding to the above-mentioned embodiment of a small target ship license plate detection method for a weak light environment, the present invention also provides an embodiment of a small target ship license plate detection system for a weak light environment. For the implementation process of each module in the system, please refer to the specific steps of the above-mentioned embodiment of a small target ship license plate detection method for a weak light environment. The system includes:

[0059] Image restoration module, used to obtain low-light images of small target ship plates, extract feature representations of different scales, perform pixel-by-pixel enhancement operations, and combine the enhanced features with the original image;

[0060] The feature extraction module is used to divide the restored image into a series of non-overlapping blocks, and perform feature extraction on each block to obtain multi-layer features of different scales;

[0061] The multi-granularity feature interaction module is used to fuse the high-dimensional features and low-dimensional features extracted from the features, restore the number of channels of the merged features, and process them layer by layer. The obtained features are then fused and the number of channels is restored again to obtain a multi-layer feature representation.

[0062] The license plate detection module is used to process and calculate each layer of features separately, perform non-maximum suppression on these results, and obtain a unified final result of small target license plate detection.

[0063] Corresponding to the aforementioned embodiment of a small target ship license plate detection method for a weak light environment, the present invention further provides an embodiment of a small target ship license plate detection device for a weak light environment.

[0064] See also Figure 4 An embodiment of the present invention provides a small target license plate detection device for a weak light environment, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a small target license plate detection method for a weak light environment in the above embodiment.

[0065] An embodiment of a small target license plate detection device for a low-light environment provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 4 As shown in FIG. 1 , a hardware structure diagram of a small target ship license plate detection device for a weak light environment provided by the present invention is provided for any device with data processing capability, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0066] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0067] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0068] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a small target ship license plate detection method for a weak light environment in the above embodiment is implemented.

[0069] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0070] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for detecting small target ship license plates in a weak light environment is implemented.

[0071] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A small target ship plate detection method for low light environment, characterized in that: The method comprises the following steps: (1) Image restoration: Obtain a low-light image of a small target ship plate, extract feature representations at different scales, perform pixel-by-pixel enhancement, and combine the enhanced features with the original image; (2) Feature extraction: The image restored in step (1) is divided into a series of non-overlapping blocks, and feature extraction is performed on each block to obtain multi-layer features of different scales; (3) Multi-granularity feature interaction: The high-dimensional features extracted from the features are merged with the low-dimensional features, the number of channels of the merged features are restored, and the features are processed layer by layer. The obtained features are then fused and the number of channels is restored again to obtain a multi-layer feature representation. (4) License plate detection: Each layer of features is processed and calculated separately, and non-maximum suppression is performed on these results to obtain a unified final result of small target license plate detection.

2. The small target ship plate detection method for weak light environment according to claim 1 is characterized in that: In step (1), the pixel-by-pixel enhancement operation is implemented using a feature enhancement network, which consists of multiple serially connected pixel-by-pixel enhancement modules, each of which is composed of a series of "point-by-point convolution-depth-separable convolution-point-by-point convolution" structures; and layer scaling technology is used to promote model convergence by multiplying the feature map with a series of scaling factors.

3. The small target ship plate detection method for weak light environment according to claim 1 is characterized in that: In step (2), feature extraction is performed by sending each block into the embedding layer one by one, and mapping the pixels in the block into a high-dimensional feature vector. During the feature extraction process, each small block will be processed by a multi-layer transposed attention mechanism and a fully connected layer, and through a hierarchical structure, the image will be divided into multiple stages according to different resolutions. Each lower resolution stage performs a self-attention operation on its corresponding higher resolution stage to achieve information transfer and feature fusion between different levels.

4. The small target ship plate detection method for weak light environment according to claim 1 is characterized in that: In step (3), the high-dimensional features obtained by feature extraction are subjected to layer-by-layer upsampling operations to achieve the same scale as the low-dimensional features.

5. The method for detecting small target ship license plates in a weak light environment according to claim 1, characterized in that: In step (4), the multi-layer features are decoupled, each feature layer is processed separately, the results of each layer are calculated separately, and then the results of features at different scales are restored to the unified input image scale.

6. The small target ship plate detection method for weak light environment according to claim 1 is characterized in that: The overall process uses Focal-EIoUloss as the loss function. The specific formula of the loss function is as follows: LFocal-EIoU=IoUγLEIoU Among them, γ is a hyperparameter used to control the curvature of the curve, c w and c h are the width and height of the minimum bounding box covering two boxes, and ρ(·) represents the Euclidean distance.

7. A small target ship plate detection system for low light environment, characterized in that: The system includes: Image restoration module, used to obtain low-light images of small target ship plates, extract feature representations of different scales, perform pixel-by-pixel enhancement operations, and combine the enhanced features with the original image; The feature extraction module is used to divide the restored image into a series of non-overlapping blocks, and perform feature extraction on each block to obtain multi-layer features of different scales; The multi-granularity feature interaction module is used to fuse the high-dimensional features and low-dimensional features extracted from the features, restore the number of channels of the merged features, and process them layer by layer. The obtained features are then fused and the number of channels is restored again to obtain a multi-layer feature representation. The license plate detection module is used to process and calculate each layer of features separately, perform non-maximum suppression on these results, and obtain a unified final result of small target license plate detection.

8. A small target ship license plate detection device for a weak light environment, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a small target ship license plate detection method for a weak light environment as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a small target ship license plate detection method for a weak light environment as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a small target ship license plate detection method for a weak light environment as described in any one of claims 1 to 6 is implemented.

Images