Multi-scale image target detection method for large-view-field small target
By segmenting the original large image into multi-scale small images and performing object detection, the problem of uncontrollable information loss in large field of view and small target scenarios is solved, and the detection capability and adaptability are improved.
Patent Information
- Application Number
- CN202411657790.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-03-11
AI Technical Summary
When processing input images, the object detection algorithm based on deep learning needs to cast the image to a specific size, resulting in uncontrollable loss of image data and information, making it difficult to adapt to scenes where the target imaging size changes greatly, especially the problem of large field of view and small object detection.
By segmenting the original large images into multi-scale small images and inputting these small images into the object detection algorithm for detection, the detection results are finally mapped back to the original large image, solving the information loss problem caused by the large difference between the input image size and the target imaging size.
The detection ability of large field of view small targets is improved, the target information loss caused by image size changes is avoided, and the adaptability of the target detection algorithm in large field of view small target scenarios is enhanced.
Smart Images
Figure CN119600263A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of artificial intelligence and image target detection. Background Art
[0002] Image object detection is an important task in the field of computer vision, and its purpose is to accurately locate and identify the object of interest in the image. With the rise of deep learning, image object detection technology has made significant progress. Deep learning-based methods such as Faster R-CNN, YOLO, SSD, etc. have become the mainstream methods in the field of object detection, solving many problems related to image recognition and visual applications.
[0003] The core of the deep learning-based image target detection algorithm is a trainable structure such as a convolutional neural network. Since neural networks generally have strict requirements on the dimension and length of input data, when processing input images, the target detection algorithm needs to force the image to be converted to a certain size, or convert the data to a certain size after layer-by-layer feature extraction, and then input the data into the classification layer to complete the final target classification and recognition. In specific implementation, the deep learning-based target detection algorithm generally limits the data size such as the dimension and length of the data by setting hyperparameters. Although this processing method solves the adaptability of the model to changes in image size and can also adapt to most common image target detection application scenarios, it is not adaptable enough to target detection scenarios with large changes in target imaging size, especially when the pixel size of the target after imaging is very different, such as ground remote sensing, high-altitude reconnaissance, long-distance detection imaging and other large field of view small target scenes, which will lead to serious algorithm generalization problems. In addition, it must be noted that large field of view and small target is a common problem in reality. As the field of view of the optical system changes, a target with a large physical size may become a small target, and a target with a small physical size may become a large target. In other words, large field of view and small target is a relative change, not an absolute concept.
[0004] The risk of forcibly transforming the input data size through hyperparameters is that when the input image is forcibly transformed to a certain size, the loss of image data and information is uncontrollable. If the original size of the input image is large, more data and information will be lost, and if the original size of the input image is small, less data and information will be lost; if the pixel size of the target after imaging is large, more data and information will be lost, and vice versa. It can be seen that when a model needs to detect different categories of targets at the same time, or the pixel size of the same target after imaging changes greatly, the difficulty of target detection caused by the uncontrollable loss of data and information caused by forced size transformation of the input image will become more and more prominent.
[0005] Although the target detection algorithm based on deep learning itself also takes into account different target types and scale changes after target imaging, such as using anchor frames of different sizes and aspect ratios, namely the multi-scale anchor frame method, in order not to excessively increase the computational complexity, the number of anchor frame sizes and aspect ratios used in target detection can only be limited, and it cannot solve the problem of detecting small targets in a large field of view well.
[0006] Therefore, although deep learning-based target detection methods, such as Faster R-CNN, R-FCN, YOLO and SSD, have become the mainstream general algorithms in the field, the deep learning-based target detection framework restricts the data dimensions such as data dimension and length during the final classification, which can easily lead to uncontrollable information loss of small-scale image targets. Mainstream deep learning-based image target detection algorithms such as Faster R-CNN, R-FCN, YOLO and SSD are generally only suitable for target detection of large or medium imaging sizes within the field of view, and are difficult to adapt to the widespread problem of small target detection in a large field of view in reality.
[0007] In general, it is difficult to directly apply the current mainstream deep learning-based algorithm framework to effectively solve the problem of target detection with large-scale variations in imaging scales in special fields such as the defense industry, military and civilian applications. There is an urgent need for an innovative method that can specifically solve the problem of small target detection in a large field of view.
[0008] Since the backend of the deep learning-based target detection algorithm must convert the data dimension to a fixed size when performing classification, which leads to uncontrollable loss of input image data and information, and thus the fundamental reason why the current mainstream target detection framework is difficult to effectively cope with scenes with drastic changes in the target imaging scale, it seems that the problem can be solved by breaking through this limitation. However, although technologies such as image pyramids and random pruning can allow neural networks to adapt to data inputs of different sizes or allow the connection mechanism within the neural network to change dynamically, the number of neurons in the output layer of the neural network and its internal structure cannot be changed at will, and it is currently difficult to propose an effective method to make the neural network structure flexible and variable. Therefore, at this stage, it is difficult to improve the deep learning-based target detection algorithm to adapt to the problem of detecting small targets in a large field of view by optimizing the internal structure of the neural network.
[0009] Through research, it is found that in addition to the direct correlation between the target detection and recognition effect and the algorithm model itself, the properties of the image to be detected will also affect the target detection effect. In addition to the well-known image quality related factors such as image clarity and contrast, the image size will also directly affect the target detection effect. Specifically, when the same target on an image is input into the target detection algorithm model with different size data, some sizes can accurately detect the target, while some sizes are missed. It can be seen that the relative size of the target imaging size and the data size of the image itself will also affect the target detection effect. The fundamental reason for this situation is the uncontrollable problem of image data and information loss caused by the forced transformation of the input data size analyzed in the previous article.
[0010] In view of this problem, since the neural network structure cannot dynamically change and adjust to meet the needs of detecting targets of different imaging sizes, the only way to solve this problem is to change the method of inputting image data into the target detection model. Because the image itself can be segmented and cut, based on the original image output by the optoelectronic device, if the original image is segmented into small images, the relative ratio of the image size to the target imaging size can be changed, thereby bringing more possibilities for detecting targets of different scales. On this basis, the mapping relationship between the detection results of the segmented small image and the original large image can be solved, so that the large image that is difficult to directly detect the target can be converted into a small image to realize the detection of small targets, thereby providing a feasible technical solution to the problem of detecting small targets in a large field of view.
[0011] Based on the above problems and analysis, the core technical idea of the present invention is to solve the problem from the data input to the target detection algorithm. For the problem of small target with large field of view, the original image is divided into small images at different scales and then input into the target detection algorithm model algorithm. After the small target is detected on the small image, the result is mapped back to the original large image. The alignment optimization of small target detection with large field of view on the original image can be achieved by small image detection. Summary of the invention
[0012] Aiming at the problem of target detection difficulty caused by the large difference between target scale and target imaging scale in reality, the present invention proposes and designs a multi-scale image target detection method for small targets with large field of view. On the basis of mainstream image target detection algorithms based on deep learning such as Faster R-CNN, YOLO, SSD, etc., the large-size image to be detected is divided into multi-scale small images and then sent to the target detection algorithm for detection processing. The problem of small target detection caused by uncontrollable image data and information loss when the current mainstream target detection algorithm forcibly transforms the input image into a certain size is solved, thereby improving the existing image target detection and recognition methods and enhancing the application capability for small target scenes with large field of view.
[0013] First, develop an image detection model based on deep learning. For the problem of target detection with small targets in a large field of view, the original image cannot be directly input into the convolutional neural network to train the model. Otherwise, the small-scale targets will be difficult to detect because the small target information will be drastically compressed. The correct approach is to set a suitable large image cutting method according to the imaging size of the small targets to be detected in different categories after obtaining the image data samples, and then use the small-size images containing the marking information as data samples to carry out model training and testing. Only by using this method can it be possible to train a target detection model with the ability to detect small targets;
[0014] Secondly, after training to obtain a target detection model based on deep learning with the ability to detect small targets, the key is to design a scientific and reasonable original image segmentation method. Considering that image target detection often requires the simultaneous detection of targets of different types and different imaging scales, there will be drastic changes and differences in the imaging scales of the same or different targets. For example, for an original image of 20000×15000 pixels, if the imaging scale of the A-type target is approximately 50×50 pixels, and the imaging scale of the B-type target is approximately 1000×1000 pixels, the 20000×15000 pixel original image is directly input into the target detection model. The detection of the A-type target can be completed with a high probability, while the B target is basically difficult to detect. Therefore, the present invention adopts a multi-scale image segmentation method, and sets different scale segmentations according to the scale characteristics of the imaging of different categories of targets to meet the detection needs of different categories or different scales of imaging targets. The selection and setting of the image segmentation scale and the number of scales should be set according to the actual difference in the imaging scale of the detection target, so as not to excessively increase the computational complexity and time overhead of the target detection and meet the detection of imaging targets of different scales. In addition, it is necessary to further explain that when performing image segmentation, an overlapping segmentation method should be adopted, such as dividing the original large image into small-sized images with a left-right and top-bottom overlap rate of 10%. This is to avoid the problem that the target is chopped up and difficult to detect. The setting of the overlap rate should be flexibly set according to the segmentation size and target imaging size.
[0015] Then, the segmented small images are input into the image target detection model in sequence. The image target detection model processes the small-size images according to the normal target detection process, so that small targets can be detected on the small-size images. The target information will not be seriously lost and it will not be difficult to correctly detect the target because the image size is much larger than the target imaging scale.
[0016] Finally, since the pixel coordinates of the original image may be related to external physical world information such as longitude, latitude, direction, and distance, when the original image is segmented into multi-scale small images, the mapping relationship between the pixel coordinates of the small image and the pixel coordinates of the original large image should be recorded. In this way, after the target is detected on the small image, the detected target pixel coordinate information on the small image can be quickly converted back to the pixel coordinates of the large image, thereby realizing small target detection on the original large image.
[0017] Through the above technical approach, we can solve the problem of insufficient detection capability of large field of view and small target in the current mainstream image target detection framework based on deep learning, and improve the ability of target detection algorithm to solve the problem of image target detection in reality. It is an effective improvement on the existing traditional target detection method for large field of view and small target detection scenario.
[0018] The innovation of the present invention lies in that by performing multi-scale segmentation on the image to be detected, the problem that the current mainstream target detection methods, when faced with target detection tasks with large scene and small target characteristics, the input data size is forcibly converted, resulting in uncontrollable loss of small target information and difficulty in detection is solved. The small target detection capability under a large field of view is improved, and through the pixel coordinate association between the original image and the small image, a rapid mapping conversion of the target detection result on the small image to the original large image can be achieved, achieving the same effect as the current general target detection method, and does not require additional cumbersome result processing. It is a practical method that has been proven in practice to be applicable to target detection in small target scenes with a large field of view.
[0019] In order to achieve the invention purpose of this application, this application adopts the following technical solutions:
[0020] The present invention provides a multi-scale image target detection method for a small target with a large field of view, wherein: it comprises the following steps:
[0021] (i) Obtaining images of small targets with large fields of view through optoelectronic systems carried by satellites, aircraft, drones, ships or ground station platforms;
[0022] (II) Determine the above small targets that need to be detected and identified
[0023] Use labelme or labelimg image annotation tools to annotate the above images, and cut out image samples containing small target features with appropriate image size. The size of the image samples should be selected so that the target information is not lost after the input target detection algorithm is forced to change its size.
[0024] (3) Create an image target detection dataset based on the ImageNet or coco dataset framework, and divide the above image target detection dataset into three parts: training set, test set and validation set;
[0025] (IV) Build a deep learning environment platform based on TensorFlow or PyTorch;
[0026] (V) Build a deep learning-based image target detection model based on Faster R-CNN, YOLO or SSD framework, use the image target detection dataset prepared in step (III), train the model on the deep learning environment platform of TensorFlow or PyTorch, and obtain an image target detection model whose target detection performance meets the application requirements;
[0027] (VI) Deploy the image object detection model in step (V) based on the TensorRT, ONNX or OpenVINO platform and encapsulate it into a flexibly callable computing API interface service;
[0028] (VII) Using the original large image multi-scale segmentation tool, the original large image is traversed for the first time by sliding the sliding window. The number, size and aspect ratio of the sliding windows are set according to the imaging scale of the target to be detected, and the obtained small image is input into the above-mentioned image target detection model for target detection, and the pixel position of the sliding window on the original large image is recorded;
[0029] (8) After a sliding window traversal is completed, the number, size and aspect ratio parameters of the sliding windows are changed, and step (7) is repeated until all the set sliding windows are used up, thereby obtaining target detection results on a series of small images;
[0030] (IX) According to the mapping relationship between the pixel coordinates of the small image and the pixel coordinates of the original large image, the target detection results on a series of small images are converted to the original large image, and the repeated detection results are eliminated to obtain the target detection results on the original large image, thereby completing the target detection of image data with large field of view and small target characteristics.
[0031] The multi-scale image target detection method for small targets with a large field of view of the present invention, wherein: the size and aspect ratio of the sliding window are determined according to the size and aspect ratio of the target to be detected.
[0032] The multi-scale image target detection method for small targets with large fields of view of the present invention comprises: the training set, the test set and the verification set accounting for 70%, 20% and 10% of the image target detection data set.
[0033] The present invention provides a multi-scale image target detection method for small targets with a large field of view, wherein: the sliding window slides from left to right and from top to bottom in an overlapping manner to traverse the original large image, and selects a small image with the same size as the sliding window.
[0034] The multi-scale image target detection method for small targets with a large field of view of the present invention has the following beneficial effects:
[0035] The present invention proposes a multi-scale image target detection method for small targets with a large field of view. By multi-scale segmentation of the original large image and the associated feedback of the small image target detection results to the large image coordinates, an effective method for solving the problem of small target detection under a large field of view is established. The method is suitable for target detection problems in many fields with large field of view and small target characteristics. It is a targeted optimization and innovative breakthrough of the current mainstream target detection method that is only suitable for the detection of medium and large target imaging sizes. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a working principle diagram of a multi-scale image target detection method for small targets with a large field of view according to the present invention. DETAILED DESCRIPTION
[0037] A multi-scale image target detection method for a small target with a large field of view of the present invention comprises the following steps:
[0038] (i) Obtaining images of small targets with large fields of view through optoelectronic systems carried by satellites, aircraft, drones, ships or ground station platforms;
[0039] (II) Determine the above small targets that need to be detected and identified
[0040] Use labelme or labelimg image annotation tools to annotate the above images, and cut out image samples containing small target features with appropriate image size. The size of the image samples should be selected so that the target information is not lost after the input target detection algorithm is forced to change its size.
[0041] (III) Create an image target detection dataset based on the ImageNet or coco dataset framework, and divide the above image target detection dataset into three parts: training set, test set and validation set. The training set, test set and validation set account for 70%, 20% and 10% of the image target detection dataset respectively.
[0042] (IV) Build a deep learning environment platform based on TensorFlow or PyTorch;
[0043] (V) Build a deep learning-based image target detection model based on Faster R-CNN, YOLO or SSD framework, use the image target detection dataset prepared in step (III), train the model on the deep learning environment platform of TensorFlow or PyTorch, and obtain an image target detection model whose target detection performance meets the application requirements;
[0044] (VI) Deploy the image object detection model in step (V) based on the TensorRT, ONNX or OpenVINO platform and encapsulate it into a flexibly callable computing API interface service;
[0045] (VII) Using the original large image multi-scale segmentation tool, the original large image is traversed for the first time by sliding the sliding window in a sliding manner. The sliding window is traversed from left to right and from top to bottom in an overlapping manner. A small image with the same size as the sliding window is selected. The size and aspect ratio of the sliding window are determined according to the size and aspect ratio of the target to be detected. The obtained small image is input into the above-mentioned image target detection model for target detection, and then the pixel position of the sliding window on the original large image is recorded;
[0046] (8) After a sliding window traversal is completed, the number, size and aspect ratio parameters of the sliding windows are changed, and step (7) is repeated until all the set sliding windows are used up, thereby obtaining target detection results on a series of small images;
[0047] (IX) According to the mapping relationship between the pixel coordinates of the small image and the pixel coordinates of the original large image, the target detection results on a series of small images are converted to the original large image, and the repeated detection results are eliminated to obtain the target detection results on the original large image, thereby completing the target detection of image data with large field of view and small target characteristics.
[0048] Based on the above technical implementation approach, the present invention proposes and designs a multi-scale image target detection method for small targets with a large field of view, which can specifically solve the problem of small target detection in the current mainstream target detection algorithm, and make up for the shortcomings of the current image target recognition and detection method through technical improvements.
[0049] The multi-scale image target detection method for small targets with a large field of view of the present invention has the following characteristics:
[0050] (1) The original large image is divided into multi-scale small images. By cutting the large image into small images, the problem of the large ratio between the input image size and the small target scale causing the data information of the small target to be severely compressed and unable to be detected can be avoided. Multi-scale segmentation can adapt to the situation where there are multiple imaging sizes in the image, and realize the effective detection of imaging targets of different scales.
[0051] (2) A repeated multi-scale segmentation method is used. When the original large image is segmented into small images, the left-right and top-bottom sliding is set to keep overlapping, so that the chopped objects can remain intact in adjacent images, avoiding the problem of difficult detection due to the image objects being chopped;
[0052] (3) The target detection results of the small image are associated with the coordinates of the large image and transmitted back. When the large image is cut into a small image, the pixel coordinate position of the small image on the original large image is recorded and retained, and an association model between the pixel coordinates of the small image and the pixel coordinates of the large image is established to ensure that the results detected on the small image can be easily mapped to the original large image, thereby achieving target detection results consistent with mainstream methods.
[0053] Based on the above technical features, a multi-scale image target detection method for small targets with a large field of view has extremely strong adaptability and detection capabilities for situations where there are multi-scale and multi-type targets in the image to be detected. It can be used to solve the problem of small target detection with a large field of view that is widely present in reality, and provides a convenient and practical method to make up for the shortcomings of the current mainstream image target detection algorithms.
[0054] The above is only one embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.
Claims
1. A multi-scale image target detection method for small targets in a large field of view, characterized by: It includes the following steps: (i) Obtaining images of small targets with large fields of view through optoelectronic systems carried by satellites, aircraft, drones, ships or ground station platforms; (II) Determine the above small targets that need to be detected and identified Use labelme or labelimg image annotation tools to annotate the above images, and cut out image samples containing small target features with appropriate image size. The size of the image samples should be selected so that the target information is not lost after the input target detection algorithm is forced to change its size. (3) Create an image target detection dataset based on the ImageNet or coco dataset framework, and divide the above image target detection dataset into three parts: training set, test set and validation set; (IV) Build a deep learning environment platform based on TensorFlow or PyTorch; (V) Build a deep learning-based image target detection model based on Faster R-CNN, YOLO or SSD framework, use the image target detection dataset prepared in step (III), train the model on the deep learning environment platform of TensorFlow or PyTorch, and obtain an image target detection model whose target detection performance meets the application requirements; (VI) Deploy the image object detection model in step (V) based on the TensorRT, ONNX or OpenVINO platform and encapsulate it into a flexibly callable computing API interface service; (VII) Using the original large image multi-scale segmentation tool, the original large image is traversed for the first time by sliding the sliding window. The number, size and aspect ratio of the sliding windows are set according to the imaging scale of the target to be detected, and the obtained small image is input into the above-mentioned image target detection model for target detection, and the pixel position of the sliding window on the original large image is recorded; (8) After a sliding window traversal is completed, the number, size and aspect ratio parameters of the sliding windows are changed, and step (7) is repeated until all the set sliding windows are used up, thereby obtaining target detection results on a series of small images; (IX) According to the mapping relationship between the pixel coordinates of the small image and the pixel coordinates of the original large image, the target detection results on a series of small images are converted to the original large image, and the repeated detection results are eliminated to obtain the target detection results on the original large image, thereby completing the target detection of image data with large field of view and small target characteristics.
2. The multi-scale image target detection method for small targets with a large field of view as claimed in claim 1, characterized in that: The size and aspect ratio of the sliding window are determined according to the size and aspect ratio of the target to be detected.
3. The multi-scale image target detection method for small targets with a large field of view as claimed in claim 2, characterized in that: The training set, test set and validation set account for 70%, 20% and 10% of the image object detection dataset.
4. The multi-scale image target detection method for small targets with a large field of view as claimed in claim 3, characterized in that: The sliding window slides in a manner of traversing the original large image from left to right and from top to bottom in an overlapping manner, and selects a small image with the same size as the sliding window.