Wide field of view target search device, system and method in degraded visual environment

Abstract

The application discloses a device, system and method for searching a wide field of view range target in a degenerated visual environment, and belongs to the technical field of machine vision. In order to solve the problem that the prior art cannot realize wide range and small target searching and identification in a degenerated visual environment, the device comprises a multi-view visual system, a connecting structure and a two-axis holder; the multi-view visual system comprises an infrared camera, a long-focus camera and a wide-angle camera; the multi-view visual system is mounted on the two-axis holder through the connecting structure, and the two-axis holder realizes two degrees of freedom rotation; the center camera on the multi-view visual system is the long-focus camera, and the infrared camera and the wide-angle camera are arranged on the two sides respectively, and the three cameras are distributed on the same holder. The system comprises a controller, an image processor and a set of the aforementioned device. The method realizes small target searching and position and posture estimation in a wide field of view range in a degenerated visual environment through image fusion, fusion evaluation, feature detection, attention switching and target identification processes.

Description

Technical Field

[0001] This invention relates to a wide field-of-view target search system and method, belonging to the field of machine vision technology. Background Technology

[0002] With the development of drone technology, drones are becoming increasingly powerful and playing a significant role in both military and civilian fields. However, currently, drones have limited autonomy, and in complex environments, human intervention is still required for operation. They lack the ability to autonomously perceive the operational or mission environment. Therefore, improving operators' situational awareness, utilizing sensors for autonomous situational perception, and reducing operator workload are key research issues.

[0003] Visual sensors are characterized by high precision, high resolution, high sensitivity, small size, and light weight, making them typical sensors used in reconnaissance and strike systems and "fire-and-forget" equipment. However, visual sensors are sensitive to changes in lighting conditions and struggle to simultaneously meet the demands of wide-field detection and high-precision identification. To adapt to all-weather, large-area operational or mission requirements, higher demands are placed on sensors in weapon systems and mobile systems, posing new challenges to visual imaging technology in continuous target tracking and accurate identification.

[0004] Currently, commonly used visual sensors include visible light sensors and infrared sensors. Visible light images typically have high spatial resolution, rich detail, and clear contrast. However, visible light images are easily affected by adverse conditions, such as insufficient lighting, fog, or other severe weather, a phenomenon known as degraded visual environment (DVE). Infrared images are more resistant to these disturbances, but their resolution is relatively low and their texture information is not rich enough. Therefore, it is difficult to provide highly robust image information in degraded visual environments using a single sensor, making multi-view sensor fusion essential. Summary of the Invention

[0005] The present invention aims to solve the problem that existing technologies cannot achieve large-scale, small-target search and identification in degraded visual environments.

[0006] A wide field-of-view target search device for degraded visual environments, the device comprising a multi-view vision system, a connecting structure, and a two-axis gimbal; wherein the multi-view vision system includes an infrared camera, a telephoto camera, and a wide-angle camera;

[0007] The multi-view vision system is mounted on a two-axis gimbal via a connecting structure, and the two-axis gimbal enables two degrees of freedom of rotation.

[0008] The central camera in the multi-view vision system is a telephoto camera, flanked by an infrared camera and a wide-angle camera, with all three cameras distributed on the same gimbal. Preferably, the three cameras are arranged in a straight line on the same gimbal.

[0009] A wide field-of-view target search system for degraded visual environments, the system comprising a controller, an image processor, and a wide field-of-view target search device for degraded visual environments;

[0010] The controller controls the wide-angle camera and infrared camera to acquire images and transmits the acquired images to the image processor;

[0011] The image processor performs image fusion and feature detection processing, and then transmits the target position information to the controller; the image processor is also used to process images captured by the telephoto camera to obtain target type and pose information;

[0012] The controller controls the rotation of the motion motor, which in turn drives the camera and position sensor to move; the controller is also used to control the shooting of the telephoto camera.

[0013] A method for wide field-of-view target search in degraded visual environments includes the following steps:

[0014] S1. For infrared and visible light images, decompose them and then use a fusion strategy to fuse them;

[0015] The infrared images and visible light images were obtained using an infrared camera and a wide-angle camera, respectively.

[0016] S2. Perform feature detection on the fused image to obtain feature description;

[0017] S3. The fused image is evaluated and filtered based on a feature-oriented image evaluation method. The filtered fused image is obtained based on infrared and visible light images and is denoted as a wide field-of-view fused image. The feature-rich regions in the wide field-of-view fused image are recorded as regions of interest.

[0018] S4. Employing an attention switching strategy, after obtaining the region of interest in the wide field-of-view fused image, the region's location is recorded. The wide field-of-view target search system under degraded visual environment is manipulated to ensure that the region of interest is located in the center of the field of view. A telephoto camera is used to acquire the image corresponding to the region of interest, and the image acquired by the telephoto camera is a high-resolution image.

[0019] S5. Use deep learning to identify targets in the images corresponding to the telephoto camera to obtain information such as target type and location; repeat the above steps until all tasks are completed.

[0020] Furthermore, in the process of decomposing and then fusing infrared and visible light images as described in S1, a multi-scale low-rank fusion method is used to divide the image into low-rank blocks of different sizes, and different fusion strategies are used to fuse low-rank images at different levels.

[0021] Furthermore, a multi-scale low-rank fusion method is used to divide the image into low-rank blocks of different sizes. The process of fusing low-rank images at different levels using different fusion strategies includes the following steps:

[0022] (1) Take the infrared image and the visible light image as the images to be decomposed, and divide the image Y to be decomposed into an image matrix composed of low-rank blocks of different sizes;

[0023] (2) The feature maps of different sizes are fused using different fusion strategies. For images whose feature map length / width is less than 50% of the original image scale, the summation fusion strategy is used; for images whose length / width is greater than or equal to 50% of the original image scale, the maximum value fusion strategy is used, and finally, multiple fusion results under different fusion strategies are obtained. The generated feature maps of different sizes are superimposed and fused to obtain a new large image, which is the fusion result, and then a series of different feature maps for the same input are obtained.

[0024] The method of fusing this part of the image based on a summation fusion strategy is represented as follows:

[0025]

[0026] In the formula, This represents the pixel value at the corresponding position in the fused image. , These represent the pixel values ​​at corresponding positions in the infrared salient image (decomposed low-rank map) and the visible light salient image (decomposed low-rank map), respectively.

[0027] The maximum value fusion rule fuses this part of the decomposed image, and the expression is:

[0028]

[0029] In the formula, The image fusion weights are selected based on the image quality evaluation method.

[0030] Furthermore, the process of performing feature detection on the fused image to obtain a feature description, as described in S2, includes the following steps:

[0031] First, the image to be fused is segmented, and each segmented region is denoted as a small region. The geometric center of each small region is O, and the intensity centroid C point within the small region is described as follows:

[0032]

[0033] In the formula, The pixel value at (x, y) in the image;

[0034] Establish an OC vector. Within each small region, using the OC vector as the starting boundary, divide the region into N equal parts. Define the geometric center of each small part as O. subi In each small portion, the intensity centroid point C is selected again. sub Repeated calculations yielded O subi C subi Vectors; and calculate the OC vector and O subi C subi The angle θ between vectors (i=1,2,…,N) i :

[0035]

[0036] θ i Perform binary encoding: The average angle is divided into Divide into equal parts, each represented by a binary code, based on the included angle θ. i correspond One of the equal parts, the corresponding binary code is taken as θ. i binary encoding;

[0037] For a small region, there are N equal parts. Each small region yields a vector of size 1xN, which is a feature description.

[0038] A 1xN vector corresponding to a small region is used as the descriptive feature of that region. Each image can obtain multiple 1xN vector feature descriptions. For each image to be feature extracted, a brute-force search method is used for image feature matching. At the same time, the feature matching results of the fused image obtained by the brute-force search method are weighted with those of the infrared and visible light images to determine the feature robustness and matching accuracy of the fused image.

[0039] Furthermore, the feature-based image evaluation method described in S3 evaluates the fused image. During the process of screening the fused image, the evaluation indicators include feature abundance. The feature abundance includes feature robustness and matching accuracy. The feature robustness and matching accuracy in feature abundance are the feature robustness and matching accuracy of the fused image corresponding to the feature description determined in step S2.

[0040] Furthermore, the feature-based image evaluation method described in S3 evaluates the fused image, and the evaluation indicators during the screening of fused images also include... Structural similarity, information entropy, and spatial frequency.

[0041] Furthermore, the specific process of S4 includes the following steps:

[0042] Based on the position (x, y) of the center of the region of interest in the fused image, the two-axis gimbal is driven to rotate, so that the center of the region of interest is always at the center of the regional image. The pitch motion relies on the pitch degree of freedom motor and the rotating shaft, and the yaw motion relies on the yaw degree of freedom motor and the rotating shaft.

[0043] Furthermore, the process described in S5 of using deep learning to identify targets in high-resolution images and obtain information such as target type and location includes the following steps:

[0044] Target recognition is performed using a Siamese network, which consists of convolutional layers, pooling layers, and fully connected layers.

[0045] Using images captured by a telephoto camera as input to a neural network, features are extracted from the images through convolutional and pooling layers, and then integrated through a fully connected layer to obtain an estimate of the target pose, p. These are estimates for six degrees of freedom, where For three-dimensional position coordinates, Yaw angle For looking up and down, This is the roll angle.

[0046] Beneficial effects:

[0047] The wide field-of-view target search device and system proposed in this invention for degraded visual environments have the following beneficial effects:

[0048] (1) In order to reduce or eliminate accidents of UAVs in degraded visual environments and improve situational awareness, an enhanced synthetic vision system that integrates infrared and visible light cameras is proposed, taking advantage of the strong anti-interference capability of infrared images and the rich information of visible light images.

[0049] (2) To address the problem that a single sensor cannot simultaneously meet the requirements of image resolution and detection field of view, a method of working together between a wide-angle camera and a telephoto camera is proposed. An attention switching mechanism and an intelligent image processing algorithm are used to solve the problem of balancing both image resolution and detection field of view.

[0050] The wide field-of-view target search method proposed in this invention for degraded visual environments has the following beneficial effects:

[0051] (1) The multi-scale low-rank decomposition method is adopted, which fully considers the image characteristics of each decomposition layer. Fusion rules are designed for different levels of saliency maps and global low-rank maps, and feature abundance evaluation criteria are introduced to finally obtain a fused image with high contrast and clear texture.

[0052] (2) By applying the attention switching mechanism, feature detection is first performed to obtain the region of interest, and then a high-resolution image of the region of interest is obtained, and finally the small target recognition task under a wide field of view is completed.

[0053] This invention proposes a wide field-of-view target search method for degraded visual environments. It employs a multi-scale low-rank image fusion method to fuse infrared and visible light images, and uses a fusion image evaluation method incorporating feature abundance to select high-contrast, texture-clear fusion images. An attention switching mechanism is applied to perform feature detection on a large area of ​​low-resolution images to obtain regions of interest (ROIs). Then, focusing on the ROI yields a local high-resolution image, on which target identification is performed. This invention provides a wide field-of-view target search method for degraded visual environments, addressing the practical problem of searching and identifying large-scale, small targets in such environments. Attached Figure Description

[0054] Figure 1 This is a framework diagram of the wide field-of-view target search method under degraded visual environment of the present invention;

[0055] Figure 2 This is a schematic diagram of the multi-scale low-rank decomposition model of the present invention;

[0056] Figure 3 This is a flowchart of the attention switching method of the present invention;

[0057] Figure 4 This is a schematic diagram of a multi-view fusion target recognition system; wherein, 1 is the multi-view vision system, 2 is the connecting structure, and 3 is the two-axis gimbal.

[0058] Figure 5 This is a schematic diagram of a multi-view vision system; it includes an infrared camera 1-1, a telephoto camera 1-2, and a wide-angle camera 1-3.

[0059] Figure 6 This is a topology diagram of the multi-view fusion target recognition system of the present invention;

[0060] Figure 7 This is a flowchart illustrating the multi-view fusion target recognition system of the present invention.

[0061] Figure 8 This is a schematic diagram of the input image partitioning when N is 4.

[0062] Figure 9 This is a schematic diagram of the components of a neural network loss function. Detailed Implementation

[0063] Specific implementation method one: Combining Figures 4 to 5 This implementation method is described below.

[0064] This embodiment is a target search device with a wide field of view in a degraded visual environment. The device includes a multi-view vision system 1, a connecting structure 2, and a two-axis gimbal 3. The multi-view vision system 1 includes an infrared camera 1-1, a telephoto camera 1-2, and a wide-angle camera 1-3.

[0065] The multi-view vision system 1 is mounted on a two-axis gimbal 3 by a connecting structure 2, and the two-axis gimbal 3 achieves two degrees of freedom of rotation.

[0066] The central camera in the multi-view vision system 1 is a telephoto camera 1-2, flanked by an infrared camera 1-1 and a wide-angle camera 1-3. All three cameras are positioned on the same pan-tilt unit, preferably arranged in a straight line on the same pan-tilt unit. This arrangement not only facilitates image fusion but also ensures the identification of the region of interest and is beneficial for processing by the telephoto camera, etc. In practice, this arrangement can include, but is not limited to, the three cameras arranged in a straight line on the same pan-tilt unit. The wide-angle camera has a large field of view but a relatively low resolution; the telephoto camera has a small field of view but a high relative resolution; and the infrared camera is sensitive to infrared features.

[0067] While the device appears simple in structure, it was actually designed based on biomimetic research. In particular, the wide-angle and infrared cameras simulate the peripheral vision effect of animal visual systems, enabling rapid searching of dynamic and suspicious targets over a wide area. The infrared camera has imaging capabilities in the infrared spectrum, enhancing the target image using infrared information when lighting conditions are unfavorable or the target's infrared characteristics are obvious. The long-focus staring camera simulates the staring and fine recognition of targets by animal visual systems, enabling the identification of target types before target locking and the fine recognition of target posture, shape, and other features after target locking. Therefore, the wide-field-of-view target search device for degraded visual environments of this invention is essentially a multi-view fusion target recognition system based on biomimetic vision. Thus, although the device appears simple in structure, this structural arrangement is not found in existing technologies, which do not consider biomimetic principles in designing multi-view vision systems.

[0068] Furthermore, although the present invention appears to have a simple structure, it should be noted that existing methods for target search are based on monocular or binocular cameras. Even with binocular cameras, the types of binocular cameras are the same (basically visible light image cameras), and they do not take into account bionics when setting up the camera device. More accurately, they are multi-view vision systems. Because existing technologies do not consider bionics at all, they do not consider the use of telephoto, infrared, and wide-angle cameras in combination, let alone the specific positioning of these cameras to achieve bionic functional simulation.

[0069] When the system starts working, it first controls the infrared camera 1-1 and the wide-angle camera 1-3 to acquire images. The image processor then obtains the image fusion and feature detection results, finds the region with a large number of features (region of interest), and transmits it to the controller. Then, the controller uses the obtained processing results to calculate and control the pitch and yaw motors. The position information returned in real time by the position sensor ensures that the motors move in a closed loop to the designated position, so that the region of interest is located in the center of the wide field of view fused image.

[0070] Then, the telephoto camera is controlled to acquire images, obtaining high-resolution images of the region of interest containing the suspected target. Target identification is then performed based on the high-resolution images to obtain information such as the target's category and pose.

[0071] The above completes the task of searching and recognizing small targets with a large field of view in a degraded visual environment.

[0072] The wide field-of-view target search device proposed in this invention for degraded visual environments has the following advantages over existing technologies:

[0073] (1) In order to reduce or eliminate aircraft accidents in degraded visual environments and improve situational awareness, an enhanced synthetic vision system that integrates infrared and visible light cameras is proposed, utilizing the strong anti-interference capability of infrared images and the rich information of visible light images.

[0074] (2) To address the problem that a single sensor cannot simultaneously meet the requirements of image resolution and detection field of view, a method of working together between a wide-angle camera and a telephoto camera is proposed. An attention switching mechanism and an intelligent image processing algorithm are used to solve the problem of balancing both image resolution and detection field of view.

[0075] Specific Implementation Method Two: Combining Figures 6 to 7 This implementation method is described below.

[0076] This embodiment is a wide field-of-view target search system under degraded visual environment. The system includes a controller, an image processor, and a set of wide field-of-view target search devices under degraded visual environment.

[0077] like Figure 7 As shown, the controller controls the motor movement and triggers image acquisition from the infrared camera, wide-angle camera, and telephoto camera, and also collects data from the position sensor. The camera images are transmitted to the image processor, which then transmits the processing results to the controller, which generates corresponding instructions.

[0078] More specifically,

[0079] First, the controller controls the wide-angle camera and infrared camera to acquire images. The acquired images are transmitted to the image processor. After image fusion and feature detection, the target position information is transmitted to the controller. The controller controls the motion motor to rotate, and the rotation of the motor drives the camera and position sensor to move. The controller collects position sensor data to perform closed-loop position control, so that the feature-rich region of interest image is located in the center of the sensor's field of view. Then, the controller controls the telephoto camera to take pictures and obtain a high-resolution image of the region of interest. The image processor then obtains the target type and pose information.

[0080] Infrared camera 1-1 and wide-angle camera 1-2 capture images respectively. The infrared and visible light images are fused together. The fused image is used for target search. If a region with rich features is identified as a suspicious target, the image miss distance of the suspicious target region is calculated and the servo gimbal 3 is rotated to center the suspicious target region in the image, thus completing the search and localization of the suspicious target. The telephoto camera 1-3 is then switched to image the target region, capturing the suspected target in the center of the field of view. Using richer features, the target is further identified through deep learning. Once confirmed as the target of this mission, the wide-angle camera locks onto and tracks the target. If the target tracking is normal, the target is located in the center of the wide-angle camera's field of view, which is also within the telephoto camera's field of view. The telephoto camera outputs the pose estimation result in real time. If the suspected target is not the target of this mission, the target lock fails, or the target tracking fails during the tracking process, the target search is restarted.

[0081] In summary, the wide field-of-view target search system under degraded visual conditions can obtain stable images and complete large-area search and small target identification tasks.

[0082] The wide field-of-view target search system proposed in this invention for degraded visual environments has the following advantages over existing technologies:

[0083] (1) In order to reduce or eliminate aircraft accidents in the DVE environment and improve situational awareness, an enhanced synthetic vision system that combines infrared and visible light cameras is proposed, utilizing the strong anti-interference capability of infrared images and the rich information of visible light images.

[0084] (2) To address the problem that a single sensor cannot simultaneously meet the requirements of image resolution and detection field of view, a method of working together between a wide-angle camera and a telephoto camera is proposed. An attention switching mechanism and an intelligent image processing algorithm are used to solve the problem of balancing both image resolution and detection field of view.

[0085] The wide-field-of-view target search device proposed in this invention for degraded visual environments consists of an infrared camera, a telephoto camera, and a wide-angle camera. The infrared and wide-angle cameras fuse images to obtain an anti-interference image for wide-area target search. The telephoto camera performs further target identification and tracking in suspected target areas. This multi-view fusion system, based on biomimetic vision, can solve the practical problems of large-area, small-target search and identification in ground reconnaissance and navigation scenarios. Specific implementation method three:

[0087] This implementation method describes a wide field-of-view target search method under degraded visual conditions, such as... Figure 1 As shown, the process comprises five steps: image fusion, fusion evaluation, feature detection, attention switching, and target identification. These steps are executed sequentially to ultimately achieve a wide field-of-view target search function in degraded visual environments, completing both large-scale search and small target identification tasks. Image fusion combines infrared and visible light images using a multi-scale low-rank fusion method to obtain a fused image. Image evaluation introduces a feature abundance criterion to evaluate the fused image, resulting in a high-resolution, texture-clear fused image. Feature detection is performed on the fused image to identify feature-rich regions of interest (ROIs). Attention switching is then performed to acquire high-resolution images of the ROIs before target identification is performed.

[0088] Specifically, the wide field-of-view target search method in a degraded visual environment described in this embodiment includes the following steps:

[0089] S1. For infrared and visible light images, a multi-scale low-rank fusion method is used to divide the image into low-rank blocks of different sizes, including local low-rank maps and global low-rank maps; different fusion strategies are used to fuse low-rank maps at different levels.

[0090] The infrared image and the visible light image were obtained by infrared camera 1-1 and wide-angle camera 1-3, respectively.

[0091] S2. Perform feature detection on the fused image using a two-level "centroid-mass" feature detection method to obtain features, calculate the direction of feature points, and perform feature description.

[0092] S3. The fused image is evaluated and filtered based on a feature-oriented image evaluation method; and the feature-rich regions in the fused image are recorded as regions of interest.

[0093] S4. Employing an attention switching strategy, after obtaining the region of interest in the wide field-of-view fused image, the region's location is recorded. The wide field-of-view target search system under degraded visual conditions is manipulated to ensure that the region of interest is located in the center of the field of view; high-resolution images of the region of interest are acquired.

[0094] like Figure 3As shown, the present invention first performs feature detection on a wide field of view image, determines the feature distribution, obtains a feature-rich region of interest, and then switches attention to further image the region of interest to obtain a high-resolution image.

[0095] S5. Use deep learning to identify targets in the images corresponding to the telephoto camera to obtain information such as target type and location; repeat the above steps until all tasks are completed.

[0096] Among them, images obtained by telephoto cameras are generally larger than 1 million pixels per image. Compared with the wide-range images obtained by wide-angle cameras and infrared cameras, images obtained by telephoto cameras are high-resolution images.

[0097] In some embodiments, the image fusion process of infrared image and visible light image using the multi-scale low-rank decomposition method described in S1 includes the following steps:

[0098] like Figure 2 As shown, (1) the infrared image and the visible light image are used as the images to be decomposed, and the image Y to be decomposed is divided into an image matrix composed of low-rank blocks of different sizes;

[0099] (2) Different fusion strategies are used to fuse feature maps of different sizes. For images whose feature map length / width is less than 50% of the original image scale (small-scale feature map), the summation fusion strategy is used; for images whose length / width is greater than or equal to 50% of the original image scale (large-scale feature map), the maximum value fusion strategy is used, and finally, multiple fusion results under different fusion strategies are obtained.

[0100] This portion of the image is fused using a summation-based fusion strategy, and is represented as follows:

[0101]

[0102] In the formula: This represents the pixel value at the corresponding position in the fused image. , These represent the pixel values ​​at corresponding positions in the infrared salient image (decomposed low-rank map) and the visible light salient image (decomposed low-rank map), respectively.

[0103] The maximum value fusion rule fuses this part of the decomposed image, and the expression is:

[0104]

[0105] In the formula, The image fusion weights are selected based on the image quality evaluation method.

[0106] The maximum absolute value weighted fusion method can better preserve the saliency information of large-scale salient parts and some texture information, and can better fuse complementary features of infrared and visible light images.

[0107] In fact, the above fusion process can be summarized as follows:

[0108] 1) Decompose into feature maps of different sizes

[0109] 2) Large feature maps are fused with large feature maps, and small feature maps are fused with small feature maps. Different strategies are used to obtain a series of fused feature maps of different sizes under the same scene and image input.

[0110] This embodiment selects the above-described fusion strategy, but the present invention includes, but is not limited to, the above-described strategy, as long as multiple fused images with different decomposition and fusion strategies are ultimately obtained.

[0111] 3) Overlay and fuse the feature maps of different sizes generated in 2) to obtain a new large image, which is the result of the fusion;

[0112] Multiple fusion strategies can be used for 2) and 3), resulting in a series of different feature maps for the same input.

[0113] The process described in S2, which involves feature detection on the fused image using a two-level "centroid-mass" feature detection method, obtaining features, calculating the orientation of feature points, and performing feature description, includes the following steps:

[0114] Features on the fused image are calculated using feature descriptors. For ease of description, all subsequent fused images will be referred to simply as images.

[0115] The corner feature detector employs a low-dimensional feature description method. First, the image to be fused is segmented, and each segmented region is denoted as a small region. The geometric center of each small region is O, and the intensity centroid C within the small region is described as follows:

[0116]

[0117] In the formula y is the pixel value at (x, y) in the image.

[0118] Establish an OC vector. Within each small region, using the OC vector as the starting boundary, divide the region into N equal parts. Define the geometric center of each small part as O. subi In each small portion, the intensity centroid point C is selected again. sub Repeated calculations yielded O subi C subi Vectors; and calculate the OC vector and O subi C subiThe angle θ between vectors (i=1,2,…,N) i :

[0119]

[0120] θ i Perform binary encoding: The average angle is divided into Divide into equal parts, each represented by a binary code, preferably using... The angle is divided into eight equal parts, each represented by eight binary codes: 000, 001, 010, 011, 100, 101, 110, and 111. This is based on the included angle θ. i correspond One of the equal parts, its corresponding binary code is taken as θ. i binary encoding;

[0121] For a small region, there are N equal parts. Each small region yields a vector of size 1xN, which is a feature description.

[0122] A 1xN vector corresponding to a small region is used as the descriptive feature. For each fused image, a brute-force search method is used to match the fused image with the features of the infrared and visible light images respectively, thus completing the feature extraction step. At the same time, the feature matching results of the fused image with the infrared and visible light images using the brute-force search method are weighted to determine the feature robustness and matching accuracy of the fused image.

[0123] like Figure 8 As shown, the following explanation uses N=4 as an example. First, the input image is divided into m×m regions, with the geometric center of the region at point O and the intensity centroid at point C, resulting in the OC vector. Starting from the OC vector, the image is divided into four sub-regions counterclockwise around point O. The geometric centroid O of each sub-region is then calculated. subi Strength centroid C subi Let i = 1, 2, 3, 4. Calculate the intersection of vector OC and vector O. subi C subi The vectors are used to form angles θ1, θ2, θ3, and θ4. These angles are then represented as 1x4 vectors using binary encoding, which serve as a description of the feature region.

[0124] Therefore, an image can be described by multiple 1xN vectors, and matching is performed in a block-based manner.

[0125] The process of evaluating the fused image using the feature-oriented image fusion quality evaluation method described in S3 includes the following steps:

[0126] To achieve robustness and stability in feature extraction, Based on objective evaluation methods such as structural similarity, information entropy, and spatial frequency, an index of image feature abundance is introduced to correlate the image fusion result with the quality of image feature extraction. It is an objective image fusion evaluation method, a pixel-level image fusion quality assessment index proposed by CS Xydeas, V. Petrović, which reflects the quality of visual information obtained from the fusion of input images.

[0127] Factors considered for feature abundance include, but are not limited to:

[0128] 1) Feature robustness: Feature robustness considers the stability of extracted features and the insensitivity of features to small deformations, including rotation, scaling, blurring, and illumination; that is, for the same target under different viewpoints, the extracted features of two images are unique and stable.

[0129] 2) Matching accuracy: Match different fused images with the infrared and visible light image features of the frame respectively, and consider the ratio of the number of correct matches to the number of corresponding features; that is, the fused image can fully reflect the features of the infrared and visible light images before fusion.

[0130] The features mentioned in the feature abundance are the descriptive features determined by the second-level "centroid-mass" in step S2. Feature robustness is considered by adding global rotation, scaling, blurring, illumination changes and other disturbances to the fused image to see if the number of descriptive features changes significantly. The matching accuracy is to perform feature matching by calculating the distance between the fused image and the infrared image and the visible light image respectively. The more successful matches, the higher the matching accuracy.

[0131] In order to improve the evaluation speed, in some embodiments, the feature abundance can be used directly for scoring. That is, based on the feature extraction results in step S2, the feature abundance factors are statistically analyzed, and the scores of each factor are weighted to obtain a comprehensive score. Based on the obtained comprehensive score, the one with the higher score is selected as the final fused image.

[0132] In some embodiments, the fused image can be selected from... Objective methods such as structural similarity, information entropy, and spatial frequency are used to score the images, while feature abundance is also scored. These scores are weighted and summed to obtain a comprehensive score, with the highest score selected as the final fused image. This comprehensive approach, considering multiple factors within the evaluation system, is suitable for low-texture images, effectively extracts features, and thus evaluates the quality of image fusion and yields the region of interest.

[0133] The specific process of S4 includes the following steps:

[0134] Next, based on the position (x, y) of the center of the region of interest in the fused image, the two-axis gimbal is driven to rotate, so that the center of the region of interest is always at the center position of the regional image. The pitch motion relies on the pitch degree of freedom motor and the rotating shaft, and the yaw motion relies on the yaw degree of freedom motor and the rotating shaft.

[0135] The process described in S5, which uses deep learning to identify targets in high-resolution images and obtain information such as target type and location, includes the following steps:

[0136] A Siamese network is used for target recognition. The Siamese network structure consists of convolutional layers, pooling layers, and fully connected layers. Images captured by a telephoto camera are used as input to the neural network. Features are extracted from the images through convolutional and pooling layers, and then the features are integrated through fully connected layers to obtain an estimate of the target pose, p. These are estimates for six degrees of freedom, where For three-dimensional position coordinates, Yaw angle For looking up and down, This is the roll angle.

[0137] For Siamese networks, several photos taken at different times are input into the same network to obtain the loss for a single frame and the loss for the difference between two frames. A new loss is defined for training.

[0138] like Figure 9 As shown, the loss function consists of two parts: the spatial pose constraint loss function and the single-frame pose loss function, i.e.:

[0139]

[0140] in, Weight value, Let be the spatial pose constraint loss function. This is the pose loss function for a single frame.

[0141] The single-frame pose loss function is:

[0142]

[0143] in, This is the estimated six-DOF pose value at the current moment. This represents the actual pose value of the six degrees of freedom at the current moment.

[0144] The spatial pose constraint loss function is:

[0145]

[0146] in, The value of the six-DOF pose estimation for a single frame at time t. The value of the six-DOF pose estimation for a single frame at time t-1. The value represents the six-DOF pose estimate for a single frame at time t-2.

[0147] During the training phase, a virtual simulation system is used to acquire continuous frame data images and corresponding pose labels. A Siamese convolutional neural network structure is employed, and training is performed according to a defined loss function. During the usage phase, the input is a continuous frame sequence of images, and the output is a six-degree-of-freedom pose result.

[0148] The above completes the entire process of image fusion, fusion evaluation, target search, attention switching, and target recognition, enabling the search and recognition of small targets with a large field of view in degraded visual environments.

[0149] The wide field-of-view target search method proposed in this invention for degraded visual environments has the following advantages over existing technologies:

[0150] (1) The multi-scale low-rank decomposition method is adopted, which fully considers the image characteristics of each decomposition layer. Fusion rules are designed for different levels of saliency maps and global low-rank maps, and feature abundance evaluation criteria are introduced to finally obtain a fused image with high contrast and clear texture.

[0151] (2) By applying the attention switching mechanism, feature detection is first performed to obtain the region of interest, and then a high-resolution image of the region of interest is obtained, and finally the small target recognition task under a wide field of view is completed.

[0152] This invention proposes a wide field-of-view target search method for degraded visual environments. It employs a multi-scale low-rank image fusion method to fuse infrared and visible light images, and uses a fusion image evaluation method incorporating feature abundance to select high-contrast, texture-clear fusion images. An attention switching mechanism is applied to perform feature detection on a large area of ​​low-resolution images to obtain regions of interest (ROIs). Then, focusing on the ROI yields a local high-resolution image, on which target identification is performed. This invention provides a wide field-of-view target search method for degraded visual environments, addressing the practical problem of searching and identifying large-scale, small targets in such environments.

[0153] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for wide field of view target search in degraded visual environments, characterized in that, Includes the following steps: S1. For infrared and visible light images, decompose them and then use a fusion strategy to fuse them; The infrared images and visible light images were obtained using an infrared camera and a wide-angle camera, respectively. For infrared and visible light images, during the decomposition and subsequent fusion process, a multi-scale low-rank fusion method is employed to divide the image into low-rank blocks of different sizes. Different fusion strategies are used for low-rank images at different levels; the process includes the following steps: (1) Take infrared images and visible light images as images to be decomposed, and divide the images to be decomposed into an image matrix composed of low-rank blocks of different sizes; (2) The feature maps of different sizes are fused using different fusion strategies. For images whose feature map length / width is less than 50% of the original image scale, the summation fusion strategy is used; for images whose length / width is greater than or equal to 50% of the original image scale, the maximum value fusion strategy is used, and finally, multiple fusion results under different fusion strategies are obtained. The generated feature maps of different sizes are superimposed and fused to obtain a new large image, which is the fusion result, and then a series of different feature maps for the same input are obtained. The method of fusing this part of the image based on a summation fusion strategy is represented as follows: In the formula, This represents the pixel value at the corresponding position in the fused image. , These represent the pixel values ​​at corresponding positions in the infrared salient image and the visible light salient image, respectively. The infrared salient image is a low-rank graph of the infrared image decomposition, and the visible light salient image is a low-rank graph of the visible light image decomposition. The maximum value fusion rule fuses this part of the decomposed image, and the expression is: In the formula, is an image fusion weight guided by a fusion image quality evaluation method; S2. Perform feature detection on the fused image to obtain a feature description; the process of performing feature detection on the fused image to obtain a feature description includes the following steps: First, the fused image is segmented, and each segmented region is denoted as a small region. The geometric center of each small region is O, and the intensity centroid C point within the small region is described as follows: wherein is the pixel value at image (x, y); Establish an OC vector. Within each small region, using the OC vector as the starting boundary, divide the region into N equal parts. Define the geometric center of each small part as O. subi In each small portion, the intensity centroid point C is selected again. sub Repeated calculations yielded O subi C subi Vectors; and calculate the OC vector and O subi C subi The angle θ between vectors i : θ i Perform binary encoding: The average angle is divided into Divide into equal parts, each represented by a binary code, based on the included angle θ. i correspond One of the equal parts, the corresponding binary code is taken as θ. i binary encoding; For a small region, there are N equal parts. Each small region yields a vector of size 1xN, which is a feature description. A 1xN vector corresponding to a small region is used as the descriptive feature of that region. Each image can obtain multiple 1xN vector feature descriptions. For each image to be feature extracted, a brute-force search method is used for image feature matching. At the same time, the brute-force search method is used to weight the feature matching results of the fused image with those of infrared and visible light images, thereby determining the feature robustness and matching accuracy of the fused image. S3. Evaluate the fused image based on feature robustness and matching accuracy, and filter the fused images. The filtered fused images are obtained based on infrared and visible light images and are denoted as wide field-of-view fused images. The feature-rich regions in the wide field-of-view fused images are recorded as regions of interest. S4. Employing an attention switching strategy, after obtaining the region of interest in the wide field-of-view fused image, the region's location is recorded. The wide field-of-view target search system under degraded visual environment is manipulated to ensure that the region of interest is located in the center of the field of view. A telephoto camera is used to acquire the image corresponding to the region of interest, and the image acquired by the telephoto camera is a high-resolution image. S5. Use deep learning to identify targets in the images corresponding to the telephoto camera to obtain target type and location information; repeat the above steps until all tasks are completed.

2. The method of claim 1, wherein, The process of evaluating and screening the fusion image includes the following indexes structure similarity, information entropy, and spatial frequency.

3. The method according to claim 1 or 2, wherein, The specific process of S4 includes the following steps: Based on the position (x, y) of the center of the region of interest in the fused image, the two-axis gimbal is driven to rotate, so that the center of the region of interest is always at the center of the regional image. The pitch motion relies on the pitch degree of freedom motor and the rotating shaft, and the yaw motion relies on the yaw degree of freedom motor and the rotating shaft.

4. The method of claim 3, wherein, The process described in S5, which uses deep learning to identify targets in high-resolution images and obtain target type and location information, includes the following steps: Target recognition is performed using a Siamese network, which consists of convolutional layers, pooling layers, and fully connected layers. Using images captured by a telephoto camera as input to a neural network, features are extracted from the images through convolutional and pooling layers, and then integrated through a fully connected layer to obtain an estimate of the target pose, p. These are estimates for six degrees of freedom, where For three-dimensional position coordinates, Yaw angle For looking up and down, This refers to the roll angle; For Siamese networks, several photos taken at different times are input into the same network to obtain the loss for a single frame and the loss for the difference between two frames. A new loss function is defined for training. The loss function consists of two parts: a spatial pose constraint loss function and a single-frame pose loss function, i.e.: wherein, weight values, for the spatial pose constraint loss function, is a single-frame pose loss function; The single-frame pose loss function is: wherein, is a six-degree-of-freedom pose estimate value at the current time, is a six-degree-of-freedom pose actual value at the current time; The spatial pose constraint loss function is: in, The value of the six-DOF pose estimation for a single frame at time t. The value of the six-DOF pose estimation for a single frame at time t-1. The value represents the six-DOF pose estimate for a single frame at time t-2.

Images