Multimode compound eye sensing method and device for complex degradation environment

Abstract

The invention relates to the field of computer vision, in particular to a multi-mode compound eye sensing method and device for a complex degradation environment. The method comprises the following steps: acquiring multiple groups of images in a complex degradation environment through multimode compound eye acquisition equipment, and inputting the images into a trained feature point prediction model to extract key feature point information of a visible light image and an infrared image; and generating a visible light spliced image and an infrared spliced image according to a nearest neighbor matching technology and the key feature point information of the visible light image and the infrared image, and inputting the visible light spliced image and the infrared spliced image into the constructed multimode sensing detection network for target detection to obtain a multimode sensing detection result. According to the method, a deep learning algorithm, a nearest neighbor matching technology and an image splicing technology are combined, and target detection perception tasks of visible light and infrared modes in a multimode compound eye sensor are realized by constructing a feature point prediction model and a multimode perception detection network.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-mode compound eye perception method and device for complex degraded environments. Background Art

[0002] With the continuous development of science and technology and the progress of society, computer vision technology plays an increasingly important role in various fields. In particular, traditional visual algorithms face many challenges in perception and recognition tasks in complex degraded environments. For example, in the fields of security monitoring, military reconnaissance, and environmental monitoring, due to the influence of complex degraded environments such as lighting conditions, weather changes, target surface characteristics, and limited viewing angles, traditional visual perception methods often fail to meet actual needs, resulting in low accuracy and robustness in target detection and recognition.

[0003] Current research on compound eye perception is limited to a single visible light modality, which makes it difficult to cope with perception tasks in complex degraded environments with weak light, dim light, or even no light. Most studies on compound eye feature point prediction use methods based on manually designed features, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded UpRobust Features), and Harris corner detection. However, these methods are too sensitive to scene changes such as changes in illumination, scale, and perspective, and have difficulty processing large-scale data and high-dimensional features. At the same time, existing deep learning-based target detection methods have the problem of high computational complexity in the application process of multi-modal stitched images, and are not suitable for multi-modal compound eye image target detection tasks. Summary of the invention

[0004] In order to solve the technical problem that the traditional visual perception method often cannot meet the actual needs due to the influence of complex degradation environment such as lighting conditions, weather changes, target surface characteristics and limited viewing angle range, resulting in low accuracy and robustness of target detection and recognition, the embodiment of the present invention provides a multi-mode compound eye perception method and device for complex degradation environment. The technical solution is as follows:

[0005] On the one hand, a multi-mode compound eye sensing method for a complex degraded environment is provided, the method being implemented by a multi-mode compound eye sensing device, the method comprising:

[0006] S1. Acquire multiple sets of images in a complex degraded environment through a multi-mode compound eye acquisition device; wherein each set of images in the multiple sets of images includes a visible light image and an infrared image.

[0007] S2. Input multiple groups of images into the trained feature point prediction model to extract key feature point information in the visible light image and key feature point information in the infrared image.

[0008] S3. Generate a visible light stitched image and an infrared stitched image according to the nearest neighbor matching technology, key feature point information in the visible light image, and key feature point information in the infrared image.

[0009] S4. Input the visible light stitching image and the infrared stitching image into the constructed multi-mode perception detection network for target detection to obtain the multi-mode perception detection result.

[0010] Optionally, the training process of the feature point prediction model in S2 includes:

[0011] S21. Acquire a visible light sample image.

[0012] S22, performing three convolution operations on the visible light sample image to obtain a first convolution feature map, a second convolution feature map, and a third convolution feature map.

[0013] S23, performing deconvolution operations on the first convolution feature map, the second convolution feature map, and the third convolution feature map, respectively, to obtain a first deconvolution image, a second deconvolution image, and a third deconvolution image.

[0014] S24, performing feature fusion on the first deconvolution image, the second deconvolution image, and the third deconvolution image to obtain a fused feature map.

[0015] S25. Input the fused feature map into the maximum pooling layer to obtain the maximum pooling layer output.

[0016] S26. Input the maximum pooling layer output to the bilinear interpolation layer to obtain the bilinear interpolation layer output.

[0017] S27. Input the output of the bilinear interpolation layer into the fully connected layer to obtain the key feature point information in the visible light sample image.

[0018] S28. Train the feature point prediction model according to the key feature point information in the visible light sample image to obtain a trained feature point prediction model.

[0019] Optionally, generating a visible light stitched image and an infrared stitched image according to the nearest neighbor matching technology, key feature point information in the visible light image, and key feature point information in the infrared image in S3 includes:

[0020] S31 . Acquire feature points of adjacent visible light images, and use a nearest neighbor matching technique to match the acquired feature points to obtain a plurality of matched visible light image feature points.

[0021] S32, acquiring feature points of adjacent infrared images, and matching the acquired feature points using a nearest neighbor matching technique to obtain a plurality of matched infrared image feature points.

[0022] S33. Establish constraint conditions according to a plurality of matched visible light image feature points.

[0023] S34, performing homography transformation alignment on the visible light image and the infrared image according to the constraint conditions, the multiple matched visible light image feature points and the multiple matched visible light image feature points, respectively, to obtain an aligned visible light image and an aligned infrared image.

[0024] S35 , respectively stitching the aligned visible light image and the aligned infrared image to obtain a visible light stitching image and an infrared stitching image.

[0025] Optionally, the constraint condition in S33 is as shown in the following formula (1):

[0026] (1)

[0027] in,

[0028] (2)

[0029] In the formula, , Represents the feature points of two adjacent visible light images a and b. represents the homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image b corresponding to image a, Represents the ordinate of the i-th feature point in image b corresponding to image a, , , , , , , , Represents the parameters in the solved homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image a corresponding to image b, Represents the ordinate of the i-th feature point in image a corresponding to image b.

[0030] Optionally, the splicing process in S35 is as shown in the following formula (3):

[0031] (3)

[0032] in,

[0033] (4)

[0034] In the formula, represents the stitched image, Indicates the number of groups of images, represents the weight factor of the splicing process, Indicates A set of visible light images, represents the pixel position within the overlapping area, Indicates the left boundary of the overlapping area, Indicates the right edge of the overlapping area.

[0035] Optionally, the process of constructing the multi-mode sensing detection network in S4 includes:

[0036] S41, obtaining a visible light stitching sample image and an infrared stitching sample image.

[0037] S42. Using MobileNet, extract feature maps of the visible light stitching sample image and the infrared stitching sample image to obtain a visible light feature map and an infrared feature map.

[0038] S43, adding the number of channels of the visible light feature map and the infrared feature map to generate an added feature map, and inputting the added feature map into three convolutional layers with jump connections respectively to obtain a first feature map, a second feature map, and a third feature map.

[0039] S44, performing convolution operations on the first feature map, the second feature map, and the third feature map, respectively, to obtain a first dimensionality reduction result, a second dimensionality reduction result, and a third dimensionality reduction result.

[0040] S45. Obtain a predicted target position and target category according to the first dimensionality reduction result, the second dimensionality reduction result, and the third dimensionality reduction result.

[0041] S46. Construct a loss function based on the predicted target position and target category, and train the multi-modal perception detection network based on the loss function to obtain a constructed multi-modal perception detection network.

[0042] Optionally, the loss function in S46 is as shown in the following equations (5)-(8):

[0043] (5)

[0044] (6)

[0045] (7)

[0046] (8)

[0047] In the formula, Represents the specific calculation formula of the loss function, represents the predicted value, represents the true value, Represents the loss value of the target bounding box, Represents the loss value of the horizontal coordinate of the center point, Represents the horizontal coordinate of the predicted target center point position, Indicates the horizontal coordinate of the actual target center point position, Represents the loss value of the ordinate of the center point, Represents the ordinate of the predicted target center point position, Indicates the ordinate of the actual target center point position, Represents the loss value of the target bounding box width, represents the width of the predicted target bounding box, Indicates the width of the actual target bounding box, Represents the loss value of the target bounding box height, represents the height of the predicted target bounding box, Indicates the height of the actual target bounding box, Represents the loss value of target category information, Indicates the number of target types. If the target category is correct, ,otherwise , Represents the probability value predicted as the target category, represents the loss function, Represents the weight parameter.

[0048] On the other hand, a multi-mode compound eye sensing device for complex degraded environments is provided, and the device is applied to a multi-mode compound eye sensing method for complex degraded environments. The device includes:

[0049] The acquisition module is used to acquire multiple groups of images in a complex degraded environment through a multi-mode compound eye acquisition device; wherein each group of images in the multiple groups of images includes a visible light image and an infrared image.

[0050] The extraction module is used to input multiple groups of images into the trained feature point prediction model to extract key feature point information in the visible light image and key feature point information in the infrared image.

[0051] The stitching module is used to generate a visible light stitching image and an infrared stitching image according to the nearest neighbor matching technology, key feature point information in the visible light image and key feature point information in the infrared image.

[0052] The output module is used to input the visible light stitching image and the infrared stitching image into the constructed multi-mode perception detection network for target detection to obtain the multi-mode perception detection result.

[0053] Optionally, the extraction module is further configured to:

[0054] S21. Acquire a visible light sample image.

[0055] S22, performing three convolution operations on the visible light sample image to obtain a first convolution feature map, a second convolution feature map, and a third convolution feature map.

[0056] S23, performing deconvolution operations on the first convolution feature map, the second convolution feature map, and the third convolution feature map, respectively, to obtain a first deconvolution image, a second deconvolution image, and a third deconvolution image.

[0057] S24, performing feature fusion on the first deconvolution image, the second deconvolution image, and the third deconvolution image to obtain a fused feature map.

[0058] S25. Input the fused feature map into the maximum pooling layer to obtain the maximum pooling layer output.

[0059] S26. Input the maximum pooling layer output to the bilinear interpolation layer to obtain the bilinear interpolation layer output.

[0060] S27. Input the output of the bilinear interpolation layer into the fully connected layer to obtain the key feature point information in the visible light sample image.

[0061] S28. Train the feature point prediction model according to the key feature point information in the visible light sample image to obtain a trained feature point prediction model.

[0062] Optionally, the splicing module is further used to:

[0063] S31 . Acquire feature points of adjacent visible light images, and use a nearest neighbor matching technique to match the acquired feature points to obtain a plurality of matched visible light image feature points.

[0064] S32, acquiring feature points of adjacent infrared images, and matching the acquired feature points using a nearest neighbor matching technique to obtain a plurality of matched infrared image feature points.

[0065] S33. Establish constraint conditions according to a plurality of matched visible light image feature points.

[0066] S34, performing homography transformation alignment on the visible light image and the infrared image according to the constraint conditions, the multiple matched visible light image feature points and the multiple matched visible light image feature points, respectively, to obtain an aligned visible light image and an aligned infrared image.

[0067] S35 , respectively stitching the aligned visible light image and the aligned infrared image to obtain a visible light stitching image and an infrared stitching image.

[0068] Optionally, the constraint condition is as shown in the following formula (1):

[0069] (1)

[0070] in,

[0071] (2)

[0072] In the formula, , Represents the feature points of two adjacent visible light images a and b. represents the homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image b corresponding to image a, Represents the ordinate of the i-th feature point in image b corresponding to image a, , , , , , , , Represents the parameters in the solved homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image a corresponding to image b, Represents the ordinate of the i-th feature point in image a corresponding to image b.

[0073] Optionally, the splicing process is as shown in the following formula (3):

[0074] (3)

[0075] in,

[0076] (4)

[0077] In the formula, represents the stitched image, Indicates the number of groups of images, represents the weight factor of the splicing process, Indicates A set of visible light images, represents the pixel position within the overlapping area, Indicates the left boundary of the overlapping area, Indicates the right edge of the overlapping area.

[0078] Optionally, the output module is further configured to:

[0079] S41, obtaining a visible light stitching sample image and an infrared stitching sample image.

[0080] S42. Using MobileNet, extract feature maps of the visible light stitching sample image and the infrared stitching sample image to obtain a visible light feature map and an infrared feature map.

[0081] S43, adding the number of channels of the visible light feature map and the infrared feature map to generate an added feature map, and inputting the added feature map into three convolutional layers with jump connections respectively to obtain a first feature map, a second feature map, and a third feature map.

[0082] S44, performing convolution operations on the first feature map, the second feature map, and the third feature map, respectively, to obtain a first dimensionality reduction result, a second dimensionality reduction result, and a third dimensionality reduction result.

[0083] S45. Obtain a predicted target position and target category according to the first dimensionality reduction result, the second dimensionality reduction result, and the third dimensionality reduction result.

[0084] S46. Construct a loss function based on the predicted target position and target category, and train the multi-modal perception detection network based on the loss function to obtain a constructed multi-modal perception detection network.

[0085] Optionally, the loss function is as shown in the following equations (5)-(8):

[0086] (5)

[0087] (6)

[0088] (7)

[0089] (8)

[0090] In the formula, Represents the specific calculation formula of the loss function, represents the predicted value, represents the true value, Represents the loss value of the target bounding box, Represents the loss value of the horizontal coordinate of the center point, Represents the horizontal coordinate of the predicted target center point position, Indicates the horizontal coordinate of the actual target center point position, Represents the loss value of the ordinate of the center point, Represents the ordinate of the predicted target center point position, Indicates the ordinate of the actual target center point position, Represents the loss value of the target bounding box width, represents the width of the predicted target bounding box, Indicates the width of the actual target bounding box, Represents the loss value of the target bounding box height, represents the height of the predicted target bounding box, Indicates the height of the actual target bounding box, Represents the loss value of target category information, Indicates the number of target types. If the target category is correct, ,otherwise , Represents the probability value predicted as the target category, represents the loss function, Represents the weight parameter.

[0091] On the other hand, a multi-mode compound eye perception device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the multi-mode compound eye perception methods for complex degraded environments is implemented.

[0092] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned multi-mode compound eye perception methods for complex degraded environments.

[0093] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0094] In an embodiment of the present invention, a deep learning algorithm is used to construct a feature point prediction model for multi-mode compound eye data in a complex degraded environment, and the nearest neighbor matching technology is used to realize the image stitching of visible light and infrared modes. In response to the feature point extraction requirements in the compound eye sensor, a feature point prediction model based on deep learning is constructed, and the convolutional neural network is used to realize the accurate prediction of key feature points. Image stitching is performed by calculating the homography transformation matrix, and the synchronous stitching of visible light and infrared images is realized. For the target detection task of multi-mode compound eye stitching images, a lightweight multi-mode perception detection network is constructed, and MobileNet is used to extract features and fuse them, realizing the perception detection task in a complex degraded environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0096] Figure 1 This is a flow chart of a multi-mode compound eye perception method for complex degraded environments provided by an embodiment of the present invention;

[0097] Figure 2 It is a schematic diagram of the framework of a multi-mode compound eye perception method for complex degraded environments provided by an embodiment of the present invention;

[0098] Figure 3 It is a framework diagram of a feature point prediction model provided by an embodiment of the present invention;

[0099] Figure 4 is a flow chart of a multi-mode sensing detection algorithm provided by an embodiment of the present invention;

[0100] Figure 5 This is a block diagram of a multi-mode compound eye sensing device for complex degraded environments provided by an embodiment of the present invention;

[0101] Figure 6 It is a structural schematic diagram of a multi-mode compound eye sensing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0102] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0103] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0104] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0105] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0106] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0107] The embodiment of the present invention provides a multi-mode compound eye sensing method for complex degraded environments. The method can be implemented by a multi-mode compound eye sensing device, which can be a terminal or a server. Figure 1 , Figure 2 The flowchart of the multi-mode compound eye perception method for complex degraded environments is shown. The processing flow of the method may include the following steps:

[0108] S1. Acquire multiple sets of images in a complex degraded environment through a multi-mode compound eye acquisition device; wherein each set of images in the multiple sets of images includes a visible light image and an infrared image.

[0109] In a feasible implementation, multi-mode compound eye data is collected in a complex degraded environment, and the multi-mode compound eye data includes multiple visible light compound eye images and infrared compound eye images.

[0110] S2. Input multiple groups of images into the trained feature point prediction model to extract key feature point information in the visible light image and key feature point information in the infrared image.

[0111] In a feasible implementation, the present invention constructs a feature point prediction model using a convolutional neural network in deep learning to meet the needs of feature point extraction in a compound eye sensor; the feature point prediction model based on the deep learning algorithm has stronger feature representation ability, generalization ability and flexibility than traditional methods, and the required feature point information can be accurately predicted through this model.

[0112] Optionally, the training process of the feature point prediction model in S2 may include the following steps S21-S28:

[0113] S21. Acquire a visible light sample image.

[0114] In a feasible implementation, Figure 3 As shown in the figure, multiple groups of visible light and infrared images in complex degraded environments are captured by multi-mode compound eye acquisition equipment. Each group of multi-mode sensors consists of a registered micro camera and a micro infrared camera. The data formula is expressed as:

[0115] (1)

[0116] (2)

[0117] In the formula, Indicates A set of visible light and infrared images, and They are Set of visible light and infrared images.

[0118] S22, performing three convolution operations on the visible light sample image to obtain a first convolution feature map, a second convolution feature map, and a third convolution feature map.

[0119] In a feasible implementation, three convolution operations are performed on the visible light image in step S21 to extract multi-scale similar feature point information in the image; the convolution formula is expressed as:

[0120] (3)

[0121] in, It is a convolution operation with a kernel of 3×3 and a stride of 2; , and They are three groups of feature maps extracted by three convolution operations; , and are the height, width and number of groups of the initial input image, is the number of channels.

[0122] S23, performing deconvolution operations on the first convolution feature map, the second convolution feature map, and the third convolution feature map, respectively, to obtain a first deconvolution image, a second deconvolution image, and a third deconvolution image.

[0123] In a feasible implementation manner, the step S22 , and Perform deconvolution operations respectively to generate , and .

[0124] S24, performing feature fusion on the first deconvolution image, the second deconvolution image, and the third deconvolution image to obtain a fused feature map.

[0125] In a feasible implementation, , and Perform feature fusion to generate feature maps .

[0126] (4)

[0127] (5)

[0128] (6)

[0129] in, yes times the deconvolution operation, ; yes , and A feature map where three feature maps are concatenated along the channel dimension; is the sigmoid activation function.

[0130] S25. Input the fused feature map into the maximum pooling layer to obtain the maximum pooling layer output.

[0131] In a feasible implementation, the output of step S24 is Input to the Max Pooling layer, which divides the input feature map into several regions and takes the maximum value of each region as the output to preserve the edge and texture information of the feature map:

[0132] (7)

[0133] in, is the output feature map Row, No. The value of the column; is the first Row, No. The input feature map area corresponding to the column has a size of 2×2. is the first input feature map Row, No. The value of the column.

[0134] S26. Input the maximum pooling layer output to the bilinear interpolation layer to obtain the bilinear interpolation layer output.

[0135] S27. Input the output of the bilinear interpolation layer into the fully connected layer to obtain the key feature point information in the visible light sample image.

[0136] In a feasible implementation, the output of step S25 is Generated by bilinear interpolation ; After a fully connected layer network outputs feature point information :

[0137] (8)

[0138] (9)

[0139] in, It is A set of feature points corresponding to a visible light image; It is In the visible light image feature points, ; It contains the location information of the feature point and the descriptor information of the feature point. The descriptor contains the statistical information, gradient information, color histogram and other information of the area around the feature point.

[0140] S28. Train the feature point prediction model according to the key feature point information in the visible light sample image to obtain a trained feature point prediction model.

[0141] In a feasible implementation, during the training phase, the obtained feature points are compared with the true values, and the interpolation with the true values ​​is compared, and the next training is repeated, and the weight parameters are continuously updated to enable the model to have the learning function.

[0142] S3. Generate a visible light stitched image and an infrared stitched image according to the nearest neighbor matching technology, key feature point information in the visible light image, and key feature point information in the infrared image.

[0143] In a feasible implementation, the homography transformation matrix is ​​calculated using the visible light image in the multi-mode compound eye and is simultaneously applied to the infrared image.

[0144] Optionally, the above step S3 may include the following steps S31-S35:

[0145] S31 . Acquire feature points of adjacent visible light images, and use a nearest neighbor matching technique to match the acquired feature points to obtain a plurality of matched visible light image feature points.

[0146] In a feasible implementation, after the feature point prediction model training is completed, the trained model is used to extract the feature points in the visible light image. ; For adjacent visible light images, the nearest neighbor matching technique is used to match feature points, and the threshold is set to 0.75; the matching formula can be expressed as:

[0147] (10)

[0148] (11)

[0149] (12)

[0150] in, is the set of feature points in adjacent images; It is the distance The smallest feature point ; and are two descriptor vectors, containing Features and Represents the first The value of a feature; yes and The similarity of and They are the feature points in the two images, and they are retained when the similarity is greater than or equal to the threshold.

[0151] S32, acquiring feature points of adjacent infrared images, and matching the acquired feature points using a nearest neighbor matching technique to obtain a plurality of matched infrared image feature points.

[0152] S33. Establish constraint conditions according to a plurality of matched visible light image feature points.

[0153] In a feasible implementation, for two adjacent visible light images, The following constraints can be established for the matched feature points:

[0154] (13)

[0155] in,

[0156] (14)

[0157] In the formula, , Represents the feature points of two adjacent visible light images a and b. represents the homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image b corresponding to image a, Represents the ordinate of the i-th feature point in image b corresponding to image a, , , , , , , , Represents the parameters in the solved homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image a corresponding to image b, It represents the ordinate of the i-th feature point in image a corresponding to image b. Formula (14) is the expanded term of formula (13).

[0158] S34, performing homography transformation alignment on the visible light image and the infrared image according to the constraint conditions, the multiple matched visible light image feature points and the multiple matched visible light image feature points, respectively, to obtain an aligned visible light image and an aligned infrared image.

[0159] S35 , respectively stitching the aligned visible light image and the aligned infrared image to obtain a visible light stitching image and an infrared stitching image.

[0160] In one feasible implementation, for all visible light images Perform homography transformation alignment; the aligned visible light images are spliced ​​in pairs using the following formula to generate a complete visible light splicing image :

[0161] (15)

[0162] in,

[0163] (16)

[0164] In the formula, represents the stitched image, Indicates the number of groups of images, represents the weight factor of the splicing process, Indicates A set of visible light images, represents the pixel position within the overlapping area, Indicates the left boundary of the overlapping area, Indicates the right edge of the overlapping area.

[0165] Furthermore, since the micro infrared camera has been registered with the micro camera, the infrared image Using the corresponding visible light image The steps to generate a complete infrared stitching image .

[0166] S4. Input the visible light stitching image and the infrared stitching image into the constructed multi-mode perception detection network for target detection to obtain the multi-mode perception detection result.

[0167] In a feasible implementation, Figure 4 As shown in the figure, a lightweight backbone network MobileNet is used to extract the feature information of visible light and infrared images after splicing the compound eye images, perform feature fusion, and finally predict the target information. Different from the existing target detection network, this paper constructs a lightweight multi-mode perception detection network to address the problem of large image scale after compound eye splicing. At the same time, it incorporates the multi-mode information in the compound eyes, and can perform perception detection tasks in real time in complex degraded environments.

[0168] Optionally, the process of constructing the multi-mode sensing detection network in S4 may include the following steps S41-S46:

[0169] S41, obtaining a visible light stitching sample image and an infrared stitching sample image.

[0170] S42. Using MobileNet, extract feature maps of the visible light stitching sample image and the infrared stitching sample image to obtain a visible light feature map and an infrared feature map.

[0171] In a feasible implementation, MobileNet is used to extract visible light stitching images respectively Stitching images with infrared The feature map of:

[0172] (17)

[0173] (18)

[0174] in, It is a lightweight backbone network; and They are The feature map and feature map.

[0175] S43, adding the number of channels of the visible light feature map and the infrared feature map to generate an added feature map, and inputting the added feature map into three convolutional layers with jump connections respectively to obtain a first feature map, a second feature map, and a third feature map.

[0176] In one feasible implementation, and Add the number of channels to generate feature maps ; Input To three convolutional layers with skip connections, a feature map with multi-scale information is obtained , and :

[0177] (19)

[0178] (20)

[0179] in, The convolution kernel is , convolution operation with a stride of 2; Is the input of the convolutional layer; is the output of the convolutional layer, ; It is downsampled by a factor of 2.

[0180] S44, performing convolution operations on the first feature map, the second feature map, and the third feature map, respectively, to obtain a first dimensionality reduction result, a second dimensionality reduction result, and a third dimensionality reduction result.

[0181] S45. Obtain a predicted target position and target category according to the first dimensionality reduction result, the second dimensionality reduction result, and the third dimensionality reduction result.

[0182] In one feasible implementation, , and , use 1×1 convolution operation to reduce dimension respectively; predict the position of image target through three dimension reduction results , category and confidence; where, is the target center point position Coordinates and the width and height of the target bounding box.

[0183] S46. Construct a loss function based on the predicted target position and target category, and train the multi-modal perception detection network based on the loss function to obtain a constructed multi-modal perception detection network.

[0184] In one feasible implementation, during the training phase, the loss function between the target position and category information obtained by calculation and the true value is , re-train the next time and continuously update the weight parameters to enable the model to have the function of learning; the loss function is as follows:

[0185] (twenty one)

[0186] (twenty two)

[0187] (twenty three)

[0188] (twenty four)

[0189] in, and Represent the predicted value and the true value respectively; and They are the predicted target location information and the real target location information; is the number of target types; if the target category is correct, ,otherwise ; is the weight parameter, set to 0.8.

[0190] The present invention uses a deep learning algorithm to construct a feature point prediction model. The nearest neighbor matching technology is used to match the feature points of adjacent images, and the homography transformation matrix is ​​solved for image stitching to generate visible light stitching images and infrared stitching images respectively. A multi-mode perception detection network is constructed to perform target detection on infrared and visible light images, and finally the center point coordinates, width, height and other information of the target bounding box in the image are obtained.

[0191] In an embodiment of the present invention, a deep learning algorithm is used to construct a feature point prediction model for multi-mode compound eye data in a complex degraded environment, and the nearest neighbor matching technology is used to realize the image stitching of visible light and infrared modes. In response to the feature point extraction requirements in the compound eye sensor, a feature point prediction model based on deep learning is constructed, and the convolutional neural network is used to realize the accurate prediction of key feature points. Image stitching is performed by calculating the homography transformation matrix, and the synchronous stitching of visible light and infrared images is realized. For the target detection task of multi-mode compound eye stitching images, a lightweight multi-mode perception detection network is constructed, and MobileNet is used to extract features and fuse them, realizing the perception detection task in a complex degraded environment.

[0192] Figure 5 The block diagram of a multi-mode compound eye sensing device for complex degraded environments is shown according to an exemplary embodiment. The device is used in a multi-mode compound eye sensing method for complex degraded environments. Figure 5 The device includes an acquisition module 310, an extraction module 320, a splicing module 330 and an output module 340. Among them:

[0193] The acquisition module 310 is used to acquire multiple groups of images in a complex degraded environment through a multi-mode compound eye acquisition device; wherein each group of images in the multiple groups of images includes a visible light image and an infrared image.

[0194] The extraction module 320 is used to input multiple groups of images into the trained feature point prediction model to extract key feature point information in the visible light image and key feature point information in the infrared image.

[0195] The stitching module 330 is used to generate a visible light stitching image and an infrared stitching image according to the nearest neighbor matching technology, key feature point information in the visible light image and key feature point information in the infrared image.

[0196] The output module 340 is used to input the visible light stitching image and the infrared stitching image into the constructed multi-mode perception detection network to perform target detection and obtain a multi-mode perception detection result.

[0197] Optionally, the extraction module 320 is further configured to:

[0198] S21. Acquire a visible light sample image.

[0199] S22, performing three convolution operations on the visible light sample image to obtain a first convolution feature map, a second convolution feature map, and a third convolution feature map.

[0200] S23, performing deconvolution operations on the first convolution feature map, the second convolution feature map, and the third convolution feature map, respectively, to obtain a first deconvolution image, a second deconvolution image, and a third deconvolution image.

[0201] S24, performing feature fusion on the first deconvolution image, the second deconvolution image, and the third deconvolution image to obtain a fused feature map.

[0202] S25. Input the fused feature map into the maximum pooling layer to obtain the maximum pooling layer output.

[0203] S26. Input the maximum pooling layer output to the bilinear interpolation layer to obtain the bilinear interpolation layer output.

[0204] S27. Input the output of the bilinear interpolation layer into the fully connected layer to obtain the key feature point information in the visible light sample image.

[0205] S28. Train the feature point prediction model according to the key feature point information in the visible light sample image to obtain a trained feature point prediction model.

[0206] Optionally, the splicing module 330 is further configured to:

[0207] S31 . Acquire feature points of adjacent visible light images, and use a nearest neighbor matching technique to match the acquired feature points to obtain a plurality of matched visible light image feature points.

[0208] S32, acquiring feature points of adjacent infrared images, and matching the acquired feature points using a nearest neighbor matching technique to obtain a plurality of matched infrared image feature points.

[0209] S33. Establish constraint conditions according to a plurality of matched visible light image feature points.

[0210] S34, performing homography transformation alignment on the visible light image and the infrared image according to the constraint conditions, the multiple matched visible light image feature points and the multiple matched visible light image feature points, respectively, to obtain an aligned visible light image and an aligned infrared image.

[0211] S35 , respectively stitching the aligned visible light image and the aligned infrared image to obtain a visible light stitching image and an infrared stitching image.

[0212] Optionally, the constraint condition is as shown in the following formula (1):

[0213] (1)

[0214] in,

[0215] (2)

[0216] In the formula, , Represents the feature points of two adjacent visible light images a and b. represents the homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image b corresponding to image a, Represents the ordinate of the i-th feature point in image b corresponding to image a, , , , , , , , Represents the parameters in the solved homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image a corresponding to image b, Represents the ordinate of the i-th feature point in image a corresponding to image b.

[0217] Optionally, the splicing process is as shown in the following formula (3):

[0218] (3)

[0219] in,

[0220] (4)

[0221] In the formula, represents the stitched image, Indicates the number of groups of images, represents the weight factor of the splicing process, Indicates A set of visible light images, represents the pixel position within the overlapping area, Indicates the left boundary of the overlapping area, Indicates the right edge of the overlapping area.

[0222] Optionally, the output module 340 is further configured to:

[0223] S41, obtaining a visible light stitching sample image and an infrared stitching sample image.

[0224] S42. Using MobileNet, extract feature maps of the visible light stitching sample image and the infrared stitching sample image to obtain a visible light feature map and an infrared feature map.

[0225] S43, adding the number of channels of the visible light feature map and the infrared feature map to generate an added feature map, and inputting the added feature map into three convolutional layers with jump connections respectively to obtain a first feature map, a second feature map, and a third feature map.

[0226] S44, performing convolution operations on the first feature map, the second feature map, and the third feature map, respectively, to obtain a first dimensionality reduction result, a second dimensionality reduction result, and a third dimensionality reduction result.

[0227] S45. Obtain a predicted target position and target category according to the first dimensionality reduction result, the second dimensionality reduction result, and the third dimensionality reduction result.

[0228] S46. Construct a loss function based on the predicted target position and target category, and train the multi-modal perception detection network based on the loss function to obtain a constructed multi-modal perception detection network.

[0229] Optionally, the loss function is as shown in the following equations (5)-(8):

[0230] (5)

[0231] (6)

[0232] (7)

[0233] (8)

[0234] In the formula, Represents the specific calculation formula of the loss function, represents the predicted value, represents the true value, Represents the loss value of the target bounding box, Represents the loss value of the horizontal coordinate of the center point, Represents the horizontal coordinate of the predicted target center point position, Indicates the horizontal coordinate of the actual target center point position, Represents the loss value of the ordinate of the center point, Represents the ordinate of the predicted target center point position, Indicates the ordinate of the actual target center point position, Represents the loss value of the target bounding box width, represents the width of the predicted target bounding box, Indicates the width of the actual target bounding box, Represents the loss value of the target bounding box height, represents the height of the predicted target bounding box, Indicates the height of the actual target bounding box, Represents the loss value of target category information, Indicates the number of target types. If the target category is correct, ,otherwise , Represents the probability value predicted as the target category, represents the loss function, Represents the weight parameter.

[0235] In an embodiment of the present invention, a deep learning algorithm is used to construct a feature point prediction model for multi-mode compound eye data in a complex degraded environment, and the nearest neighbor matching technology is used to realize the image stitching of visible light and infrared modes. In response to the feature point extraction requirements in the compound eye sensor, a feature point prediction model based on deep learning is constructed, and the convolutional neural network is used to realize the accurate prediction of key feature points. Image stitching is performed by calculating the homography transformation matrix, and the synchronous stitching of visible light and infrared images is realized. For the target detection task of multi-mode compound eye stitching images, a lightweight multi-mode perception detection network is constructed, and MobileNet is used to extract features and fuse them, realizing the perception detection task in a complex degraded environment.

[0236] Figure 6 is a schematic diagram of the structure of a multi-mode compound eye sensing device provided by an embodiment of the present invention, such as Figure 6 As shown, the multi-mode compound eye sensing device may include the above Figure 5 The multi-mode compound eye sensing device for complex degraded environments is shown. Optionally, the multi-mode compound eye sensing device 410 may include a first processor 2001.

[0237] Optionally, the multi-mode compound eye sensing device 410 may also include a memory 2002 and a transceiver 2003 .

[0238] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0239] Combine the following Figure 6 The components of the multi-mode compound eye sensing device 410 are specifically introduced:

[0240] The first processor 2001 is the control center of the multi-mode compound eye sensing device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0241] Optionally, the first processor 2001 can perform various functions of the multi-mode compound eye sensing device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0242] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 6 CPU0 and CPU1 are shown in FIG.

[0243] In a specific implementation, as an embodiment, the multi-mode compound eye sensing device 410 may also include multiple processors, such as Figure 6 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0244] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0245] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and access the first processor 2001 through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0246] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0247] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 6 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0248] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0249] It should be noted that Figure 6 The structure of the multi-mode compound eye sensing device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0250] In addition, the technical effects of the multi-mode compound eye sensing device 410 can refer to the technical effects of the multi-mode compound eye sensing method for complex degraded environments described in the above method embodiment, and will not be repeated here.

[0251] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0252] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0253] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0254] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0255] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0256] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0257] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0258] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0259] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0260] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0261] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0262] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0263] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-mode compound eye perception method for complex degraded environments, characterized in that: The method comprises: S1. Acquire multiple sets of images in a complex degraded environment through a multi-mode compound eye acquisition device; wherein each set of images in the multiple sets of images includes a visible light image and an infrared image; S2, inputting the plurality of groups of images into a trained feature point prediction model, and extracting key feature point information in the visible light image and key feature point information in the infrared image; S3, generating a visible light stitched image and an infrared stitched image according to the nearest neighbor matching technology, the key feature point information in the visible light image, and the key feature point information in the infrared image; S4. Input the visible light stitching image and the infrared stitching image into the constructed multi-mode perception detection network to perform target detection and obtain a multi-mode perception detection result.

2. The multi-mode compound eye perception method for complex degraded environments according to claim 1 is characterized in that: The training process of the feature point prediction model in S2 includes: S21, obtaining a visible light sample image; S22, performing three convolution operations on the visible light sample image to obtain a first convolution feature map, a second convolution feature map, and a third convolution feature map; S23, performing deconvolution operations on the first convolution feature map, the second convolution feature map, and the third convolution feature map, respectively, to obtain a first deconvolution image, a second deconvolution image, and a third deconvolution image; S24, performing feature fusion on the first deconvolution image, the second deconvolution image, and the third deconvolution image to obtain a fused feature map; S25, inputting the fused feature map into the maximum pooling layer to obtain the maximum pooling layer output; S26, inputting the maximum pooling layer output into a bilinear interpolation layer to obtain a bilinear interpolation layer output; S27, inputting the output of the bilinear interpolation layer into a fully connected layer to obtain key feature point information in the visible light sample image; S28. Train a feature point prediction model according to key feature point information in the visible light sample image to obtain a trained feature point prediction model.

3. The multi-mode compound eye perception method for complex degraded environments according to claim 1, characterized in that: The step of generating a visible light stitched image and an infrared stitched image according to the nearest neighbor matching technology, the key feature point information in the visible light image, and the key feature point information in the infrared image in S3 includes: S31, acquiring feature points of adjacent visible light images, and matching the acquired feature points using a nearest neighbor matching technique to obtain a plurality of matched visible light image feature points; S32, acquiring feature points of adjacent infrared images, and matching the acquired feature points using a nearest neighbor matching technique to obtain a plurality of matched infrared image feature points; S33, establishing constraint conditions according to the multiple matched visible light image feature points; S34, performing homography transformation alignment on the visible light image and the infrared image according to the constraint condition, the multiple matched visible light image feature points, and the multiple matched visible light image feature points, respectively, to obtain an aligned visible light image and an aligned infrared image; S35 , respectively stitching the aligned visible light image and the aligned infrared image to obtain a visible light stitched image and an infrared stitched image.

4. The multi-mode compound eye perception method for complex degraded environments according to claim 3 is characterized in that: The constraint condition in S33 is as shown in the following formula (1): （1） in, （2） In the formula, , Represents the feature points of two adjacent visible light images a and b. represents the homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image b corresponding to image a, Represents the ordinate of the i-th feature point in image b corresponding to image a, , , , , , , , Represents the parameters in the solved homography transformation matrix, Represents the horizontal coordinate of the i-th feature point in image a corresponding to image b, Represents the ordinate of the i-th feature point in image a corresponding to image b.

5. The multi-mode compound eye perception method for complex degraded environments according to claim 3, characterized in that: The splicing process in S35 is shown in the following formula (3): （3） in, （4） In the formula, represents the stitched image, Indicates the number of groups of images, represents the weight factor of the splicing process, Indicates A set of visible light images, represents the pixel position within the overlapping area, Indicates the left boundary of the overlapping area, Indicates the right edge of the overlapping area.

6. The multi-mode compound eye perception method for complex degraded environments according to claim 1, characterized in that: The construction process of the multi-mode sensing detection network in S4 includes: S41, obtaining a visible light stitching sample image and an infrared stitching sample image; S42, extracting feature maps of the visible light stitching sample image and the infrared stitching sample image through MobileNet to obtain a visible light feature map and an infrared feature map; S43, adding the number of channels of the visible light feature map and the infrared feature map to generate an added feature map, and inputting the added feature map into three convolutional layers with skip connections respectively to obtain a first feature map, a second feature map, and a third feature map; S44, performing convolution operations on the first feature map, the second feature map, and the third feature map, respectively, to obtain a first dimensionality reduction result, a second dimensionality reduction result, and a third dimensionality reduction result; S45, obtaining a predicted target position and target category according to the first dimensionality reduction result, the second dimensionality reduction result, and the third dimensionality reduction result; S46. Construct a loss function according to the predicted target position and target category, and train the multimodal perception detection network according to the loss function to obtain a constructed multimodal perception detection network.

7. The multi-mode compound eye perception method for complex degraded environments according to claim 6, characterized in that: The loss function in S46 is shown in the following equations (5)-(8): （5） （6） （7） （8） In the formula, Represents the specific calculation formula of the loss function, represents the predicted value, represents the true value, Represents the loss value of the target bounding box, Represents the loss value of the horizontal coordinate of the center point, Represents the horizontal coordinate of the predicted target center point position, Indicates the horizontal coordinate of the actual target center point position, Represents the loss value of the ordinate of the center point, Represents the ordinate of the predicted target center point position, Indicates the ordinate of the actual target center point position, Represents the loss value of the target bounding box width, represents the width of the predicted target bounding box, Indicates the width of the actual target bounding box, Represents the loss value of the target bounding box height, represents the height of the predicted target bounding box, Indicates the height of the actual target bounding box, Represents the loss value of target category information, Indicates the number of target types. If the target category is correct, ,otherwise , Represents the probability value predicted as the target category, represents the loss function, Represents the weight parameter.

8. A multi-mode compound eye sensing device for complex degraded environments, the multi-mode compound eye sensing device for complex degraded environments is used to implement the multi-mode compound eye sensing method for complex degraded environments as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used to acquire multiple sets of images in a complex degraded environment through a multi-mode compound eye acquisition device; wherein each set of images in the multiple sets of images includes a visible light image and an infrared image; An extraction module, used for inputting the plurality of groups of images into a trained feature point prediction model to extract key feature point information in the visible light image and key feature point information in the infrared image; A stitching module, used to generate a visible light stitching image and an infrared stitching image according to a nearest neighbor matching technology, key feature point information in the visible light image and key feature point information in the infrared image; The output module is used to input the visible light stitching image and the infrared stitching image into the constructed multi-mode perception detection network for target detection to obtain a multi-mode perception detection result.

9. A multi-mode compound eye sensing device, characterized in that: The multi-mode compound eye sensing device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images