A remote sensing target detection method and device integrating visible light, SAR and electronic reconnaissance

Abstract

This invention provides a method and device for remote sensing target detection that integrates visible light, SAR, and electronic reconnaissance. This method, which belongs to the field of space remote sensing technology, uses multi-agent reinforcement learning to implement on-orbit autonomous observation mission planning for remote sensing satellites. It acquires multi-source remote sensing data, including visible light remote sensing images, SAR remote sensing images, and electronic reconnaissance information. It then uses a generative adversarial network to perform image registration, and then fuses visible light-SAR feature information using improved compression and excitation techniques. It then fuses imaging-electronic reconnaissance feature information based on improved point-pair local topology and attribute information (D-S) evidence. Finally, it utilizes an interactive multi-model particle filter for more accurate target positioning and tracking, improving the robustness of target detection and making it suitable for target detection in complex environments. This invention can be applied to remote sensing reconnaissance satellite systems within satellite internet networks, providing strong support for both military and civilian applications, and possesses broad application prospects and value.

Description

Technical Field

[0001] The present invention relates to the field of space remote sensing technology, and in particular to a remote sensing target detection method and device that integrates visible light, SAR, and electronic reconnaissance. The method improves the accuracy and robustness of target detection and tracking by fusing multimodal remote sensing image data. Background Art

[0002] With the rapid development of space technology in recent years, remote sensing satellite observations of ground targets have evolved from a single method to a multi-source, multi-scale, and multi-functional approach. Remote sensing technology, as a technique for ground perception and observation from a distance, primarily uses satellite platforms equipped with sensors such as visible light, SAR, and electronic reconnaissance to capture ground data in various forms, including optical imagery, SAR imagery, and electronic reconnaissance information. Furthermore, due to the varying sensors carried by different remote sensing observation platforms, current remote sensing image data exhibits multi-sensor, multi-platform, and multi-temporal characteristics. The most commonly used sensors include optical sensors, synthetic aperture radar (SAR), and electronic reconnaissance payloads.

[0003] Optical sensors collect visual information from the spectrum to form images, providing spectral information and excellent visual observation, which is beneficial for image analysis research. However, due to external factors such as sunlight, weather, and time of day, image quality can be significantly reduced at night or in inclement weather. SAR is an active microwave radar detector. The image it generates reflects two characteristics of the target: structural characteristics and electromagnetic scattering characteristics. Synthetic aperture radar has penetrating properties, allowing it to detect obscured targets and is not affected by changes in time or weather. However, SAR images also have significant shortcomings. SAR images have low resolution and contain a large amount of coherent noise, which makes it difficult to fully interpret the image and accurately extract the image's contour information. An electronic reconnaissance payload is a payload that conducts reconnaissance on external radiation sources, obtains information on their location and type, and then performs detection and identification. Unlike optical, hyperspectral and other remote sensing equipment, microwave payloads are not affected by severe weather conditions such as clouds, rain, and fog, and can achieve all-day and all-weather surveillance. However, electronic reconnaissance is easily deceived and interfered by false signals, has a high false alarm rate, and relatively low positioning accuracy. When ground radars or radio stations are temporarily shut down, they can also avoid satellite reconnaissance, and the probability of missed detection is high.

[0004] Object detection in multimodal data faces the problem that traditional registration methods have difficulty achieving high-precision registration of multimodal data in complex scenes. Traditional registration methods are usually based on grayscale or feature point matching, but complex scenes are subject to problems such as geometric distortion, resolution differences, and changes in imaging conditions. These factors make it difficult for traditional methods to accurately align multimodal images. For example, geometric distortion may cause deformation of the image shape, while resolution differences and changes in imaging conditions may lead to inconsistent information between images. Most existing feature extraction methods rely on feature extraction of a single modality or simple multimodal feature fusion, failing to fully explore and utilize the complementary information of multimodal data. When processing multimodal data, they often fail to effectively combine the advantages of each modality. At present, although some deep learning-based detection methods have high accuracy, their computational complexity limits their promotion in real-time applications. The use of single-modal sensor data for ground target detection has poor robustness in complex environments. Therefore, the use of multi-source sensor data for target detection has gradually become a research hotspot. However, due to the heterogeneity of data types, data scales, and data distribution intervals between different modal data, the multimodal data application methods of simple data addition and data concatenation often fail to improve the result indicators and even lead to mutual influence between multimodal data.

[0005] To address the above issues, the present invention proposes a remote sensing target detection method that integrates visible light, SAR, and electronic reconnaissance, which has the following innovations and advantages:

[0006] Based on pre-deployed remote sensing satellites, user needs, and the location and attributes of ground targets, multi-satellite collaborative observation mission planning is implemented, and multi-source remote sensing data is acquired through remote sensing observation. The remote sensing satellites carry optical sensors, synthetic aperture radar sensors, and electronic reconnaissance payloads. The multi-source remote sensing data includes visible light remote sensing image data, SAR remote sensing image data, and electronic reconnaissance information data. The multi-source remote sensing data is fused through visible light-SAR feature information and imaging-electronic reconnaissance feature information. Finally, based on the fused feature data, more accurate target detection is performed, and continuous tracking of suspicious targets is achieved.

[0007] A distributed multi-satellite collaborative observation method based on improved multi-agent reinforcement learning adopts a multi-agent reinforcement learning algorithm to describe the multi-satellite collaborative task planning problem as a Markov intelligent game process. A decision network is obtained through distributed decision-making centralized training. Each remote sensing satellite can make decisions by referring to the status and strategies of other remote sensing satellites. The decision network is applied to the on-orbit autonomous observation task planning of remote sensing satellites, meeting remote sensing image observation tasks in scenarios such as disaster monitoring, environmental monitoring, and land use, and achieving good dynamic observation needs and collaborative observation effects.

[0008] An image registration method based on an improved generative adversarial network is proposed. The generative adversarial network is combined with the traditional feature algorithm to reduce the difference between optical images and SAR images and enhance the similarity of spectral information. A high-quality pseudo-color image is generated from the SAR image using the generative adversarial network. The gradient of the pseudo-color image generated by the visible light image and the SAR image is calculated using the Sobel operator. The scale space is constructed using the multi-scale Harris function (DIH). Key points with high repeatability and strong stability are detected. Feature descriptors are established based on the GLOH algorithm to enhance the stability of the registration results. The nearest neighbor distance ratio algorithm is used for coarse matching of feature points. The FSC algorithm is used to remove mismatched pairs to solve the problem of large differences in image texture information and structure between visible light and SAR modalities. In different scenarios where remote sensing targets exist, the image registration algorithm based on the adversarial network is effectively trained and tested to achieve remote sensing image registration tasks in new scenarios, achieving good registration effects and high robustness.

[0009] Based on the improved compression and excitation technology, the visible light-SAR feature information fusion technology obtains features from different encoders through cascade operation to roughly fuse the feature maps extracted from the multimodal data sources of visible light remote sensing images and SAR remote sensing images. In order to capture the contextual information of the feature maps between visible light and SAR modalities while performing feature fusion, global average pooling is used as a compression operation, which greatly reduces the number of parameters and the amount of calculation without affecting the feature fusion effect. Then, channel-adaptive multiplication and convolution layers are used to reconstruct the original cascade features to obtain the enhanced feature map of cascade fusion, thereby realizing the feature information fusion of visible light-SAR.

[0010] Based on the feature information fusion technology of imaging-electronic reconnaissance information combining improved point-to-point local topology and attribute information DS evidence, the radiation source features are extracted by the radiation source feature extraction module, and the hierarchical discriminant regression (HDR) classifier is constructed based on the hierarchical discriminant regression technology. The radiation source type is identified and judged by combining historical information and the radiation source information database; the imaging target type is further identified and the corresponding confidence is given by combining the target database; the probability distribution function of the attribute information is constructed according to the target recognition results of the satellite image information and electronic information; the probability distribution function based on the position information characteristics of the satellite image information and electronic information is constructed; the probability distribution function of the attribute information and the position information is combined based on the DS evidence theory to obtain the comprehensive probability distribution function; the association matching result is calculated by the comprehensive probability distribution function to obtain the target detection result.

[0011] A target tracking method based on an improved interactive multi-model particle filter is proposed. The current observation information is introduced to correct the model transfer probability to increase the proportion of the adapted model in the model interaction. At the same time, a particle filter method using particle Kalman optimization is proposed to replace the suboptimal filter in the model. The target tracking accuracy is improved through the interaction between the particle filter and the interactive multi-model.

[0012] Through these innovations, the present invention can resolve the contradictory problems of information redundancy in multi-source remote sensing image data and the limited application scope of single sensor images, as well as the low portability of existing multimodal remote sensing data fusion models, thereby achieving high-precision and high-efficiency target detection in complex scenarios, and significantly improving the reliability and practicality of remote sensing data processing and target tracking. Summary of the Invention

[0013] In view of this, the present invention provides a method and device for remote sensing target detection that integrates visible light, SAR, and electronic reconnaissance. Based on the positions and attributes of pre-deployed optical remote sensing satellites, SAR remote sensing satellites, electronic reconnaissance remote sensing satellites, and ground targets, and in combination with user needs, multi-satellite collaborative observation mission planning is carried out. Visible light, SAR, and electronic reconnaissance multi-source remote sensing data are acquired through remote sensing observation. The multi-source remote sensing data is fused with visible light-SAR feature information and imaging-electronic reconnaissance feature information. Finally, based on the fused feature data, more accurate target detection is performed, and continuous tracking of suspicious targets is achieved. The method and device mainly include the following steps:

[0014] Step 1: Based on an improved multi-agent reinforcement learning-based distributed multi-satellite collaborative observation method, the system receives user observation requests, implements on-orbit autonomous observation mission planning for remote sensing satellites, and selects the optimal observation mission through learning and optimization strategies to maximize the achievement of mission objectives.

[0015] Step 2: Based on the improved generative adversarial network image registration method, the preprocessing of visible light remote sensing image data, SAR remote sensing image data, and electronic reconnaissance information is completed, and image registration is achieved based on the generative adversarial network;

[0016] Step 3: Visible light-SAR feature information fusion technology based on improved compression and excitation technology, which fuses the features of visible light and SAR using improved compression and excitation technology to capture the contextual information between them;

[0017] Step 4: Feature fusion technology of imaging and electronic reconnaissance information based on improved point-to-point local topology and attribute information DS evidence is used to achieve feature fusion of imaging and electronic reconnaissance information, identify targets and provide confidence.

[0018] Step 5: Based on the target tracking method of the improved interactive multi-model particle filter, the current observation information is introduced, multiple tracking models are used, and switching and updating between different models are performed interactively to adapt to the appearance changes and motion mode conversion of the target. The observation noise and uncertainty are effectively processed by particle filtering, thereby improving the performance and reliability of target tracking and its robustness in complex scenarios.

[0019] The present invention eliminates or improves the defects in the prior art, solves the contradiction between the redundancy of multi-source remote sensing image data and the limited application range of single sensor images in the space remote sensing satellite target detection process, and the problem of low transferability of existing multimodal remote sensing data fusion models, thereby improving the accuracy and robustness of target detection.

[0020] The present invention provides a distributed multi-satellite collaborative observation method based on improved multi-agent reinforcement learning. The method adopts a multi-agent reinforcement learning algorithm to describe the multi-satellite collaborative task planning problem as a Markov intelligent game process. A decision network is obtained through distributed decision-making centralized training to complete multi-satellite collaborative observation. The method includes the following steps:

[0021] Mission requirement preprocessing stage: given the satellite status information s participating in the multi-satellite collaborative mission u The attitude angle at time t is θ, the current number of observations is n. Task status information s m Contains the execution time t and the corresponding attitude angle θ, and the weight w of the mission target, the information of other satellites s c .

[0022] Mission planning: The satellite's action space determines whether the observation mission is to be executed. If the mission is to be executed, the satellite must complete the maneuver within a specified timeframe; otherwise, the satellite's state remains unchanged. If the mission does not meet constraints such as the maneuver duration, the mission is abandoned and the observation action is not performed. For multi-satellite coordinated missions, due to mission uniqueness constraints, if the target has already been included in the observation sequence by another satellite, the mission will not be executed. After each satellite executes the action, its corresponding state is updated, and the total benefit of the multi-satellite coordinated mission is calculated.

[0023] Termination phase: When all satellite storage resources reach the upper limit n=n max , or when the remaining targets cannot be observed through attitude maneuvers within the mission cycle, the mission planning ends.

[0024] The Markov intelligent game process describes the relationship between cooperation and competition among multiple intelligent agents. Assume that in a Markov game process involving n intelligent agents, each intelligent agent i observes the environmental state S as O i, choose to take action A i The strategy is described as π i :O i ×A i , the next transformation of the environment is T:S×A1×…A n Depends on the actions of all agents. Each agent obtains a benefit r based on the state of the environment, that is, its own actions. i :S×A i , the total benefit of each agent γ is the discount rate, t is any time, and T is the termination time. The final goal is to find the expected reward of each agent under the connection strategy so that the strategy selected by any agent is the best when the strategies of other agents are determined.

[0025] The multi-agent reinforcement learning process is given the satellite's orbit r0, v0 at the initial time t0, and its attitude θ0 information, select task M from the observation task through the multi-agent reinforcement learning algorithm, the benefit of the observation task and the weight w of the task and the posture angle at the time of observation θ, when the target can no longer be observed before the termination time or the on-board storage resources are exhausted, the decision-making process of the observation task sequence ends, and the total benefit obtained by the observation sequence is maximized.

[0026] The present invention provides an image registration method based on an improved generative adversarial network, characterized in that the image registration method based on the improved generative adversarial network generates a high-quality pseudo-color image from a SAR image through an adversarial network, uses a multi-scale Harris algorithm and a GLOH algorithm to detect and describe feature points, and completes image registration. The method comprises the following steps:

[0027] Pseudo-color image generation stage: It consists of two neural networks: generator and discriminator, which transform the SAR remote sensing image I SAR The output z is fed into the generator G to generate a pseudo-color image I VG , let the output be x g ; By training the generator, it is expected to generate a pseudo-color image I VG Can be compared with real optical image x r The real data distribution p r (x) is similar; by θ g The nonlinear mapping function learned by the parameterized generator is represented as x g =G(z;θ g ); Discriminator D on the real optical image I V , generated pseudo color I VG For identification, its input is either a real sample or a generated false sample. The output y1 of the discriminator is a value, which indicates the probability that the input is a real sample or a false sample.d The nonlinear mapping function learned by the parameterized discriminator is expressed as y1=D(x;θ d ); through continuous updating, the generated pseudo-color image is closer to the real optical image, so that it can contain more spectral information. The image setting has a symmetric encoder-decoder with 8 convolutional layers and 3 jump connections, which extracts and utilizes the context information of the SAR image to form an image colorization neural network. The network learns the mapping relationship between grayscale SAR remote sensing images and colors, and combines the L1 loss and adversarial loss with appropriate weights pixel by pixel to form a new refined loss function, which adds color information to the SAR remote sensing image and converts it into a pseudo-color image.

[0028] Gradient calculation stage: Use the edge detection Sobel operator to quickly calculate the directional convolution kernel required for key point detection in the subsequent Harris algorithm, and define the horizontal template f H and the template f in the vertical direction V , use two templates to convolve with the image grayscale value to obtain the gradient values ​​in the horizontal and vertical directions as well as in Represented as the convolution of two rectangular sub-windows and Gaussian kernel function in the horizontal and vertical directions, respectively, with scale parameter α i Ensure the scale invariance of the image and satisfy The gradient magnitude and direction are as well as in represents the gradient magnitude matrix of the image, Represents the gradient direction matrix.

[0029] Feature point detection stage: Harries algorithm is used to extract feature points. Based on the gradient calculation, the multi-scale Harries function is used to construct the scale space. The local window is moved on the image to determine whether the grayscale changes significantly. For each window, the corresponding corner response function is calculated. Where λ1 and λ2 are two eigenvalues ​​of the image. If sliding the window in any direction causes the change in the image grayscale value to be greater than the set threshold, then it can be determined that there are corner features in this area. By calculating the local maximum, the candidate key points of each layer are extracted and non-maximum suppression is performed.

[0030] Feature point description and matching stage: The GLOH algorithm is used to calculate the gradient direction histogram of the sub-region around the key point and combine them into a comprehensive feature descriptor. On the basis of preserving the image structure information to the greatest extent, the algorithm calculation speed is improved and the stability of the registration result is increased; finally, the nearest neighbor distance ratio algorithm is used to calculate the Euclidean distance between the two images. The first two key points closest to the point (x1, y1) are selected, where ρ is the Euclidean distance between the point (x2, y2). This allows for rough matching of feature points, and the FSC algorithm is used to extract a subset C with a high correct matching rate from the set C. h , then in subset C h Sampling, and finding the maximum consistency set in set C, to remove the mismatched pairs, where C={C1,C2,…,C i ,…,C n}, p i The coordinates of the reference image are (x i ,y i ) feature points, and finally achieve image registration.

[0031] The present invention provides a feature information fusion technology for visible light SAR based on an improved compression and excitation technology. The technology is characterized in that, based on the compression and excitation technology, global average pooling is used to compress each channel of the current feature map, and an excitation function is used to perform nonlinear reconstruction of the global features, thereby fusing features from different encoders to achieve an improved feature fusion effect in the remote sensing data fusion model. The method comprises the following steps:

[0032] Global average pooling operation: By compressing each channel of the current feature map, the global spatial feature representation of each channel is obtained, and the context information of the feature maps between different modalities is captured while performing feature fusion. The cascade input is set to u 2c , 2c is the cascade feature u 2c The number of channels, W and H represent the cascade feature u 2c The width and height of the feature map obtained after the compression operation

[0033] Fully connected layer operation: The feature dimension is first reduced to 1 / 16 (1 / R) of the input, which greatly reduces the number of parameters and the amount of computation without affecting the feature fusion effect, thereby achieving attention operations that can be implemented with a small amount of computing resources.

[0034] ReLU activation operation: further nonlinear correction is performed on the global features. The feature value input is set to x. When x is greater than 0, the linear rectifier function ReLU(x) = x. When x is less than 0, the linear rectifier function ReLU(x) = 0. Through the ReLU activation operation, the corrected feature representation of the global features of each channel can be obtained.

[0035] Sigmoid activation operation: The original cascade features are reconstructed to obtain the enhanced feature map of the cascade fusion. The feature value input is set to x, e is a natural constant, and the Sigmoid activation layer maps the feature value to [0, 1]. The eigenvalues ​​obtained in this way can be regarded as the importance parameters corresponding to each feature channel. Finally, the obtained parameters are combined with the original feature map through the inner product method to obtain an enhanced representation of the feature.

[0036] The compression and excitation technology reconstructs features based on a soft attention mechanism, utilizes a continuous distribution algorithm in the range of [0, 1], optimizes its own parameters in the back propagation of the neural network, and ultimately achieves a more effective feature extraction method for channels or certain specific areas, and uniformly applies the attention between pixels and the attention to the impact of feature channels on the results to the construction of the neural network.

[0037] The present invention provides a feature information fusion method for imaging-electronic reconnaissance information based on an improved point-to-point local topology combined with attribute information DS evidence. The method is characterized in that target attribute information and location information of satellite image information and electronic information are extracted, an HDR classifier is constructed based on hierarchical discriminant regression technology, and the type of radiation source is identified and judged by combining historical information and a radiation source information library, and a probability distribution function of the attribute information and location information is constructed; the probability distribution function of the attribute information and location information is synthesized using DS evidence theory to obtain a comprehensive probability distribution function, an associated matching set is calculated, and a target detection result is obtained. The method comprises the following steps:

[0038] Image information target attribute information extraction, according to the rough set theory, the use of the same degree of separability attribute reduction method to select target features, the construction of the reduced training sample feature set X (Q, D), where Q = {C1, C2, ..., C i ,…,C l} is the result of feature selection, l is the dimension of the feature after reduction, C i is the i-th feature set, and D is the decision set of the target.

[0039] Extraction of target attribute information from electronic reconnaissance information. Assume that the set of radiation sources detected by satellite electronic reconnaissance information is T = {t1, t2, ..., t i ,…,tn}, then the carrier frequency, repetition frequency and pulse width are all n. These electromagnetic signal features are used to construct feature vectors and input into the HDR classifier to identify M types of radars. According to the assigned intelligence of radars and targets, M types of targets can be identified.

[0040] Extract local topological features of point pairs, calculate the Euclidean distance of any point pair in the point set p, select three-quarters of the maximum value of the Euclidean distance of the point pair as the radius, and randomly select point p i As the origin, we get p i The local area and p i The total number of adjacent points M a , where the random neighbor point p e Represented as a reference point; based on point pair p i and p e The local topology is to let is the positive axis of the polar coordinate system, and the other M a The quantized distance and angle vector set of the -1 point relative to the directed point pair pipeline is based on p i and p e The local topology of .

[0041] The calculation process of the comprehensive probability distribution function assumes that the detection data set of the imaging remote sensing satellite is S = {s1,s2,…,s i ,…,s n}, the detection data set of the electronic reconnaissance satellite is T = {t1,t2,…,t i ,…,t n}, construct the undirected edge weights of the distribution graph G Where v p =(s i ,t i ),v q =(s j ,t j ) represent the vertices of undirected edges; w{v p .v q}=1 means if For any target s in the satellite image information j , there is a unique target t in the satellite electronic reconnaissance information j With target j associated; using the proximity matrix M n×n To describe the distribution graph G, its non-diagonal elements are the weights of the undirected edges of the distribution graph, and solve the proximity matrix M n×n The main eigenvector of the optimal solution is the binary indicator vector x, which can be binarized to obtain the correct association set C M , and then the imaging data and electronic reconnaissance information data can be fused to obtain fused feature information.

[0042] The present invention provides a target tracking method based on an improved interactive multi-model particle filter, characterized in that model sampling is performed according to model transition probability, particle state prediction and gain optimization particles are implemented based on the Kalman filter algorithm, the motion state of the target is accurately estimated, and effective tracking and monitoring of the target is achieved. The method comprises the following steps:

[0043] Model transition probability process, for the r sub-models in the model, the probability u of sub-model i at time k i The larger (k) is, the more closely this sub-model matches the actual target motion pattern, and the greater the probability that other sub-models will transfer to this matching model. The sub-models include a uniform motion model, a uniformly accelerated motion model, and a maneuvering turning model, which are used to describe the situations where the target performs uniform linear motion, uniformly accelerated linear motion, and nonlinear motion, respectively. The calculation of the model probability is based on the measurement residual. This correction method takes into account the current measurement information and is more in line with the current system state.

[0044] In the interactive multi-model particle filtering process, a particle filter is run for each particle with model information. The sampling particles are optimized and updated through the rational use of the latest measurement information, thereby enhancing the effectiveness of the sampling particles in approximating the real state. At the same time, the particle filter of particle Kalman optimization is used to optimize the particles. The sampling particles with model information are used to copy and retain the large weight particles that can better represent the posterior distribution, and the small weight particles that can be ignored are discarded to reduce the filtering calculation amount and set the particle number threshold N. th , effective number of particles N eff , when N th >N eff When , resampling is performed, otherwise resampling is skipped. Finally, the final estimated state is output based on the normalized weight of the particle after resampling, and the latest observation information is introduced to enhance the effectiveness of the particle's estimation of the current state.

[0045] On the other hand, the present invention provides a system for implementing the above method, characterized in that the system adopts the above steps and methods to improve the accuracy and robustness of target detection by fusing multimodal remote sensing data, solves the problems existing in the prior art, and provides an effective solution for remote sensing image processing and target detection.

[0046] The beneficial effects of the present invention are at least:

[0047] The present invention provides a method and device for remote sensing target detection that integrates visible light, SAR, and electronic reconnaissance, belonging to the field of space remote sensing technology. The method is based on an improved distributed multi-satellite collaborative observation method using multi-agent reinforcement learning, and makes decisions based on the status and strategies of other remote sensing satellites. An image registration method based on an improved generative adversarial network combines the generative adversarial network with traditional feature algorithms to reduce the differences between optical images and SAR images, enhance the similarity of spectral information, and address the significant differences in image texture information and structure between visible light and SAR modalities, achieving good registration results and high robustness. A visible light-SAR feature information fusion technology based on an improved compression and excitation technique simultaneously captures contextual information of feature maps between visible light and SAR modalities during feature fusion, significantly reducing the number of parameters and computational complexity without affecting the feature fusion effect, thus achieving visible light-SAR feature information fusion. An HDR classifier is constructed based on hierarchical discriminant regression technology. The fused features of the imaging image and the probability distribution function of the attribute information and position information in the electronic reconnaissance information are synthesized based on the DS evidence theory to calculate the associated matching results, thereby improving the robustness of multi-source data fusion detection and being applicable to target detection in complex environments. A target tracking method based on an improved interactive multi-model particle filter improves the tracking accuracy of the target through the interaction between particle filtering and interactive multi-model.

[0048] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0049] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0051] Figure 1 The figure is a schematic diagram of the steps of a remote sensing target detection method integrating visible light, SAR and electronic reconnaissance in one embodiment of the present invention.

[0052] Figure 2 The figure is a flow chart of a method and device for remote sensing target detection integrating visible light, SAR and electronic reconnaissance in one embodiment of the present invention.

[0053] Figure 3The figure is a flow chart of a distributed multi-satellite collaborative observation method according to an embodiment of the present invention.

[0054] Figure 4 The figure is a flowchart of an image registration method based on a generative adversarial network in one embodiment of the present invention.

[0055] Figure 5 The figure is a flow chart of a visible light-SAR feature information fusion technology according to an embodiment of the present invention.

[0056] Figure 6 The figure is a flow chart of a feature information fusion technology of imaging and electronic reconnaissance information in one embodiment of the present invention.

[0057] Figure 7 The figure is a flow chart of a target tracking method based on improved interactive multi-model particle filtering in one embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0059] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0060] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0061] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0062] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0063] In order to solve the contradiction between the redundancy of multi-source remote sensing image data and the limited application scope of single sensor images in the process of space remote sensing satellite target detection, as well as the problem that the existing multimodal remote sensing data fusion model is not highly transferable, the present invention provides a visible light, SAR and electronic reconnaissance fusion remote sensing target detection method and device, the method realizes the on-orbit autonomous observation mission planning of remote sensing satellites based on multi-agent reinforcement learning, and obtains multi-source remote sensing data through remote sensing satellites for execution, the remote sensing satellites carry optical sensors, SAR sensors and electronic reconnaissance payloads, the multi-source remote sensing data include visible light remote sensing image data, SAR remote sensing image data and electronic reconnaissance information data, the multi-source remote sensing data forms image registration based on generative adversarial networks, the visible light-SAR feature information fusion is based on improved compression and excitation technology, and the imaging-electronic reconnaissance information feature information fusion is based on improved point-to-point local topology and attribute information DS evidence, and finally, based on the fused feature data, interactive multi-model particle filtering is used to perform more accurate target positioning and tracking functions, such as Figure 1 As shown, the method includes the following steps S101 to S105:

[0064] Step S101: Receive user observation requests and implement autonomous on-orbit observation mission planning for the remote sensing satellite. Through learning and optimization strategies, the optimal observation mission is selected to maximize the achievement of mission objectives. The satellite utilizes onboard optical sensors, SAR sensors, and electronic reconnaissance payloads to acquire visible light remote sensing image data, SAR remote sensing image data, and electronic reconnaissance information.

[0065] Step S102: Preprocess the visible light remote sensing image data, SAR remote sensing image data, and electronic reconnaissance information, and perform image registration based on a generative adversarial network (GAN). Using methods such as a generative adversarial network (GAN), the visible light remote sensing image data and the SAR remote sensing image data are registered to eliminate spatial and geometric differences between them. This registered multi-source remote sensing data enables better feature fusion and information extraction.

[0066] Step S103: Based on the processed visible light remote sensing image data and SAR remote sensing image data, visible light-SAR feature information fusion is completed, a feature extraction network is constructed, effective features are extracted, and multi-level feature fusion is achieved. This improves the accuracy of target detection and recognition and fully utilizes the complementary information of the visible light and SAR data sources.

[0067] Step S104: The fused image data and processed electronic reconnaissance information data are subjected to electronic information-imaging feature information fusion. Clustering is performed on the features to eliminate false alarms, complete detection, and calculate correlation matching results. Fusion of features from different information sources provides more comprehensive and accurate target intelligence.

[0068] Step S105: Introduce current observation information, use multiple tracking models, switch and update between different models in an interactive manner to adapt to the target's appearance changes and motion mode conversion, and effectively handle observation noise and uncertainty through particle filtering to improve the performance and reliability of target tracking, as well as its robustness in complex scenarios.

[0069] In the present invention, the remote sensing satellite payload is composed of a visible light sensor, a SAR sensor and an electronic reconnaissance payload. The remote sensing satellite acquires three types of data: visible light remote sensing image data, SAR remote sensing image data and electronic reconnaissance information data.

[0070] like Figure 2 FIG. 1 is a flow chart of a remote sensing target detection method that integrates visible light, SAR, and electronic reconnaissance. FIG.

[0071] In step S101, when a remote sensing satellite receives observation requirements uploaded by a ground system, these requirements may include observation requirements for a specific area, specific target, or specific time period. Key parameters and constraints, such as the boundaries of the observation area and the characteristics of the target, are obtained from the requirements information provided by the user or system. Simultaneously, based on the observation requirements, a multi-agent reinforcement learning method is used to design an agent to plan the remote sensing satellite's observation mission. Based on the current state and environmental information, the agent selects the optimal observation task through learning and optimization strategies to maximize the achievement of the mission objectives. Finally, the visible light remote sensing image data, SAR remote sensing image data, and corresponding electronic information data acquired by the satellite's visible light, SAR, and electronic reconnaissance payloads are processed and collected.

[0072] In step S102, preprocessing and image registration are performed on visible light remote sensing image data, SAR remote sensing image data, and electronic reconnaissance information. This includes functions such as thin cloud removal, geometric correction, radiometric correction, and speckle noise suppression for visible light and SAR remote sensing images, as well as preprocessing of electronic reconnaissance information. The thin cloud removal function uses image processing algorithms and convolutional neural networks to detect and remove thin clouds and atmospheric contamination from images, thereby improving image clarity and quality. Geometric correction converts remote sensing images from arbitrary positions and angles on the Earth's surface into a geometric reference frame with a unified coordinate system. Radiometric correction eliminates variations in brightness and contrast in remote sensing images caused by factors such as atmospheric and lighting conditions. The speckle noise suppression function uses image processing algorithms to filter SAR images to reduce or remove speckle noise, thereby improving image quality and visualization. Preprocessing of electronic reconnaissance information includes signal filtering, denoising, amplification, and spectrum analysis to improve signal quality and readability.

[0073] For visible light remote sensing image data and SAR remote sensing image data, a generative adversarial network is combined with traditional feature algorithms. The generative adversarial network is used to generate high-quality pseudo-color images from SAR images. The Sobel operator is used to calculate the gradient of the pseudo-color images generated by visible light images and SAR images. The multi-scale Harris function (DIH) is used to construct the scale space, and key points with high repeatability and strong stability are detected. Feature descriptors are established based on the GLOH algorithm to enhance the stability of the registration results. The nearest neighbor distance ratio algorithm is used for coarse matching of feature points, and the FSC algorithm is used to remove mismatched pairs, solving the problem of large differences in image texture information and structure between visible light and SAR modalities, and achieving good registration effects and high robustness.

[0074] In step S103 , visible light remote sensing image data and SAR remote sensing image data are fused to complete visible light-SAR feature information.

[0075] Specifically, each channel of the current feature map is compressed to obtain the global spatial feature representation of each channel. While performing feature fusion, the context information of the feature maps between different modalities is captured. The cascade input is set to u 2c , 2c is the cascade feature u 2c The number of channels, W and H represent the cascade feature u 2c The width and height of the feature map obtained after the compression operation

[0076] The feature dimension is first reduced to 1 / 16 (1 / R) of the input, which greatly reduces the number of parameters and the amount of computation without affecting the feature fusion effect, thereby realizing the attention operation with a small amount of computing resources.

[0077] The global features are further nonlinearly corrected, and the eigenvalue input is set to x. When x is greater than 0, the linear rectification function ReLU(x)=x, and when x is less than 0, the linear rectification function ReLU(x)=0. Through the activation operation of ReLU, the corrected feature representation of the global features of each channel can be obtained.

[0078] The original cascade features are reconstructed to obtain the enhanced feature map of the cascade fusion. The feature value input is set to x, e is a natural constant, and the Sigmoid activation layer maps the feature value to [0, 1]. The eigenvalues ​​obtained in this way can be regarded as the importance parameters corresponding to each feature channel. Finally, the obtained parameters are combined with the original feature map through the inner product method to obtain an enhanced representation of the feature.

[0079] In step S104 , the fused imaging image data and the processed electronic reconnaissance information data are subjected to feature information fusion of imaging-electronic reconnaissance information.

[0080] Specifically, the fused imaging image data and the processed electronic reconnaissance information data are subjected to feature information fusion of imaging and electronic reconnaissance information. The radiation source features are extracted by the radiation source feature extraction module. Based on the hierarchical discriminant regression technology, a hierarchical discriminant regression HDR classifier is constructed, and the radiation source type is identified and judged by combining historical information and the radiation source information database. The imaging target type is further identified and the corresponding confidence level is given in combination with the target database. According to the target recognition results of satellite image information and electronic information, the probability distribution function of attribute information is constructed. According to the target location information characteristics of satellite image information and electronic information, the probability distribution function based on location information is constructed. Based on the DS evidence theory, the probability distribution functions of attribute information and location information are integrated to obtain a comprehensive probability distribution function. The association matching result is calculated through the comprehensive probability distribution function to obtain the target detection result.

[0081] In step S105, current observation information is introduced and multiple tracking models are used to perform subsequent continuous tracking on the detected suspicious target.

[0082] Specifically, model sampling is performed based on model transition probabilities, with three sub-models set up: uniform motion, uniformly accelerated motion, and maneuvering turning models. These models are used to describe the target's uniform linear motion, uniformly accelerated linear motion, and nonlinear motion, respectively. A particle filter is run on each particle with model information. The Kalman filter algorithm is used to predict particle states and optimize particle gains. Using sampled particles with model information, large-weight particles that best represent the posterior distribution are replicated and retained, while negligible small-weight particles are discarded. This reduces the filtering computational effort, accurately estimates the target's motion state, and enables effective tracking and monitoring of the target.

[0083] The present invention provides a multi-satellite collaborative observation method based on the present invention to perform multi-satellite collaborative mission planning to obtain multi-source remote sensing data, such as Figure 3 As shown, the process specifically includes the following steps S301 to S303:

[0084] Step S301: Given the state information s of the satellites participating in the multi-satellite collaborative mission u The attitude angle at time t is θ, the current number of observations is n. Task status information s m Contains the execution time t and the corresponding attitude angle θ, and the weight w of the mission target, the information of other satellites s c .

[0085] Step S302: The satellite's action space indicates whether the observation mission is executed. If the mission is executed, the satellite must complete an attitude maneuver within a specified timeframe. Otherwise, the satellite's state remains unchanged. If the mission does not meet constraints such as the maneuver duration, the mission is abandoned and the observation action is not performed. For multi-satellite collaborative missions, due to task uniqueness constraints, if the target has already been included in the observation sequence by another satellite, the mission will not be executed. After each satellite executes the action, its corresponding state is updated, and the total benefit of the multi-satellite collaborative mission is calculated.

[0086] Step S303: When all satellite storage resources reach the upper limit n=n max , or when the remaining targets cannot be observed through attitude maneuvers within the mission cycle, the mission planning ends.

[0087] like Figure 4 As shown, the image registration method based on the improved generative adversarial network of the present invention specifically includes the following steps S401 to S404:

[0088] Step S401: Considering the image colorization problem, a symmetric encoder-decoder with 8 convolutional layers and 3 skip connections is set up to extract and utilize the context information of the SAR image to form an image colorization neural network. The mapping relationship between grayscale SAR remote sensing images and colors is learned through the network. The L1 loss and adversarial loss are combined pixel by pixel with appropriate weights to form a new refinement loss function, which adds color information to the SAR remote sensing image and converts it into a pseudo-color image.

[0089] Step S402: Before extracting feature points, first calculate the gradient of the pseudo-color image generated by the original optical image and the SAR image. The Sobel operator can quickly calculate the image gradient and provide the directional convolution kernel required by the Harries key point detection algorithm. Define the template f in the horizontal direction H and the template f in the vertical direction V .

[0090]

[0091] Use two templates to convolve with the image grayscale value to obtain the gradient values ​​in the horizontal and vertical directions. The multi-scale Sobel operator used can be expressed as the following formula:

[0092]

[0093] in, Represented as the convolution of two rectangular sub-windows and Gaussian kernel function in the horizontal and vertical directions respectively. The scale parameter, α i is a scale parameter used to ensure the scale invariance of the image and satisfies the following formula:

[0094]

[0095] Therefore, the gradient magnitude and direction are:

[0096]

[0097] in represents the gradient magnitude matrix of the image, Represents the gradient direction matrix.

[0098] Step S403: Feature point extraction is performed using the Harries algorithm. The multi-scale Harries algorithm can detect key points with high repeatability and strong stability, and is more accurate and faster than the minimum kernel similarity region and other center detection methods. Based on gradient calculation, the multi-scale Harries function is used to construct the scale space. Candidate key points at each layer are extracted by calculating local maxima, and non-maximum suppression is performed.

[0099] Step S404: After feature point detection, the GLOH algorithm is used to create feature descriptors. This algorithm improves computational speed and enhances the stability of the registration results while preserving image structural information to the greatest extent possible. Finally, a nearest neighbor distance ratio algorithm is used for coarse feature point matching, and the FSC algorithm is used to remove mismatched pairs, achieving image registration.

[0100] like Figure 5 As shown, the method for performing visible light-SAR feature information fusion based on visible light remote sensing image data and SAR remote sensing image data provided by the present invention specifically includes the following steps S501 to S504:

[0101] Step S501: extracting high-level and low-level features of visible light and SAR remote sensing image data respectively.

[0102] Step S502: Cross-modal feature fusion is performed on the high-level and low-level features of the visible light and SAR remote sensing image data. Global average pooling is used to compress each channel of the current feature map to obtain a global spatial feature representation for each channel. Since the length and width of the global average pooling unit used are the same as the feature map, the features obtained are global features of each channel with a size of C×1×1.

[0103] Step S503: After obtaining the global feature representation of each channel, in order to further obtain the channel importance parameter, the fully connected layer and different activation functions are used alternately to perform nonlinear reconstruction of the global features. The global features are reduced in dimension by the fully connected layer, and the original number of feature channels C is reduced to 1 / R of the original, so as to achieve attention operation that can be achieved with a small amount of computing resources. In order to further perform nonlinear correction on the global features, after the first fully connected operation, the linear rectification function (ReLU) activation function is used for nonlinear mapping. The specific formula is as follows:

[0104]

[0105] Where x is the eigenvalue input. Through the ReLU activation operation, the modified feature representation of the global feature of each channel can be obtained. After the feature is dimensionally upgraded through the second fully connected operation, a Sigmoid activation layer is used to map the feature value to [0, 1]. The formula is as follows:

[0106]

[0107] Where x is the eigenvalue of the input, e is a natural constant, and the eigenvalue obtained in this way can be regarded as the importance parameter corresponding to each feature channel.

[0108] Step S504: The obtained parameters are applied to the original feature map by inner product to obtain an enhanced representation of the features. The high-level and low-level features are combined to output the fused features of visible light and SAR.

[0109] The present invention also provides a method for fusing imaging image data and electronic reconnaissance information data into characteristic information of imaging-electronic reconnaissance information, such as Figure 6 As shown, the process specifically includes the following steps S601 to S603:

[0110] Step S601: extracting target attribute information and location information of satellite image information and electronic information respectively;

[0111] Step S602: Based on the hierarchical discriminant regression technique, an HDR classifier is constructed. The radiant source type is identified and determined by combining historical information with the radiant source database. Based on the target recognition results from the satellite imagery and electronic information, a probability distribution function for attribute information is constructed. Based on the target location features from the satellite imagery and electronic information, a probability distribution function for location information is constructed.

[0112] Step S603: Using the DS evidence theory, the probability distribution functions of the attribute information and the location information are integrated to obtain a comprehensive probability distribution function, and the associated matching set is calculated.

[0113] like Figure 7As shown, the target tracking method based on the improved interactive multi-model particle filter provided by the present invention specifically includes the following steps S701 to S705:

[0114] Step S701: Model sampling is performed based on the model transition probability. The model is sampled based on the calculation result of the model probability obtained by measuring the residual, and switching is performed between different sub-models. The sub-models include a uniform motion model, a uniformly accelerated motion model, and a maneuvering turning model, which are used to describe the target's uniform linear motion, uniformly accelerated linear motion, and nonlinear motion, respectively, to adapt to the target's appearance changes and motion mode conversion.

[0115] Step S702: Use the Kalman filter algorithm to predict the particle state, and predict the future development of the state based on the system's dynamic model and observation information. Use the Kalman filter to predict the particle's position, velocity, and other states to estimate the target's position in the next frame.

[0116] Step S703: Utilize the Kalman filter algorithm to optimize particle gain. By comparing the particle's predicted state with the observed data, the particle's weight is calculated to reflect its fit to the observed data. A particle filter is run on each particle with model information. By utilizing the latest measurement information, the sampled particles are optimized and updated. By optimizing the particle weights, the tracker's accuracy and robustness to the target are improved.

[0117] Step S704: Resample based on the number of valid particles. The valid number of particles refers to the number of particles with higher weights. The purpose of resampling is to retain the better particles and generate a new set of particles through copying and deleting operations. Resampling can increase particle diversity and improve the tracker's coverage of the target state.

[0118] Step S705: State output: Calculate the selection probability of each tracking model and output the target tracking result. The final estimated state is output based on the resampled particle normalized weights. The latest observation information is introduced to enhance the validity of the particle's current state estimate. The model with the highest probability is selected as the final target tracking result, and state information such as the target's position and velocity is output.

[0119] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program implements the steps of a target detection method and a target detection system that fuses satellite-borne visible light, SAR and electronic reconnaissance information data.

[0120] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0121] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0122] In summary, the present invention provides a method and device for remote sensing target detection that integrates visible light, SAR, and electronic reconnaissance, belonging to the field of space remote sensing technology. This method uses multi-agent reinforcement learning to implement on-orbit autonomous observation mission planning for remote sensing satellites, and executes the task by acquiring multi-source remote sensing data from remote sensing satellites. The remote sensing satellites carry optical sensors, SAR sensors, and electronic reconnaissance payloads. The multi-source remote sensing data includes visible light remote sensing image data, SAR remote sensing image data, and electronic reconnaissance information data. The multi-source remote sensing data is image-registered using a generative adversarial network. Feature information fusion of visible light and SAR is performed using an improved compression and excitation technique, and feature information fusion of imaging and electronic reconnaissance information is performed using an improved point-to-point local topology and attribute information DS evidence. Finally, based on the fused feature data, an interactive multi-model particle filter is used to perform more accurate target positioning and tracking. This method improves the robustness of multi-source data fusion detection and is suitable for target detection in complex environments. This invention can be directly applied to remote sensing reconnaissance satellite systems in satellite internet, providing strong support for both military and civilian applications, and possesses broad and significant application prospects and value.

[0123] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0124] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0125] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0126] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A remote sensing target detection method integrating visible light, SAR and electronic reconnaissance, characterized in that: The method comprises the following steps: Step 1: Based on an improved multi-agent reinforcement learning-based distributed multi-satellite collaborative observation method, the system receives user observation requests, implements on-orbit autonomous observation mission planning for remote sensing satellites, and selects the optimal observation mission through learning and optimization strategies to maximize the achievement of mission objectives. Step 2: Based on the improved generative adversarial network image registration method, the preprocessing of visible light remote sensing image data, SAR remote sensing image data, and electronic reconnaissance information is completed, and image registration is achieved based on the generative adversarial network; Step 3: Visible light-SAR feature information fusion technology based on improved compression and excitation technology, which fuses the features of visible light and SAR using improved compression and excitation technology to capture the contextual information between them; Step 4: Feature fusion technology of imaging and electronic reconnaissance information based on improved point-to-point local topology and attribute information DS evidence is used to achieve feature fusion of imaging and electronic reconnaissance information, identify targets and provide confidence. Step 5: Based on the target tracking method of the improved interactive multi-model particle filter, the current observation information is introduced, multiple tracking models are used, and switching and updating between different models are performed interactively to adapt to the appearance changes and motion mode conversion of the target. The observation noise and uncertainty are effectively processed by particle filtering, thereby improving the performance and reliability of target tracking and its robustness in complex scenarios.

2. The method for remote sensing target detection by integrating visible light, SAR and electronic reconnaissance according to claim 1, characterized in that: The distributed multi-satellite collaborative observation method based on improved multi-agent reinforcement learning adopts a multi-agent reinforcement learning algorithm to describe the multi-satellite collaborative task planning problem as a Markov intelligent game process. The decision network is obtained through distributed decision-making centralized training to complete the multi-satellite collaborative observation, which includes the following steps: The Markov game process describes the relationship between cooperation and competition among multiple agents. Assume that in the Markov game process involving n agents, each agent i observes the environment state S as O i , choose to take action A i The strategy is described as π i :O i ×A i , the next transformation of the environment is T:S×A1×…A n Depends on the actions of all agents. Each agent obtains a benefit r based on the state of the environment, that is, its own actions. i :S×A i , the total benefit of each agent γ is the discount rate, t is any time, and T is the termination time. The final goal is to find the expected reward of each agent under the connection strategy so that the strategy selected by any agent is the best when the strategies of other agents are determined. The process of multi-agent reinforcement learning, given the satellite's orbit r0, v0, and attitude at the initial time t0 θ0 information, select task M from the observation task through the multi-agent reinforcement learning algorithm, the benefit of the observation task and the weight w of the task and the posture angle at the time of observation θ, when the target can no longer be observed before the termination time or the on-board storage resources are exhausted, the decision-making process of the observation task sequence ends, and the total benefit obtained by the observation sequence is maximized.

3. The method for remote sensing target detection by integrating visible light, SAR and electronic reconnaissance according to claim 1, characterized in that: The image registration method based on the improved generative adversarial network generates a high-quality pseudo-color image from the SAR image through the adversarial network, uses the multi-scale Harris algorithm and the GLOH algorithm to detect and describe feature points, and completes the image registration, including the following steps: The pseudo-color image generation stage consists of two neural networks: a generator and a discriminator, which transform the SAR remote sensing image I SAR The output z is fed into the generator G to generate a pseudo-color image I VG , let the output be x g ; By training the generator, it is expected to generate a pseudo-color image I VG Can be compared with real optical image x r The real data distribution p r (x) is similar; by θ g The nonlinear mapping function learned by the parameterized generator is represented as x g =G(z;θ g ); Discriminator D on the real optical image I V , generated pseudo color I VG For identification, its input is either a real sample or a generated false sample. The output y1 of the discriminator is a value, which indicates the probability that the input is a real sample or a false sample. d The nonlinear mapping function learned by the parameterized discriminator is expressed as y1=D(x;θ d ); Through continuous updating, the generated pseudo-color image is closer to the real optical image, so that it can contain more spectral information. The image setting has a symmetric encoder-decoder with 8 convolutional layers and 3 skip connections, which extracts and utilizes the context information of the SAR image to form an image colorization neural network. The network learns the mapping relationship between grayscale SAR remote sensing images and colors. The L1 loss and adversarial loss are combined with appropriate weights pixel by pixel to form a new refined loss function, which adds color information to the SAR remote sensing image and converts it into a pseudo-color image. In the gradient calculation stage, the edge detection Sobel operator is used to quickly calculate the directional convolution kernel required for key point detection in the subsequent Harris algorithm, and the horizontal template f is defined. H and the template f in the vertical direction V , use two templates to convolve with the image grayscale value to obtain the gradient values ​​in the horizontal and vertical directions as well as in Represented as the convolution of two rectangular sub-windows and Gaussian kernel function in the horizontal and vertical directions, respectively, with scale parameter α i Ensure the scale invariance of the image and satisfy The gradient magnitude and direction are as well as in represents the gradient magnitude matrix of the image, represents the gradient direction matrix; In the feature point detection stage, the Harries algorithm is used to extract feature points. Based on the gradient calculation, the multi-scale Harries function is used to construct the scale space. The local window is moved on the image to determine whether the grayscale changes significantly. For each window, the corresponding corner response function is calculated. Where λ1 and λ2 are two eigenvalues ​​of the image. If sliding the window in any direction causes the change in the image grayscale value to be greater than the set threshold, then it can be determined that there is a corner feature in this area. By calculating the local maximum, the candidate key points of each layer are extracted and non-maximum suppression is performed. In the description and matching stage of feature points, the GLOH algorithm is used to calculate the gradient direction histogram of the sub-region around the key point and combine them into a comprehensive feature descriptor. On the basis of preserving the image structure information to the greatest extent, the calculation speed of the algorithm is improved and the stability of the registration result is increased. Finally, the nearest neighbor distance ratio algorithm is used to calculate the Euclidean distance between the two images. The first two key points closest to the point (x1, y1) are selected, where ρ is the Euclidean distance between the point (x2, y2). This allows for rough matching of feature points, and the FSC algorithm is used to extract a subset C with a high correct matching rate from the set C. h , then in subset C h Sampling, and finding the maximum consistency set in set C, to remove the mismatched pairs, where C={C1,C2,…,C i ,…,C n }, p i The coordinates of the reference image are (x i ,y i ) feature points, and finally achieve image registration.

4. The method for remote sensing target detection by integrating visible light, SAR and electronic reconnaissance according to claim 1, characterized in that: The visible light-SAR feature information fusion technology based on the improved compression and excitation technology is based on compression and excitation technology, uses global average pooling to compress each channel of the current feature map, and uses the excitation function to perform nonlinear reconstruction of the global features, thereby fusing features from different encoders to achieve an improved feature fusion effect in the remote sensing data fusion model, including the following steps: The compression and excitation technology reconstructs features based on a soft attention mechanism, using a continuous distribution algorithm in the range [0, 1] to optimize its own parameters during the back propagation of the neural network, ultimately achieving a more effective feature extraction method for channels or certain specific areas, and uniformly applying the attention between pixels and the attention to the impact of feature channels on the results to the construction of the neural network; The global average pooling operation compresses each channel of the current feature map to obtain the global spatial feature representation of each channel, and captures the context information of the feature maps between different modalities while performing feature fusion. The cascade input is set to u 2c , 2c is the cascade feature u 2c The number of channels, W and H represent the cascade feature u 2c The width and height of the feature map obtained after the compression operation The fully connected layer operation first reduces the feature dimension to 1 / 16 (1 / R) of the input, which greatly reduces the number of parameters and the amount of computation without affecting the feature fusion effect, thereby achieving attention operation with a small amount of computing resources; The linear rectification function activation operation performs further nonlinear correction on the global features. The eigenvalue input is set to x. When x is greater than 0, the linear rectification function ReLU(x)=x. When x is less than 0, the linear rectification function ReLU(x)=0. Through the ReLU activation operation, the corrected feature representation of the global features of each channel can be obtained. The Sigmoid activation operation is characterized by reconstructing the original cascade features to obtain the enhanced feature map of the cascade fusion. The feature value input is set to x, e is a natural constant, and the Sigmoid activation layer maps the feature value to [0, 1]. The eigenvalues ​​obtained in this way can be regarded as the importance parameters corresponding to each feature channel. Finally, the obtained parameters are combined with the original feature map through the inner product method to obtain an enhanced representation of the feature.

5. The method for remote sensing target detection by integrating visible light, SAR and electronic reconnaissance according to claim 1, characterized in that: The feature information fusion technology of imaging-electronic reconnaissance information based on the improved point-to-point local topology and attribute information DS evidence extracts target attribute information and location information of satellite image information and electronic reconnaissance information, constructs an HDR classifier based on hierarchical discriminant regression technology, and combines historical information and radiation source information database to identify and judge the type of radiation source, and constructs the probability distribution function of attribute information and location information; uses DS evidence theory to synthesize the probability distribution function of attribute information and location information to obtain a comprehensive probability distribution function, calculates the associated matching set, and obtains the target detection result. The following steps are included: The target attribute information of satellite image information is extracted. According to the rough set theory, the attribute reduction method with equal separability is used to select the target features and construct the reduced training sample feature set X(Q,D), where Q = {C1, C2, ..., C i ,…,C l } is the result of feature selection, l is the dimension of the feature after reduction, C i is the i-th feature set, and D is the decision set of the target; Extraction of target attribute information from electronic reconnaissance information. Assume that the set of radiation sources detected by satellite electronic reconnaissance information is T = {t1, t2, ..., t i ,…,t n }, then the number of carrier frequency, repetition frequency and pulse width are all n. These electromagnetic signal features are used to construct feature vectors and input into the HDR classifier to identify M types of radars. Based on the assigned intelligence of radars and targets, M types of targets can be identified. Point pair local topological feature technology calculates the Euclidean distance of any point pair in the point set p, selects three-quarters of the maximum value of the Euclidean distance of the point pair as the radius, and randomly selects a point p i As the origin, we get p i The local area and p i The total number of adjacent points M a , where the random neighbor point p e Represented as a reference point; based on point pair p i and p e The local topology is to let is the positive axis of the polar coordinate system, and the other M a The quantized distance and angle vector set of the -1 point relative to the directed point pair pipeline is based on p i and p e The local topology of The calculation process of the comprehensive probability distribution function assumes that the detection data set of the imaging remote sensing satellite is S = {s1,s2,…,s i ,…,s n }, the detection data set of the electronic reconnaissance satellite is T = {t1,t2,…,t i ,…,t n }, construct the undirected edge weights of the distribution graph G Where v p =(s i ,t i ),v q =(s j ,t j ) represent the vertices of undirected edges; w{v p .v q }=1 means if For any target s in the satellite image information j , there is a unique target t in the satellite electronic reconnaissance information j With target j associated; using the proximity matrix M n×n To describe the distribution graph G, its non-diagonal elements are the weights of the undirected edges of the distribution graph, and solve the proximity matrix M n×n The main eigenvector of the optimal solution is the binary indicator vector x, which can be binarized to obtain the correct association set C M , and then the imaging data and electronic reconnaissance information data can be fused to obtain fused feature information.

6. The method for remote sensing target detection by integrating visible light, SAR and electronic reconnaissance according to claim 1, characterized in that: The target tracking method based on the improved interactive multi-model particle filter performs model sampling according to the model transition probability, realizes particle state prediction and gain optimization particles based on the Kalman filter algorithm, accurately estimates the motion state of the target, and realizes effective tracking and monitoring of the target, including the following steps: Model transition probability process, for the r sub-models in the model, the probability u of sub-model i at time k i The larger (k) is, the more closely this sub-model matches the actual target motion pattern, and the greater the probability that other sub-models will transfer to this matching model. The sub-models include a uniform motion model, a uniformly accelerated motion model, and a maneuvering turning model, which are used to describe the target's uniform linear motion, uniformly accelerated linear motion, and nonlinear motion, respectively. The model probability is calculated based on the measurement residual. This correction method takes into account the current measurement information and is more consistent with the current system state. The interactive multi-model particle filtering process runs a particle filter on each particle with model information, optimizes and updates the sampled particles through the rational use of the latest measurement information, and thus enhances the effectiveness of the sampled particles in approximating the real state. At the same time, the particle filter optimized by particle Kalman optimization is used to optimize the particles. The sampled particles with model information are used to copy and retain the large weight particles that can better represent the posterior distribution, and the small weight particles that can be ignored are discarded to reduce the amount of filtering calculations and set the particle number threshold N. th , effective number of particles N eff , when N th >N eff When , resampling is performed, otherwise resampling is skipped. Finally, the final estimated state is output based on the normalized weight of the particle after resampling, and the latest observation information is introduced to enhance the effectiveness of the particle's estimation of the current state.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images