Satellite networking constellation-based space-ground collaborative remote sensing agent system

Abstract

The invention discloses a satellite networking constellation-based space-ground cooperative remote sensing agent system, which comprises a ground station and a satellite networking constellation, and is characterized in that the satellite networking constellation is divided into link establishment satellites and non-link establishment satellites according to link establishment conditions with the ground station; in the system, in response to a user input or sudden disaster event, a ground station generates an instruction file by using a ground large model and uploads the instruction file to a link establishment satellite; the link building satellite analyzes the instruction file and judges whether the link building satellite itself is a main responsibility satellite or not, if not, the link building satellite transmits the instruction file to the main responsibility satellite, and the main responsibility satellite generates a corresponding task plan through a satellite-borne large model; the main responsibility satellite generates a corresponding calculation request and / or shooting request based on the actual conditions of the main responsibility satellite and the other satellites, sends the calculation request and / or shooting request to the corresponding other satellites, receives returned calculation results and / or shooting results, generates a result file based on all the calculation results and / or shooting results, and downloads the result file to the ground station through the link building satellite; and the ground station analyzes and visualizes the result file.

Description

Technical Field

[0001] The present invention belongs to the frontier intersection field of aerospace engineering and intelligent computing, and in particular relates to a ground-ground collaborative remote sensing intelligent system based on a satellite networking constellation. Background Art

[0002] A single remote sensing satellite usually cannot meet the requirements of strong timeliness and high coverage, which requires multiple remote sensing satellites to form a constellation to jointly complete monitoring tasks. In order to reduce the waste of effective information in remote sensing data, on-orbit data analysis based on onboard computing power and inter-satellite networking is increasingly becoming a hot development trend in the field of satellite remote sensing.

[0003] At present, the resource configuration of many satellite network constellations is sufficient to support the normalized on-orbit reasoning and result transmission of artificial intelligence (AI) remote sensing image models based on deep learning technology. However, this model still has obvious limitations and cannot fully explore the functional potential of satellite network constellations. On the one hand, traditional AI models usually only reason for preset tasks and lack sufficient flexibility and the ability to handle complex remote sensing tasks; on the other hand, for ordinary users who lack knowledge and experience in the field of satellite remote sensing, the threshold for submitting requirements through traditional satellite measurement, operation and control platforms is still high. Summary of the invention

[0004] The purpose of the present invention is to improve the problems of high user threshold of satellite networking constellation platform, insufficient on-orbit computing flexibility, insufficient ability to cope with complex remote sensing tasks, and untimely response to emergencies. A space-ground collaborative remote sensing intelligent system based on satellite networking constellation is proposed to improve the overall intelligence level of satellite remote sensing applications, and explore a new space-ground collaborative intelligent application paradigm for the field of satellite remote sensing.

[0005] According to a first aspect of an embodiment of the present application, a space-ground collaborative remote sensing intelligent agent system based on a satellite networking constellation is provided, comprising a ground station and a satellite networking constellation, wherein the satellite networking constellation comprises a plurality of satellites, and is divided into link-building satellites and non-link-building satellites according to a link-building situation with the ground station;

[0006] In response to user input or sudden disaster events, the ground station generates a command file using a large ground model and uploads it to the link-building satellite;

[0007] In response to the command file, the link-building satellite parses the command file and determines whether it is the primary satellite. If not, the command file is transmitted to the primary satellite, and the primary satellite generates a corresponding mission plan using the onboard large model.

[0008] In response to the mission planning, the lead satellite generates corresponding calculation requests and / or shooting requests based on the computing power occupancy, energy consumption status, remote sensing equipment type and satellite position inclination of itself and other satellites, sends them to the corresponding other satellites and receives the returned calculation results and / or shooting results, generates a result file based on all the calculation results and / or shooting results, and transmits the result file to the ground station through the link-building satellite;

[0009] In response to the result file, the ground station parses and visualizes the result file.

[0010] Furthermore, the ground station communicates with the satellite networking constellation through a space distributed file system, and the satellites in the satellite networking constellation communicate with each other through an inter-satellite handshake.

[0011] Furthermore, if the instruction input by the user is an invalid instruction, the ground station uses the ground large model to encode the invalid instruction into a vector, matches the vector with the vector in the satellite remote sensing image library of the ground station, and generates several recommended instructions based on the satellite remote sensing image information corresponding to several vectors with the highest similarity. In response to the valid instruction input by the user or the recommended instruction selected by the user, the ground station uses the ground large model to parse the instruction, extract key information and pre-process it, thereby generating an instruction file and uploading it to the link establishment satellite.

[0012] Furthermore, in response to a sudden disaster event, the ground station uses a large ground model to analyze the sudden disaster event, extract key information and pre-process it, generate an instruction file for monitoring the sudden disaster event, and upload it to the link-building satellite.

[0013] Furthermore, the parameters of the onboard large model include base model parameters and LoRA incremental parameters, and the onboard large model of each satellite in the satellite networking constellation is updated, including:

[0014] The ground station updates the pre-saved copy of the onboard large model, solidifies the updated model capability to the LoRA incremental parameter part through the model distillation technology, quantifies the parameters of this part, forms a parameter update file, and uploads the parameter update file to the link-building satellite;

[0015] The link-building satellite synchronizes the parameter update file to all satellites in the satellite networking constellation;

[0016] Each satellite implements an on-orbit model update based on the fixed base model parameters and the updated LoRA incremental parameters according to the parameter update file.

[0017] Furthermore, each satellite conducts on-orbit testing and transmits the test results to the ground station, which is manually reviewed and selected to roll back the on-orbit model parameters or confirm the on-orbit model update.

[0018] Furthermore, the ground-based large model and the satellite-borne large model are large language models or multimodal large models including large language models.

[0019] According to a second aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the system as described in the first aspect when executed by a processor.

[0020] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0021] one or more processors;

[0022] A memory for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the system as described in the first aspect.

[0024] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the system described in the first aspect is implemented.

[0025] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0026] It can be seen from the above embodiments that the present application makes full use of onboard computing resources and inter-satellite communication capabilities, and builds a variety of professional intelligent entities for remote sensing tasks by using a rich external tool library such as map API, mission planning system, onboard digital image processing module, onboard AI model, remote sensing and communication payloads, etc., around the large model with tens of billions of parameters deployed on ground station servers and onboard computers. It also relies on the satellite-to-ground and inter-satellite microwave + laser high-speed communication infrastructure and the space distributed file system to realize a ground-to-earth collaborative intelligent entity system that integrates special functions such as natural language interaction, autonomous mission planning, multi-satellite instant interconnection, and on-orbit version iteration, supporting a new mode of on-orbit autonomous analysis and processing of complex remote sensing tasks. While expanding the boundaries of onboard intelligent computing capabilities, an open user interaction interface based on natural language is used to achieve multi-level autonomous ground intelligent detection capabilities with strong timeliness and high flexibility, significantly improving the accessibility of remote sensing satellite platforms to ordinary users.

[0027] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0029] Figure 1 It is a schematic diagram of a space-ground collaborative remote sensing intelligent agent system based on a satellite networking constellation according to an exemplary embodiment.

[0030] Figure 2 The following is a flowchart of generating a recommendation instruction according to an exemplary embodiment.

[0031] Figure 3 The present invention is a flowchart of updating parameters of a satellite-borne large model according to an exemplary embodiment.

[0032] Figure 4 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0034] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0036] like Figure 1As shown, the present application provides a space-ground collaborative remote sensing intelligent system based on a satellite networking constellation, the system comprising a ground station and a satellite networking constellation, the satellite networking constellation comprising a plurality of satellites, and divided into link-building satellites and non-link-building satellites according to the link-building conditions with the ground station, wherein the link-building satellite refers to a satellite that passes over a communication accessible sky range within a certain period of time and establishes a communication connection with the ground station, and after the link is established, the satellite can perform file transfer and information interaction with the ground station;

[0037] In response to user input or sudden disaster events, the ground station generates a command file and uploads it to the link-building satellite;

[0038] In response to the command file, the link-building satellite parses the command file and determines whether it is the primary satellite. If not, the command file is transmitted to the primary satellite, and the primary satellite generates a corresponding mission plan using the onboard large model.

[0039] In response to the mission planning, the lead satellite generates corresponding calculation requests and / or shooting requests based on the computing power occupancy, energy consumption status, remote sensing equipment type and satellite position inclination of itself and other satellites, sends them to the corresponding other satellites and receives the returned calculation results and / or shooting results, generates a result file based on all the calculation results and / or shooting results, and transmits the result file to the ground station through the link-building satellite;

[0040] In response to the result file, the ground station parses and visualizes the result file.

[0041] In this system, the ground station and the satellite networking constellation communicate through a space distributed file system, and the satellites in the satellite networking constellation communicate through inter-satellite handshake. In one embodiment, the space distributed file system can be an InterPlanetary File System (IPFS) or a SET protocol distributed file system, etc.

[0042] In a specific implementation, the ground station can set an interactive UI (User Interface) to obtain natural language instructions input by the user through the interactive UI, use the ground large model to parse the instructions, extract key information and pre-process them, generate instruction files, and upload them to the link-building satellite through the space distributed file system.

[0043] The ground station can also detect sudden disasters such as earthquakes, fires, mudslides, and landslides through the website of the Earthquake Bureau, news clients, and professional surveillance cameras, sensors, and drone aerial photography deployed in key areas. It can use the ground large model to analyze the sudden disaster event, extract key information, and pre-process it. It will generate a command file for monitoring the sudden disaster event by default and upload it to the link-building satellite through the space distributed file system.

[0044] In one possible implementation, the ground-based big model and the satellite-borne big model can be either pure language big models or multimodal big models with LLM (Large Language Model) parts. You can choose open source or commercial big models such as GPT, LLaMA, Mixtral, Claude, Qwen, MiniCPM, etc., or you can use a vertical domain big model that has been trained or fine-tuned from scratch using professional corpus in the field of satellite remote sensing.

[0045] In the specific implementation, after receiving the command file uploaded by the ground station, the link-building satellite first parses the command file and determines whether it is the main responsible satellite for the relevant task. If not, it needs to pass the command file to the target main responsible satellite through inter-satellite handshake communication. Then the onboard large model deployed on the main responsible satellite will automatically generate the mission plan according to the content in the command file.

[0046] In addition to supporting ordinary conversation functions based on large language models, since the time window for each satellite-ground link establishment is less than 8 minutes, the primary goal of the ground large model and interactive UI is to efficiently guide users so that they can submit valid instructions for corresponding satellite tasks in the shortest possible time. In one embodiment, a corresponding recommended instruction list can be provided for invalid user input (irrelevant questions, unsupported categories, unsupported time, and unsupported location). Among them, unsupported categories refer to categories that are not supported by the onboard AI model; unsupported time or location refers to the absence of pictures of the corresponding time or location in the question in the onboard image library; irrelevant questions refer to questions such as "Hello", "Have you eaten?", "Introduce yourself", etc. The ground large model determines whether it is a valid question. If not, a recommended instruction list is generated.

[0047] In order to improve the guidance efficiency and give effective instruction recommendations in combination with user preferences to the maximum extent, the present invention adopts RAG (Retrieval-augmented Generation) technology. The corresponding shooting time, longitude and latitude of the sub-satellite point, content sketch (generated by the onboard AI model) and other information in the satellite remote sensing image library synchronized to the ground station through the distributed file system are encoded into vectors in the latent space, stored in the remote sensing image vector library and continuously updated and maintained, such as Figure 2As shown in the figure, the invalid instructions input by the user through the interactive UI will first be encoded into a vector, and then the similarity will be matched with the vectors in the remote sensing image vector library. The satellite remote sensing image information corresponding to the most similar vectors will be input into the ground model and generate valid user instructions for the corresponding satellite mission, which will be displayed in the UI dialogue interface in the form of a clickable button list. The most commonly used recommended instruction forms are: "Is there a certain object near a certain place? If so, please draw the range / mark the location", "How many certain objects are there around a certain place?", "What is the total area of ​​a certain object around a certain place?", where a certain place is the place name covered by the satellite image, and an object is a category supported by the satellite AI model. The specific question form matches the object characteristics.

[0048] In one embodiment, when the user enters a valid instruction through a dialog box in the interactive UI or clicks a recommended instruction button, the ground model will automatically parse out key elements such as time, location, and user intent, and then obtain the latitude and longitude (range) of the location by querying the map API, and convert it into the ISO standard format together with the time information. If the large model determines that the instruction submitted by the user is a time-sensitive task that requires one or more satellites to take the latest remote sensing images of the target area in real time, it will automatically call the task planning software API provided by the ground measurement and control system, and obtain the satellite number list for performing the shooting task and the initial shooting plan for these satellites. The above information will be included in the instruction file and sent to the link-building satellite.

[0049] The process of processing sudden disaster event information is similar to the process of users actively inputting commands mentioned above. First, the ground large model automatically extracts key elements such as time, location, and event type from the acquired sudden disaster information, and after querying the map API and converting to the ISO standard format, it is packaged into a command file prompt.json and stored in the specified path, and then automatically uploaded to the preset storage path of the link-building satellite through the space distributed file system. In order to ensure the efficient and automatic progress of the above process, the prompt project of the ground-deployed large model is optimized for sudden disaster event response. For example, the disaster event identification standards, information extraction requirements for different disaster events, the list of ground callable / query tools and functional descriptions are described in detail, and the ground task planning process for different disaster events is also described, and specific examples are given respectively.

[0050] In addition, the ground-based big model will automatically select a satellite as the (initial) responsible satellite for each task based on the mission attributes and the satellite shooting planning information (including a clear flyby order) returned by the ground measurement, operation and control system, from multiple dimensions such as link establishment, shooting field of view, and immediacy. The reasoning, execution and surrounding resource scheduling of the onboard intelligent agent will all rely on the big model on the responsible satellite. For remote sensing tasks involving multi-satellite collaborative continuous shooting, the responsible satellite will automatically switch to the current leading shooting satellite as the satellite position and field of view change. All the above information (time, longitude and latitude, user intent, initial shooting plan, responsible satellite number, etc.) obtained by the ground-based big model through autonomous parsing and calling external tools will be packaged into the instruction file prompt.json together with the original user instructions and stored in the specified path of the distributed file system, and finally uploaded by the latter to the preset storage path of the link-building satellite.

[0051] According to the mission planning, the satellite-borne software system (for software tools) or the satellite-borne integrated power system (for hardware tools) of the main responsible satellite will gradually call the corresponding tools in the available tool set on the satellite. The satellite-borne large model will adjust the mission planning online based on the feedback of the execution of each step, and can automatically generate structured calculation requests and / or shooting requests based on the computing power occupancy, energy consumption status, remote sensing equipment type and satellite position inclination of the main responsible satellite itself and other satellites, and send them to the corresponding remaining satellites through inter-satellite communication and wait for the calculation results and / or shooting results to be returned. After the mission planning is fully executed, the final calculation results will be output, and the result files will be automatically packaged and generated, and then downloaded to the ground station through the space distributed file system. If the main responsible satellite is not a link-building satellite, it will first be transmitted to the link-building satellite through inter-satellite handshake communication, and then downloaded to the ground station.

[0052] In a possible implementation, after the link-building satellite reads the prompt.json file uploaded by the space distributed file system under the preset path, it first parses out the initial shooting plan and the main responsible satellite number. The former is distributed to all relevant satellites through a dedicated channel relying on intersatellite communication; the latter is used to determine whether the link-building satellite is the main responsible satellite for executing the intelligent task. If not, the command file will be delivered to the target satellite (main responsible satellite) through intersatellite handshake communication based on the QUIC protocol; if it is, or the non-link-building satellite receives the command file sent from other satellites as the main responsible satellite, the command file will be completely parsed on the spot, and the original command content (user-defined request, disaster event monitoring request, etc.) and key information such as time, longitude and latitude, disaster event description (optional), shooting plan of the satellite and relay satellite (optional) are input into the current onboard large model to obtain the task plan corresponding to the command file, including a complete list of steps required to complete the task (thinking chain), onboard tools to be called at each step, and the expected result form;

[0053] In one possible implementation, the tool sets available on each satellite in the satellite network constellation include the following categories:

[0054] 1. Digital image processing algorithm module: including relative radiation correction, geometric correction, denoising, geometric correction, panchromatic & multi-spectral fusion, image segmentation, geographic information registration and other functions;

[0055] 2. Onboard AI Model:

[0056] a. Super-resolution model, responsible for reprocessing a low-resolution remote sensing image into a high-resolution remote sensing image;

[0057] b. Scene classification model, responsible for classifying a remote sensing image into one or more of the following labels: '1': 'forest', '2': 'grassland', '3': 'cultivated land', '4': 'commercial area', '5': 'industrial area', '6': 'mining area', '7': 'power generation facility', '8': 'residential area', '9': 'airport', '10': 'road', '11': 'port area', '12': 'bridge', '13': 'train station', '14': 'viaduct', '15': 'bare land', '16': 'open waters', '17': 'small water bodies';

[0058] c. Object detection model, responsible for detecting the following categories of objects in a remote sensing image and outputting the location, size and category confidence of the object box: small car, large car, airplane, storage tank, ship, port, track and field, football field, tennis court, swimming pool, baseball field, overpass, basketball court, bridge, helicopter;

[0059] d. Semantic segmentation model, responsible for segmenting the semantic masks of the following objects in a remote sensing image: '1': 'Unknown building', '2': 'Greenhouse', '3': 'Storage tank', '4': 'Chimney', '5': 'Wind turbine', '6': 'Transmission tower', '7': 'Residential area', '8': 'Industrial area', '9': 'Commercial area', '10': 'Mining area', '11': 'Dam', '12': 'Photovoltaic power station', '13': 'Bridge', '14': 'Transportation hub', '15': 'Railway station', '16': 'Port', '17': 'Ship', '18': 'Airport', '19': 'Airplane', '20': 'Road', '21': 'Playground', '22': 'Stadium', '23': 'Mariculture', '24': 'Cultivated Land', '25': 'Grassland', '26': 'Forest', '27': 'Bare Land', '28': 'Lake', '29': 'River';

[0060] e. Instance segmentation model, responsible for segmenting instance masks of the following objects in a remote sensing image: '1': 'Unknown building', '2': 'Greenhouse', '3': 'Storage tank', '4': 'Chimney', '5': 'Wind turbine', '6': 'Transmission tower', '7': 'Residential area', '8': 'Industrial area', '9': 'Commercial area', '10': 'Mining area', '11': 'Dam', '12': 'Photovoltaic power station', '13': 'Bridge', '14': 'Transportation hub', '15': 'Railway station', '16': 'Port', '17': 'Ship', '18': 'Airport', '19': 'Airplane', '20': 'Road', '21': 'Playground', '22': 'Stadium', '23': 'Mariculture', '24': 'Cultivated Land', '25': 'Grassland', '26': 'Forest', '27': 'Bare Land', '28': 'Lake', '29': 'River';

[0061] f. CLIP (Contrastive Language-Image Pre-Training) model, after fine-tuning with remote sensing image data, is responsible for encoding a remote sensing image into a latent space vector and performing similarity matching with the vector encoded by the user command, thus forming a satellite image search tool with three dimensions of time, longitude and latitude, and semantics - the image retriever;

[0062] g. Disaster identification model:

[0063] (i) Fire monitoring model, which is responsible for identifying whether a remote sensing image contains burning points or burning areas with potential risks, and quantitatively estimating the extent and development trend of the fire;

[0064] (ii) Flood detection model, which is responsible for identifying whether a remote sensing image contains a flooded area and giving a reference confidence level;

[0065] (iii) Landslide detection model, which is responsible for identifying and marking the semantic mask of the landslide area in a remote sensing image;

[0066] 3. Intersatellite communication equipment: Based on laser communication or microwave communication hardware, with the help of pre-configured paths in the distributed file system, it can achieve information synchronization for specific satellites and satellite combinations, and can flexibly control the communication objects, communication content and communication timing.

[0067] 4. Satellite remote sensing equipment: Based on satellite remote sensing cameras (multi-spectral, hyperspectral), synthetic aperture radar and other imaging equipment, special compound detection equipment, etc., remote sensing images are taken at a specified time, at a specified latitude and longitude coordinates of the sub-satellite point, and at a specified angle.

[0068] For tasks involving multi-satellite collaborative continuous shooting, such as continuous monitoring of disaster events and continuous tracking of dynamic targets, the onboard large model on the current primary satellite is responsible for online generation and execution of the mission plan on the satellite. It is also necessary to output structured adjustment suggestions for the initial shooting plan (part of the input information of the current primary satellite large model) uploaded from the ground and distributed to the relay satellite according to various dynamic factors such as the development trend of the disaster event, the real-time changes in the orientation and speed of the moving target, and the relative attitude of the relay satellite (determined by the satellite shooting plan and the flyby order returned by the ground measurement and control system), and send it to the relay satellite through inter-satellite communication. When the latter becomes the new primary satellite, it will submit the latest shooting request to the onboard integrated power system for execution based on the adjusted shooting plan and the status of the onboard sensors. The above process will be sequentially transmitted between the new primary satellite and the new relay satellite until the task is completed.

[0069] The following three examples illustrate the task planning and its execution process.

[0070] (1) The user actively asks: "Is there any farmland near site A? If so, please mark its scope."

[0071] Ground station: After receiving the user's question through the interactive UI, the ground model first parses out key information such as time (unlimited, non-real-time request), location (location A), and range (unspecified, default coverage of one scene image), then calls the map API to query the location's longitude and latitude coordinates ([119.902104, 30.267681]), and determines the main responsible satellite number (the current link-building satellite, number 3, retrieved from the satellite-borne remote sensing image library). The above information and user input are packaged into prompt.json and uploaded to the link-building satellite through the space distributed file system.

[0072] Satellite network constellation: The satellite building the chain reads prompt.json in the specified path and parses it, confirms that the main satellite is consistent with the satellite number, inputs the parsed information into the satellite-borne large model, and gradually outputs the thinking chain for the task:

[0073] (a) Retrieve a remote sensing image that best matches the semantics of “farmland” from the satellite image library based on time, latitude and longitude coordinates, and call the tool – image retriever;

[0074] (b) A remote sensing image is successfully retrieved. It is necessary to further determine the scope of the farmland and call on the tool - semantic segmentation model;

[0075] (c) The mask of farmland is successfully obtained. Now the farmland area is estimated by the number of mask pixels and remote sensing imaging resolution;

[0076] (d) Successfully obtain an estimate of the farmland area and generate a final answer: "There is farmland near site A, which covers an area of ​​approximately 10.93 square kilometers. I have shown a picture of this area."

[0077] (e) the mission is successfully completed;

[0078] The above reasoning results together with the remote sensing images are packaged and transmitted to the ground station.

[0079] (2) The user asks a question: “How many ships are there near Port B now?”

[0080] Ground station: After receiving the user's question through the UI, the ground deployment large model first parses out key information such as time (now, strong real-time request), location (Port B), and range (unspecified, default coverage of one scene image), and then calls the Baidu Map API to query the location's longitude and latitude coordinates ([104.003747,1.305218]). It is judged to be a strong real-time, non-continuous shooting task, and the ground measurement and control system API is accessed to obtain the number of the best shooting satellite and determine it as the primary satellite (not the current link-building satellite, number 4, which requires real-time shooting in orbit). At the same time, the shooting plan for the primary satellite is obtained. The above information and the original user question are packaged into prompt.json and uploaded to the link-building satellite through the space distributed file system.

[0081] Satellite network constellation: The link-building satellite reads prompt.json in the specified path and parses it. It confirms that the numbers of the main responsible satellite and the local satellite are inconsistent. First, the command file is transmitted to the main responsible satellite through the QUIC protocol. After the main responsible satellite receives prompt.json and confirms that the numbers are consistent, it inputs the parsed information into the onboard large model and gradually outputs the thinking chain for the task:

[0082] (a) Take a remote sensing image of port B according to the ground plan, and use the tool - the satellite-borne multispectral camera;

[0083] (b) Successfully obtain a raw remote sensing image, which requires further preprocessing, calling the tool - digital image processing algorithm module;

[0084] (c) After obtaining the corrected RGB image, the ship target needs to be detected, and the tool - target detection model is called;

[0085] (d) The detection target boxes of all ships are successfully obtained. Now count the target boxes;

[0086] (e) The ship count result is successfully obtained, and the final answer is generated: "There are 93 ships near Port B now, and I have shown a picture containing this area";

[0087] (f) the mission is successfully completed;

[0088] The above calculation results together with the remote sensing images are packaged and first transmitted to the current link-building satellite, and then transmitted to the ground station by the latter.

[0089] (3) Real-time push notification from the Earthquake Bureau: "Location C, warning magnitude 3.3, earthquake waves are expected to arrive in 67 seconds"

[0090] Ground station: The ground-deployed large model receives the message pushed by the Earthquake Bureau. It first confirms it as an earthquake event warning and parses out key information such as time (67 seconds later, strong real-time request), location (C), and range (unspecified, default coverage of one scene image). Then it calls the Baidu Map API to query the location's longitude and latitude coordinates ([117.469000, 31.887601]), determines it as a strong real-time, continuous shooting task, accesses the ground measurement and control system API to obtain a list of satellite numbers suitable for monitoring and their initial shooting plan, and identifies the first satellite that passes over the target area as the initial primary satellite (not the current link-building satellite, number 8, which needs to be shot in orbit). The above information is packaged into prompt.json and uploaded to the link-building satellite through the space distributed file system.

[0091] Satellite network constellation: The satellite building the chain reads prompt.json in the specified path and parses it. It confirms that the numbers of the primary and local satellites are inconsistent. First, the command file is passed to the initial primary satellite through the QUIC protocol. After the primary satellite receives prompt.json and confirms that the numbers are consistent, it inputs the parsed information into the onboard large model and gradually outputs the thinking chain for the task:

[0092] (a) Take a remote sensing image of location C according to the ground plan, using the tool - a satellite-borne multispectral camera;

[0093] (b) Successfully obtain a raw remote sensing image, which requires further preprocessing, calling the tool - digital image processing algorithm module;

[0094] (c) After obtaining the calibrated RGB image, it is necessary to detect secondary disasters such as damaged buildings and fires, and call the tools - target detection model and fire monitoring model;

[0095] (d) Successfully obtain the target detection frame and the set of fire risk points, and generate a disaster report for the current primary responsible satellite;

[0096] (f) Based on the distribution of detected targets and the development trend of fire, a shooting plan correction plan for the next satellite (relay satellite) is generated and sent to the relay satellite together with prompt.json. When the relay satellite arrives above the target area, it will automatically become the new primary satellite. The final shooting plan is formed according to the shooting plan correction plan provided by the previous primary satellite, and a similar task process is repeated. Then, it is passed on from satellite to satellite until all continuous monitoring tasks are completed, and the tool - intersatellite communication equipment is called;

[0097] (g) the mission is successfully completed;

[0098] The above calculation results together with the remote sensing images are packaged and first transmitted to the current link-building satellite through inter-satellite handshake communication, and then transmitted to the ground station by the latter.

[0099] In a possible implementation, after all mission plans are completed, the main responsible satellite automatically packages the on-board calculation results and / or shooting results into result files. These results may include: RGB image slices (such as L3 slices) of the original remote sensing image after the digital image processing module, the inference output of the onboard AI model, and the secondary processing results of these inference outputs by the onboard large model. Considering the potential instability of satellite-to-ground and intersatellite communications and the need to improve user experience, different types of results will be dispersed and packaged and transmitted asynchronously. Specifically, large-volume remote sensing image data will be packaged in slices and named [x,y].jpg.zip according to its two-dimensional coordinates in the complete scene (composed of 9×9=81 slices), representing the xth row and the yth column; smaller text information will be packaged as results.json. The above result files are directly stored in the specified path by the main responsible satellite, or the intersatellite communication device interface is called to send them back to the specified path of the current link-building satellite in the form of intersatellite handshake communication, and then downloaded to the ground station through the space distributed file system.

[0100] In one possible implementation, after the ground station receives the result file transmitted from the link-building satellite in the specified path of the space distributed file system, it will decompress and parse the content as necessary and then put it into the ground station user interaction UI for visual display. Specifically, the text information will be displayed in the form of a message on the UI dialogue interface as a direct response to the user prompt; since the image slices are received asynchronously, the UI dialogue interface will first reserve a blank area containing a complete scene image of 9×9 slices below the text information. Each time a slice is received and decompressed, it will be filled in the corresponding position according to the two-dimensional coordinates in its file name until all slices are displayed. In addition to being displayed on the UI, the parsed results can also be distributed to other designated information recipients as needed, and trigger a specific form of alarm signal when necessary.

[0101] In one possible implementation, the ground model uses VLM (Visual Language Model). Users can automatically enter the visual question-answering interface by clicking on the interactive UI interface to receive remote sensing images transmitted from satellites. In the visual question-answering interface, users can also manually upload custom remote sensing images. In addition, users can also use the various area selection tools (rotatable rectangles, polygons, etc.) provided by the visual question-answering interface to select areas of interest on the original remote sensing image. Relying on the visual-language multimodal remote sensing model trained with massive remote sensing data, users can perform scene classification, target positioning, image analysis, target counting, multi-round dialogue and other intelligent interactions for relevant remote sensing images or image areas.

[0102] In a possible implementation, the historical dialogue records generated by the space-ground collaborative intelligent agent task and the visual question-answering task on the ground UI interactive interface will be automatically stored in the database for users to view at any time. Considering that the space-ground collaborative intelligent agent task is generally completed within a single satellite-ground link window period, this part of the historical dialogue record is set to only be able to view and click on the remote sensing slice to enter the visual question-answering, and no further questions are allowed; the difference is that the visual question-answering is not limited to the satellite-ground link window period. After loading, this part of the historical dialogue record will automatically extract the last valid remote sensing image and input it into the VLM large model, so that questions can continue to be asked about it. In the VLM large model, the remote sensing image is encoded using a visual encoder, and after being modally aligned with the ground large model by the modal adapter, it is input into the ground large model, thereby realizing visual question-answering based on the ground large model.

[0103] In one possible implementation, the onboard large model consists of two parts: the base model parameters and the LoRA incremental parameters. After the satellite is launched, ground R&D personnel can still perform subsequent fine-tuning and updates on the onboard large model copy, and there is no need to limit the specific fine-tuning method. LoRA fine-tuning, full parameter fine-tuning or other fine-tuning technologies that update model parameters are all possible, such as Figure 3 As shown, it is only necessary to solidify the updated model capability to the LoRA incremental parameter part of the base model + LoRA binary structure through the model distillation technology, and then quantize the parameters to compress the parameter update file to a level that can be fully uploaded in a single satellite-to-ground communication (<20MB). The parameter update file is sent to the link-building satellite through the distributed file system, and the parameter update file is automatically synchronized to all satellites in the constellation through inter-satellite communication. For the large satellite-borne model on each satellite, a fixed base model and variable LoRA incremental parameters are used to implement on-orbit model updates. After satellite-by-satellite on-orbit testing, downlink of inference results, and manual review, the on-board parameter update or rollback operation can be confirmed.

[0104] It can be seen from the above embodiments that the present application generates a recommended instruction list through RAG technology to guide users to efficiently propose effective instructions within the link building window period; uses the ground large model to automatically extract key information from user instructions and third-party push, and calls external tools such as map API and mission planning system API to obtain longitude and latitude coordinates and shooting plans; uses the main responsible satellite numbering mechanism to realize automatic forwarding of instruction files and calculation results and on-site reasoning of the satellite-borne large model through the space distributed file system; uses the ReAct intelligent body framework, and the satellite-borne large model outputs and executes mission planning thinking for specific needs chain; using the decentralized packaging, asynchronous transmission and blank explicitness of different types of result data, to cope with the potential instability of satellite-to-ground and inter-satellite communications and improve the user experience; by directly clicking on the satellite to upload remote sensing images or user-defined upload methods, to realize the visual question-and-answer function of remote sensing images, and provide a list of recommended instructions similar to the space-ground collaborative intelligent task; using the "base model + LoRA incremental parameter" binary structure and model distillation, quantization and other technologies, and standardized parameter file annotation, on-orbit testing, result confirmation and parameter update and rollback processes, to achieve efficient on-orbit updates of tens of billions of satellite-borne large models. In summary, this application relies on the satellite-to-ground and inter-satellite microwave + laser high-speed communication infrastructure and the space distributed file system, and builds a multi-agent system around the tens of billions of parameter large models deployed on the ground and on the satellite. Through the collaboration and cooperation at different levels such as data, models and tasks, a multi-level autonomous ground intelligent detection capability with strong timeliness and high flexibility is formed. While expanding the boundaries of satellite-borne intelligent computing capabilities, it also significantly improves the accessibility of remote sensing satellite platforms to ordinary users.

[0105] Accordingly, the present application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned earth-ground collaborative remote sensing intelligent system based on a satellite network constellation. Figure 4 As shown, it is a hardware structure diagram of a device with data processing capability in a satellite network constellation-based earth-ground collaborative remote sensing intelligent system provided by an embodiment of the present invention, except Figure 4 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0106] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by the processor, a space-ground collaborative remote sensing intelligent system based on a satellite networking constellation as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0107] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0108] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A space-ground collaborative remote sensing intelligent system based on a satellite network constellation, characterized in that: It includes a ground station and a satellite networking constellation, wherein the satellite networking constellation includes a plurality of satellites, and is divided into link-building satellites and non-link-building satellites according to the link-building conditions with the ground station; In response to user input or sudden disaster events, the ground station generates a command file using a large ground model and uploads it to the link-building satellite; In response to the command file, the link-building satellite parses the command file and determines whether it is the primary satellite. If not, the command file is transmitted to the primary satellite, and the primary satellite generates a corresponding mission plan using the onboard large model. In response to the mission planning, the lead satellite generates corresponding calculation requests and / or shooting requests based on the computing power occupancy, energy consumption status, remote sensing equipment type and satellite position inclination of itself and other satellites, sends them to the corresponding other satellites and receives the returned calculation results and / or shooting results, generates a result file based on all the calculation results and / or shooting results, and transmits the result file to the ground station through the link-building satellite; In response to the result file, the ground station parses and visualizes the result file.

2. The system according to claim 1, characterized in that The ground station and the satellite networking constellation communicate through the space distributed file system, and the satellites in the satellite networking constellation communicate through inter-satellite handshake.

3. The system according to claim 1, characterized in that If the instruction input by the user is an invalid instruction, the ground station uses the ground large model to encode the invalid instruction into a vector, matches the vector with the vector in the satellite remote sensing image library of the ground station, and generates several recommended instructions based on the satellite remote sensing image information corresponding to several vectors with the highest similarity. In response to the valid instruction input by the user or the recommended instruction selected by the user, the ground station uses the ground large model to parse the instruction, extract key information and pre-process it, thereby generating an instruction file and uploading it to the link establishment satellite.

4. The system according to claim 1, characterized in that In response to a sudden disaster event, the ground station uses a large ground model to analyze the sudden disaster event, extract key information and pre-process it, generate an instruction file for monitoring the sudden disaster event, and upload it to the link-building satellite.

5. The system according to claim 1, characterized in that The parameters of the onboard large model include base model parameters and LoRA increment parameters, and the onboard large model of each satellite in the satellite networking constellation is updated, including: The ground station updates the pre-saved copy of the onboard large model, solidifies the updated model capability to the LoRA incremental parameter part through the model distillation technology, quantifies the parameters of this part, forms a parameter update file, and uploads the parameter update file to the link-building satellite; The link-building satellite synchronizes the parameter update file to all satellites in the satellite networking constellation; Each satellite implements an on-orbit model update based on the fixed base model parameters and the updated LoRA incremental parameters according to the parameter update file.

6. The system according to claim 5, characterized in that Each satellite conducts on-orbit testing and transmits the test results to the ground station, which is manually reviewed and selected to roll back the on-orbit model parameters or confirm the on-orbit model update.

7. The system according to claim 1, characterized in that The ground-based large model and the satellite-borne large model are large language models or multimodal large models including large language models.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the system according to any one of claims 1 to 7 is implemented.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the system according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the system according to any one of claims 1 to 7 is implemented.

Images