An interactive situation awareness resource perception and planning method and system

Abstract

The application belongs to the technical field of detection resource planning, and specifically discloses an interactive situation detection resource sensing and planning method and system, which comprises the following steps: collecting and organizing the detection resources inside and outside the cluster in real time; acquiring the situation information of a detection task and sensing the target state information of a detection target in the detection task in real time according to the situation information; calculating and determining the various relationship information of the detection node and the detection target according to the node state information and the target state information; combining the detection task, the node state information, the target state information and the relationship information with a user input instruction and a first cooperative request as input information, and inputting the input information into a multimodal large model for task understanding; and calling multiple different professional models in parallel to generate a planning scheme according to the task type of the detection task by using the multimodal large model, so that the flexibility and efficiency of the execution of the detection task can be improved.

Description

Technical Field

[0001] This application belongs to the field of exploration resource planning technology, and more specifically, relates to an interactive situational awareness and planning method and system for resource perception. Background Technology

[0002] When unmanned platform swarms such as drones or unmanned surface vessels are performing detection or search and rescue missions, in order to achieve timely detection and real-time tracking of targets or target groups, it is necessary to control the swarm's detection resources in real time and quickly plan economical and feasible resource detection schemes.

[0003] Currently, methods applied to resource planning for exploration can be divided into two categories. One category employs traditional algorithms, such as dynamic programming and heuristic search, to solve resource planning. This type of method requires feature engineering, first abstracting the model's input to a higher order, and then substituting it into the corresponding mathematical and algorithmic models for solution. This type of method has advantages such as rapid solution and stable results. However, it heavily relies on expert experience and is often based on statistical assumptions, leading to solutions that are not optimal or sometimes infeasible. The other category is based on deep learning methods. This type of method is more intelligent, belonging to data-driven "end-to-end" analysis methods, possessing the ability to autonomously identify tasks and situations, and often employing reinforcement learning. This type of method takes tasks and states as input in a standardized format and directly outputs a scheduling plan. However, this type of method often cannot allow for manual intervention in the plan, and the model processing efficiency is low when facing highly dynamic task environments. Meanwhile, most current planning schemes only consider controllable nodes within the cluster, neglecting collaboration between clusters.

[0004] In summary, in unmanned collaborative scenarios, existing deep learning methods suffer from drawbacks such as difficulty in human intervention, low learning efficiency, and neglect of external collaborative nodes when detecting or rescuing targets, resulting in poor flexibility and efficiency in task execution. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this application is to provide an interactive situational awareness and planning method for resource perception, which aims to solve the problem of poor flexibility and efficiency in the execution of detection tasks.

[0006] Firstly, this application provides an interactive situational awareness and planning method, including:

[0007] Real-time collection and organization of detection resources inside and outside the cluster;

[0008] Acquire situational information of the detection mission and perceive the target status information of the detection target in the detection mission in real time based on the situational information;

[0009] Based on node status information and target status information, various relationship information between the detection nodes and the detection targets is calculated and determined;

[0010] The detection task, node status information, target status information, and relationship information are combined with user input instructions and the first collaboration request as input information, and the input information is input into the multimodal large model for task understanding;

[0011] The multimodal large model is used to call multiple different professional models in parallel to generate planning schemes according to the mission type of the exploration mission, and the large model is used to control the deduction and optimization of each planning scheme.

[0012] Optionally, a multimodal large model is used to call multiple different specialized models in parallel to generate planning schemes based on the task type of the exploration mission, and the large model is used to control the derivation and optimization of each planning scheme, including:

[0013] By combining the first collaborative request with other cluster nodes, and based on the principles of internal resource coordination and external resource operation and requests, the multimodal large model is used to call widely used multi-type dynamic programming, heuristic search and deep reinforcement learning professional models in parallel, thereby generating a variety of different alternative planning schemes.

[0014] Using a prompting learning approach with a large model, the planning schemes generated by various professional models are deduced from multiple dimensions such as schedulability, economy, maintainability, and efficiency. The schemes are ranked according to the deduction scores, and the optimal scheme is recommended.

[0015] The first collaboration request is a collaboration request for standardized probe resources from other clusters or nodes.

[0016] Optionally, it also includes:

[0017] The second collaboration request is output through the multimodal large model. The second collaboration request is a collaboration request to other visible clusters or nodes. The second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

[0018] Optionally, the node status information is collected in real time from detection nodes inside and outside the cluster nodes at a fixed time period; the node status information includes time and location information, detection method, detection range, performance parameters, and driving range information.

[0019] Optionally, the target status information of the detection mission is perceived in real time based on the situational information, including:

[0020] Real-time sensing and detection of the target's time location information and multimodal information according to a fixed time period;

[0021] Based on node status information and target status information, various relationship information between the probe nodes and the probe targets is determined, including:

[0022] Based on the time and location information of the detection nodes and the detection targets, the distance and detection capability matching degree between the detection nodes and the detection targets are determined.

[0023] Optionally, the input information is input into a multimodal large model for task understanding, including:

[0024] The input information is input into a multimodal large model for unified encoding to obtain an encoding vector;

[0025] The encoded vector is compressed and adjusted using the multi-layer neural network encoder of the multimodal large model to obtain compressed features;

[0026] Based on the analysis of the compressed feature model, task understanding of multimodal large models can be achieved.

[0027] Optionally, the method further includes:

[0028] Receive the user's first input, which is a natural language intervention task instruction;

[0029] By combining the compressed features with natural language intervention tasks, the task understanding of the multimodal large model can be achieved.

[0030] Secondly, this application also provides an interactive situational awareness and planning system, comprising:

[0031] The node information acquisition module is used to acquire node status information of probe nodes inside and outside the cluster nodes;

[0032] The perception module is used to acquire situational information of the detection mission and perceive the target status information of the detection target in the detection mission in real time based on the situational information.

[0033] The relationship information acquisition module is used to determine various relationship information between the probe nodes and the probe targets based on node status information and target status information;

[0034] The model understanding module is used to combine the detection task, node status information, target status information, and relationship information with user input instructions and the first collaboration request as input information, and input the input information into the multimodal large model for task understanding;

[0035] The scheme generation module is used to call multiple different professional models based on the task type of the exploration mission using a multimodal large model, generate planning schemes in parallel based on the professional models, and select the optimal scheme from the planning schemes.

[0036] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0037] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0038] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0039] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0040] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0041] (1) This application embodiment, by collecting and fusing the state information of detection nodes and targets in real time, can quickly adapt to environmental changes and improve the flexibility of detection tasks. Utilizing a multimodal large model for task understanding enables a more accurate understanding of complex detection tasks, improving the accuracy and efficiency of task execution. By generating planning schemes through parallel invocation of multiple professional models, multiple feasible schemes can be generated in a short time, improving the efficiency of detection tasks. By comprehensively considering the feasibility, economy, and collaborative requests of the schemes, the optimal scheme can be selected, improving the execution efficiency of detection tasks.

[0042] (2) Unlike traditional machine learning and deep learning methods, the interactive scheme of this application can give full play to the guiding role of humans in the situational awareness and planning process. By utilizing the ability of large models to understand the first external collaborative request in the planning stage, as well as their simulation and deduction capabilities in the scheme optimization stage, the flexibility and efficiency of the detection mission can be improved.

[0043] (3) The real-time perception of situational information and analysis of various relationship information in this application can improve the accuracy of target identification and the early warning capability of potential threats.

[0044] (4) This application can quickly adapt to different task requirements by introducing a multimodal large model, and generate flexible solutions that can cope with various situations through the collaborative work of professional models. Attached Figure Description

[0045] Figure 1 This is one of the flowcharts illustrating the interactive situational awareness and planning method provided in this application embodiment;

[0046] Figure 2 This is the second flowchart illustrating the interactive situational awareness and planning method provided in this application embodiment;

[0047] Figure 3 This is a schematic diagram of the structure of the interactive situational awareness, resource perception, and planning system provided in the embodiments of this application;

[0048] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0051] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0052] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0053] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0054] Next, the technical solutions provided in the embodiments of this application will be described.

[0055] Reference Figure 1 This application provides an interactive situational awareness and planning method, including:

[0056] S101. Real-time collection and organization of detection resources inside and outside the cluster;

[0057] S102. Acquire situational information of the detection mission and perceive the target status information of the detection targets in the detection mission in real time based on the situational information;

[0058] S103. Calculate and determine various relationship information between the detection nodes and the detection targets based on the node status information and the target status information;

[0059] S104. Combine the detection task, node status information, target status information, and relationship information with user input instructions and the first cooperation request as input information, and input the input information into the multimodal large model for task understanding;

[0060] S105. Utilize a multimodal large model to call multiple different specialized models in parallel to generate planning schemes based on the mission type of the exploration task, and use the large model to control the deduction and optimization of each planning scheme.

[0061] First, the detection nodes collect internal and external information in real time through sensors, monitoring equipment, or network interfaces, including their own operational status (such as battery level, location, sensor status, etc.) and environmental conditions (such as weather conditions, obstacle detection, etc.). Then, the collected status information is sent to the cluster management system to ensure that the status information of all nodes is integrated and shared.

[0062] Furthermore, the node status information is collected in real time from the detection nodes inside and outside the cluster nodes at fixed time intervals; the node status information includes time and location information, detection methods, detection range, performance parameters, and remaining range information.

[0063] Information from different detection missions and node status is fused to generate high-quality situational awareness. Based on this fused situational awareness, the temporal location and multimodal information of the detection targets are perceived in real time at fixed time intervals, thereby enabling real-time perception of the target status of the detection mission.

[0064] Based on situational awareness, various relationship information between the detection nodes and the detection targets is determined, including target distance and detection capability matching degree. That is, based on the time and location information of the detection nodes and the detection targets, the distance and detection capability matching degree between the detection nodes and the detection targets are determined.

[0065] The detection task, node status information, target status information, and relationship information are combined with user instructions and the first collaboration request to form an input set containing multidimensional data. To ensure that the data format and features can be effectively processed by the subsequent multimodal large model, in some embodiments, this input set may require preprocessing and standardization.

[0066] The constructed input set is fed into a multimodal large-scale model, which performs task understanding, target recognition, and dynamic decision-making through deep learning. Based on the type of exploration task, an appropriate specialized model is automatically selected for subsequent solution generation. Multiple specialized models generate additional solutions for specific exploration tasks, considering different strategies and methods. By calling multiple models to generate different solutions, the diversity of generated solutions is ensured to adapt to different task requirements and environmental conditions.

[0067] Finally, the generated planning schemes are evaluated, taking into account multiple dimensions such as schedulability, economy, maintainability, and efficiency, to generate evaluation scores. The best scheme is selected from the evaluations and fine-tuned as needed to ensure it adapts to current task requirements and environmental changes.

[0068] This application's embodiments, by collecting and fusing state information from detection nodes and targets in real time, can quickly adapt to environmental changes and improve the flexibility of detection tasks. Utilizing a multimodal large model for task understanding enables a more accurate understanding of complex detection tasks, improving the accuracy and efficiency of task execution. By generating planning schemes through parallel invocation of multiple specialized models, multiple feasible solutions can be generated in a short time, improving the efficiency of detection tasks. By comprehensively considering the feasibility, economy, and coordination requests of the schemes, the optimal solution can be selected, further improving the execution efficiency of detection tasks.

[0069] The interactive situational awareness and planning method provided in this application significantly improves the flexibility and efficiency of the detection mission through real-time data fusion, the application of multimodal large models, parallel processing of professional models, selection of optimal solutions, and the implementation of manual intervention.

[0070] Optionally, a multimodal large model is used to call multiple different specialized models in parallel to generate planning schemes based on the task type of the exploration mission, and the large model is used to control the derivation and optimization of each planning scheme, including:

[0071] Combining the first collaborative request with other cluster nodes, and based on the principles of internal resource coordination and external resource operation and requests, the multimodal large model is used to call the dynamic programming model and the reinforcement learning model in parallel to generate planning schemes corresponding to different models.

[0072] The planning schemes generated by each professional model are scored and ranked from multiple dimensions, including schedulability, economy, maintainability, and efficiency, and the preferred scheme is selected based on the score ranking.

[0073] The first collaboration request is a collaboration request for standardized probe resources from other clusters or nodes.

[0074] Specifically, in this embodiment, the first collaborative request comes from other clusters or nodes, and the specific information includes the quantity and type of normalized detection resources and the required task characteristics (e.g., detection time, range, accuracy, etc.).

[0075] Based on the nature of the requested detection task, a multimodal large model is invoked, and a dynamic programming model and a reinforcement learning model are integrated on top of it. The specific steps are as follows:

[0076] Dynamic programming models are mainly used to solve resource allocation and task allocation problems. By constructing a state space and an action space, they provide the optimal allocation path for each resource.

[0077] Reinforcement learning models learn and adjust their strategies through interaction with the environment to maximize long-term rewards, enabling the model to adapt to dynamically changing environments and needs.

[0078] To ensure that the selected solution is optimal, the generated planning solutions need to be scored and ranked from multiple dimensions:

[0079] Scheduling: Evaluating whether a plan can be scheduled to be completed within a given timeframe, taking into account factors such as resource availability and task urgency.

[0080] Economic efficiency: Analyze the resource costs of the proposed solution, including time costs, actual resource consumption, and potential benefits.

[0081] Maintainability: Evaluate the solution's response speed, maintenance difficulty, and cost in the event of a failure.

[0082] Efficiency: Evaluate the efficiency of the scheme in resource utilization and scheduling execution during actual exploration missions.

[0083] Based on the scores, the solution with the highest score is selected as the preferred solution. This process requires comprehensive consideration of trade-offs between different objectives, such as finding the optimal balance between schedulability, economy, maintainability, and efficiency.

[0084] This application's embodiments ensure optimal resource utilization through effective internal coordination and external resource requests. A first collaborative request mechanism promotes cooperation between clusters, effectively integrating resources from all parties. The combination of dynamic programming and reinforcement learning ensures rapid response and effective solutions regardless of task changes. Monitoring and feedback mechanisms enable the system to dynamically adjust based on actual conditions, improving task execution effectiveness and flexibility.

[0085] This application's embodiments not only achieve an effective combination of multimodal large models and specialized models, but also ensure the efficient execution of exploration missions through meticulous resource management and comprehensive scheme evaluation. It can flexibly respond to mission changes, optimize resource allocation, and achieve intelligent collaborative exploration of targets.

[0086] Optionally, it also includes:

[0087] The second collaboration request is output through the multimodal large model. The second collaboration request is a collaboration request to other visible clusters or nodes. The second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

[0088] Specifically, the second collaboration request in this embodiment is based on the output of a multimodal large model and aims to enhance resource coordination and task collaboration among clusters or nodes. It is mainly used to request support from other nodes to achieve more efficient resource utilization and task execution.

[0089] The second collaboration request in this application is a collaboration request to other visible clusters or nodes, that is, other identifiable clusters or nodes in the current cluster that can provide the necessary resources or support. Visibility is generally determined based on information such as network connectivity, task requirements, and resource status.

[0090] Leveraging the capabilities of multimodal large models, data from diverse sources can be processed, including sensor information, task descriptions, and resource status. The model analyzes this information to generate a second collaborative request, ensuring the accuracy and appropriateness of the request.

[0091] Through the second coordination request, other nodes can quickly identify their own schedulable resources and optimize resource allocation. This enables collaborative task execution; for example, different clusters or nodes can work together to complete a complex probing task. In this process, coordinated actions are rationally allocated, improving the overall efficiency of task completion.

[0092] Optionally, the input information is input into a multimodal large model for task understanding, including:

[0093] The input information is input into a multimodal large model for unified encoding to obtain an encoding vector;

[0094] The encoded vector is compressed and adjusted using the multi-layer neural network encoder of the multimodal large model to obtain compressed features;

[0095] Based on the analysis of the compressed feature model, task understanding of multimodal large models can be achieved.

[0096] Furthermore, the method also includes:

[0097] Receive the user's first input, which is a natural language intervention task instruction;

[0098] By combining the compressed features with natural language intervention tasks, the task understanding of the multimodal large model can be achieved.

[0099] In this embodiment, various input information is first fed into a multimodal large model for unified encoding. This encoding stage converts data from different modalities into a unified representation, typically a high-dimensional encoding vector. This process ensures information consistency, enabling multimodal features to be compared and analyzed on the same dimension.

[0100] The generated encoded vectors are processed using a multi-modal, large-scale, multi-layer neural network encoder. Through the network's multi-layered structure, higher-level features can be extracted, which can more effectively express the core semantics of the input information.

[0101] The encoder compresses and adjusts the encoded vector through nonlinear transformations and dimensionality reduction operations to generate compressed features. These compressed features are typically low-dimensional and more representative, retaining the key information of the original input while eliminating redundancy.

[0102] By utilizing the generated compressed features, the model performs task understanding, identifying the task objectives and requirements hidden within the input data. These compressed features can then be further fed into deep learning models, such as classifiers or regression models, for specific task understanding analysis. For example, they can determine the category of the input task or predict the task's execution outcome.

[0103] Furthermore, by introducing user interaction, the system can receive initial user input, typically in the form of natural language instructions for the intervention task. Users can describe their needs and provide specific task instructions using natural language. Combining compressed features with the user's natural language instructions further enhances the model's ability to understand the task. This process is usually achieved through text embedding and feature fusion techniques, ensuring that the user's intent is effectively integrated into the model's task understanding process. Incorporating user input provides additional contextual information, helping large multimodal models to more accurately understand and execute tasks.

[0104] This application utilizes a multimodal large model, combining unified encoding of input information, extraction of compressed features, and user natural language commands to effectively understand complex tasks. This not only improves the model's intelligence level but also enhances the interaction between the user and the system, making the detection task execution more flexible and efficient.

[0105] Reference Figure 2 The complete flowchart of this application includes:

[0106] S1. Real-time collection of resource status both inside and outside the cluster;

[0107] S2. Real-time sensing and detection of the target status;

[0108] S3. Calculate the relationship between exploration resources and mission objectives;

[0109] S4. Combine artificial natural language intervention to understand large-scale model planning tasks;

[0110] S5. Large-scale model control allows multiple specialized models to generate planning schemes in parallel;

[0111] S6. Combine collaborative request intervention to deduce and optimize large-scale model control planning schemes.

[0112] Reference Figure 3 This application also provides an interactive situational awareness and planning system, comprising:

[0113] The node information acquisition module 310 is used to acquire node status information of probe nodes inside and outside the cluster nodes;

[0114] The sensing module 320 is used to acquire situational information of the detection task and perceive the target status information of the detection target in the detection task in real time based on the situational information.

[0115] The relationship information acquisition module 330 is used to determine various relationship information between the detection node and the detection target based on the node status information and the target status information;

[0116] The model understanding module 340 is used to combine the detection task, node status information, target status information, and relationship information with user input instructions and the first collaboration request as input information, and input the input information into the multimodal large model for task understanding;

[0117] The scheme generation module 350 is used to call multiple different professional models according to the task type of the detection task using a multimodal large model, generate planning schemes in parallel based on the professional models, and select the preferred scheme from each planning scheme.

[0118] Optionally, a multimodal large model is used to call multiple different specialized models in parallel to generate planning schemes based on the task type of the exploration mission, and the large model is used to control the derivation and optimization of each planning scheme, including:

[0119] Combining the first collaborative request with other cluster nodes, and based on the principles of internal resource coordination and external resource operation and requests, the multimodal large model is used to call the dynamic programming model and the reinforcement learning model in parallel to generate planning schemes corresponding to different models.

[0120] The planning schemes generated by each professional model are scored and ranked from multiple dimensions, including schedulability, economy, maintainability, and efficiency, and the preferred scheme is selected based on the score ranking.

[0121] The first collaboration request is a collaboration request for standardized probe resources from other clusters or nodes.

[0122] Optionally, it also includes a collaboration request module for:

[0123] The second collaboration request is output through the multimodal large model. The second collaboration request is a collaboration request to other visible clusters or nodes. The second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

[0124] Optionally, the node status information is collected in real time from detection nodes inside and outside the cluster nodes at a fixed time period; the node status information includes time and location information, detection method, detection range, performance parameters, and driving range information.

[0125] Optionally, the target status information of the detection mission is perceived in real time based on the situational information, including:

[0126] Real-time sensing and detection of the target's time location information and multimodal information according to a fixed time period;

[0127] Based on node status information and target status information, various relationship information between the probe nodes and the probe targets is determined, including:

[0128] Based on the time and location information of the detection nodes and the detection targets, the distance and detection capability matching degree between the detection nodes and the detection targets are determined.

[0129] Optionally, the input information is input into a multimodal large model for task understanding, including:

[0130] The input information is input into a multimodal large model for unified encoding to obtain an encoding vector;

[0131] The encoded vector is compressed and adjusted using the multi-layer neural network encoder of the multimodal large model to obtain compressed features;

[0132] Based on the analysis of the compressed feature model, task understanding of multimodal large models can be achieved.

[0133] Optionally, it also includes an intervention module for:

[0134] Receive the user's first input, which is a natural language intervention task instruction;

[0135] By combining the compressed features with natural language intervention tasks, the task understanding of the multimodal large model can be achieved.

[0136] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0137] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0138] Reference Figure 4 Based on the methods in the above embodiments, this application provides an electronic device that may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions stored in the memory 430 to execute the methods in the above embodiments.

[0139] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0140] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0141] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0142] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0143] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0144] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0145] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0146] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An interactive situational awareness and planning method, characterized in that, include: Real-time collection and organization of detection resources inside and outside the cluster; Acquire situational information of the detection mission and perceive the target status information of the detection target in the detection mission in real time based on the situational information; Based on node status information and target status information, various relationship information between the detection nodes and the detection targets is calculated and determined; The detection task, node status information, target status information, and relationship information are combined with user input instructions and the first collaboration request as input information, and the input information is input into the multimodal large model for task understanding; By combining the first collaborative request with other cluster nodes, and based on the principles of internal resource coordination and external resource operation and requests, the multimodal large model is used to call widely used multi-type dynamic programming, heuristic search and deep reinforcement learning professional models in parallel, thereby generating a variety of different alternative planning schemes. Using a prompting learning approach, a large model is employed to deduce the planning schemes generated by various professional models from multiple dimensions, including schedulability, economy, maintainability, and efficiency. The schemes are ranked according to the deduction scores, and the preferred scheme is recommended. The first collaboration request is a collaboration request for standardized probe resources from other clusters or nodes.

2. The situational awareness and planning method according to claim 1, characterized in that, Also includes: The second collaboration request is output through the multimodal large model. The second collaboration request is a collaboration request to other visible clusters or nodes. The second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

3. The situational awareness and planning method according to claim 1, characterized in that, The node status information is collected in real time from internal and external detection nodes of the cluster at fixed time intervals; the node status information includes time and location information, detection methods, detection range, performance parameters, and remaining range information.

4. The situation detection resource perception and planning method according to claim 1, characterized in that, Based on the situational information, the target status information of the detection mission is perceived in real time, including: Real-time sensing and detection of the target's time location information and multimodal information according to a fixed time period; Based on node status information and target status information, various relationship information between the probe nodes and the probe targets is determined, including: Based on the time and location information of the detection nodes and the detection targets, the distance and detection capability matching degree between the detection nodes and the detection targets are determined.

5. The situational awareness and planning method according to claim 1, characterized in that, The input information is fed into a multimodal large model for task understanding, including: The input information is input into a multimodal large model for unified encoding to obtain an encoding vector; The encoded vector is compressed and adjusted using the multi-layer neural network encoder of the multimodal large model to obtain compressed features; Based on the analysis of the compressed feature model, task understanding of multimodal large models can be achieved.

6. The situation detection resource perception and planning method according to claim 5, characterized in that, The method further includes: Receive the user's first input, which is a natural language intervention task instruction; By combining the compressed features with natural language intervention tasks, the task understanding of the multimodal large model can be achieved.

7. An interactive situational awareness, resource perception, and planning system, characterized in that, include: The node information acquisition module is used to acquire node status information of probe nodes inside and outside the cluster nodes; The perception module is used to acquire situational information of the detection mission and perceive the target status information of the detection target in the detection mission in real time based on the situational information. The relationship information acquisition module is used to determine various relationship information between the probe nodes and the probe targets based on node status information and target status information; The model understanding module is used to combine the detection task, node status information, target status information, and relationship information with user input instructions and the first collaboration request as input information, and input the input information into the multimodal large model for task understanding; The scheme generation module is used to combine the first collaborative request with other cluster nodes, and based on the principles of internal resource coordination and external resource operation and request, it uses a multimodal large model to call widely used multi-type dynamic programming, heuristic search and deep reinforcement learning professional models in parallel, thereby generating a variety of different alternative planning schemes. Using a prompting learning approach, a large model is employed to deduce the planning schemes generated by various professional models from multiple dimensions, including schedulability, economy, maintainability, and efficiency. The schemes are ranked according to the deduction scores, and the preferred scheme is recommended. The first collaboration request is a collaboration request for standardized probe resources from other clusters or nodes.

8. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-6.

Images