Interactive situation detection resource awareness and planning method and system

Abstract

The invention belongs to the technical field of exploration resource planning, and particularly discloses an interactive situation exploration resource awareness and planning method and system, and the method comprises the steps: collecting and organizing exploration resources inside and outside a cluster in real time; acquiring situation information of the detection task and sensing target state information of a detection target in the detection task in real time according to the situation information; calculating and determining various relation information of the detection node and the detection target according to the node state information and the target state information; the detection task, the node state information, the target state information and the relation information are combined with a user input instruction and a first cooperation request to serve as input information, and the input information is input into a multi-mode large model for task understanding; the multi-modal large model is utilized to call a plurality of different professional models according to the task type of the detection task to generate the planning scheme in parallel, and the flexibility and efficiency of detection task execution can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of detection resource planning, and more specifically, to an interactive situation detection resource perception and planning method and system. Background Art

[0002] When a cluster of unmanned platforms such as drones or unmanned boats performs detection or search and rescue missions, in order to achieve timely discovery and real-time tracking of targets or target groups, it is necessary to control the cluster's detection resources in real time and quickly plan economically feasible resource detection plans.

[0003] The methods currently used in exploration resource planning can be divided into two categories. One category uses traditional algorithms represented by dynamic programming and heuristic search to solve the planning of exploration resources. This category of methods requires the use of feature engineering methods. First, the input of the model is abstracted at a high level, and then the corresponding mathematical and algorithmic models are introduced for solution. This category of methods has the advantages of rapid solution and stable results. However, this category of methods relies heavily on expert experience and is often based on statistical assumptions, which leads to the fact that the solutions derived by this category of methods are not optimal solutions, or the solutions given are sometimes infeasible. The other category is based on deep learning methods. This category of methods is more intelligent and belongs to data-driven "end-to-end" analysis methods. It has the ability to autonomously identify tasks and situations, and mostly adopts reinforcement learning. This category of methods inputs tasks and states in a standardized format and directly outputs scheduling plans. However, this category of methods often cannot achieve manual intervention in the plan, and the efficiency of model processing is low when facing a highly dynamic task environment. At the same time, most of the current plan planning only considers the internal nodes of the cluster that can be controlled, ignoring the collaboration between clusters.

[0004] In summary, in unmanned collaborative scenarios, existing deep learning methods have the defects of difficulty in manual intervention, low learning efficiency, and neglect of collaborative nodes outside the cluster when detecting or rescuing targets, resulting in poor flexibility and efficiency of task execution. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of this application is to provide an interactive situation detection resource perception and planning method, aiming to solve the problems of poor flexibility and efficiency in the execution of detection tasks.

[0006] In a first aspect, the present application provides an interactive situation detection resource perception and planning method, comprising:

[0007] Collect and organize detection resources inside and outside the cluster in real time;

[0008] Acquire situation information of the detection task and perceive target state information of the detection target in the detection task in real time according to the situation information;

[0009] Calculate and determine various relationship information between detection nodes and detection targets based on node status information and target status information;

[0010] Taking the detection task, node status information, target status information, relationship information, user input instructions, and the first collaborative request as input information, and inputting the input information into the multimodal large model for task understanding;

[0011] A multimodal large model is used to call multiple different professional models to generate planning schemes in parallel according to the task type of the detection task, and the large model is used to control the deduction and optimization of each planning scheme.

[0012] Optionally, a multimodal large model is used to call multiple different professional models in parallel to generate planning schemes according to the task type of the detection task, and the large model is used to control the deduction and optimization of each planning scheme, including:

[0013] Combined with the first collaborative request, interact with other cluster nodes, and based on the principles of internal resource coordination and external resource operation and request, use a multi-modal large model to call widely used multi-type dynamic programming, heuristic search and deep reinforcement learning professional models in parallel, so as to generate a variety of different alternative planning plans;

[0014] The large model is used in a prompt learning manner to deduce the planning schemes generated by various professional models from multiple dimensions such as schedulability, economy, maintainability and efficiency. The schemes are ranked according to the deduction scores and the optimal scheme is recommended.

[0015] The first collaborative request is a collaborative request for standardized detection resources from other clusters or nodes.

[0016] Optionally, it also includes:

[0017] A second collaboration request is output through the multimodal large model, where the second collaboration request is a collaboration request for other visible clusters or nodes, and the second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

[0018] Optionally, the node status information is collected in real time from detection nodes inside and outside the cluster node according to a fixed time period; the node status information includes time location information, detection means, detection range, performance parameters and cruising range information.

[0019] Optionally, sensing target state information of the detection task in real time according to the situation information includes:

[0020] Real-time perception of the time position information and multi-modal information of the detection target according to a fixed time period;

[0021] According to the node status information and the target status information, various relationship information between the detection node and the detection target is determined, including:

[0022] According to the time position information of the detection node and the time position information of the detection target, the distance and detection capability matching degree between the detection node and the detection target are determined.

[0023] Optionally, inputting the input information into a multimodal large model for task understanding includes:

[0024] Inputting the input information into a multimodal large model for unified encoding to obtain an encoding vector;

[0025] Using the multi-layer neural network encoder of the multimodal large model, compressing and adjusting the encoding vector to obtain a compression feature;

[0026] Based on the compressed feature model analysis, task understanding of multimodal large models can be achieved.

[0027] Optionally, the method further comprises:

[0028] receiving a first input from a user, wherein the first input is a natural language intervention task instruction;

[0029] The compressed features are combined with natural language intervention tasks to achieve task understanding of the multimodal large model.

[0030] In a second aspect, the present application also provides an interactive situation detection resource perception and planning system, including:

[0031] The node information acquisition module is used to obtain the node status information of the detection nodes inside and outside the cluster node;

[0032] A perception module is used to obtain situation information of the detection task and perceive the target state information of the detection target in the detection task in real time according to the situation information;

[0033] A relationship information acquisition module is used to determine various relationship information between the detection node and the detection target according to the node status information and the target status information;

[0034] A model understanding module, used to combine the detection task, node status information, target status information, relationship information with the user input instruction and the first collaborative request as input information, and input the input information into the multimodal large model for task understanding;

[0035] The scheme generation module is used to use the multimodal large model to call multiple different professional models according to the task type of the detection task, generate planning schemes in parallel based on the professional models, and select the preferred scheme from each of the planning schemes.

[0036] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0038] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0039] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0040] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0041] (1) The embodiments of the present application can quickly adapt to environmental changes and improve the flexibility of detection tasks by collecting and fusing the status information of detection nodes and targets in real time. The use of multimodal large models for task understanding can more accurately understand complex detection tasks and improve the accuracy and efficiency of task execution. By calling multiple professional models in parallel to generate planning plans, multiple feasible plans can be generated in a short time, thereby improving the efficiency of detection tasks. By comprehensively considering the feasibility, economy and collaborative requests of the plans, the optimal plan can be selected to improve the execution efficiency of the detection tasks.

[0042] (2) This application is different from traditional machine learning and deep learning methods. The interactive solution of this application can give full play to the guiding role of people in the situational awareness and planning process, and by utilizing the big model's ability to understand the external first coordination request in the planning stage, and its simulation and deduction capabilities in the solution optimization stage, it can improve the flexibility and efficiency of the detection task execution.

[0043] (3) The real-time perception of situation information and analysis of multiple relationship information of the present application can improve the accuracy of target identification and the early warning capability of potential threats.

[0044] (4) This application can quickly adapt to different task requirements through the introduction of a multimodal large model, and generate flexible solutions that can cope with various situations through the collaborative work of professional models. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is one of the flow charts of the interactive situation detection resource perception and planning method provided in the embodiment of the present application;

[0046] Figure 2 This is the second flow chart of the interactive situation detection resource perception and planning method provided in the embodiment of the present application;

[0047] Figure 3 It is a structural diagram of an interactive situation detection resource perception and planning system provided in an embodiment of the present application;

[0048] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0051] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0052] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0053] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0054] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0055] Reference Figure 1 The present application provides an interactive situation detection resource perception and planning method, including:

[0056] S101. Collect and organize detection resources inside and outside the cluster in real time;

[0057] S102. Acquire situation information of the detection task and perceive the target state information of the detection target in the detection task in real time according to the situation information;

[0058] S103. Calculate and determine various relationship information between the detection node and the detection target based on the node status information and the target status information;

[0059] S104. The detection task, node status information, target status information, relationship information, user input instructions, and the first collaborative request are combined as input information, and the input information is input into the multimodal large model for task understanding;

[0060] S105. Use the multimodal big model to call multiple different professional models in parallel to generate planning schemes according to the task type of the detection task, and use the big model to control the deduction and optimization of each planning scheme.

[0061] First, the detection node collects internal and external information in real time through sensors, monitoring equipment or network interfaces, including its own operating status (such as power, location, sensor status, etc.) and environmental conditions (such as weather conditions, obstacle detection, etc.). The collected status information is then sent to the cluster management system to ensure that the status information of all nodes is integrated and shared.

[0062] Furthermore, the node status information is collected in real time from detection nodes inside and outside the cluster nodes according to a fixed time period; the node status information includes time location information, detection means, detection range, performance parameters and cruising range information.

[0063] The detection task information from different sources is integrated with the node status information to generate high-quality situation information. Based on the integrated situation information, the time position information and multi-modal information of the detection target are perceived in real time according to a fixed time period, and the target status of the detection task is perceived in real time.

[0064] On the basis of situational awareness, various relationship information between the detection node and the detection target is determined, including target distance and detection capability matching degree, etc. That is, according to the time position information of the detection node and the time position information of the detection target, the distance and detection capability matching degree between the detection node and the detection target is determined.

[0065] Combine the detection task, node status information, target status information, relationship information, user instructions, and the first collaborative request to form an input set containing multidimensional data. Ensure that the format and characteristics of the data can be effectively processed by the subsequent multimodal large model. In some embodiments, the input set may need to be preprocessed and standardized.

[0066] The constructed input set is input into the multimodal large model, which performs task understanding, target recognition, and dynamic decision-making through deep learning of information. According to the type of detection task, the appropriate professional model is automatically selected for subsequent solution generation. Multiple professional models generate additional solutions for specific detection tasks, considering different strategies and methods. By calling multiple models to generate different solutions, it ensures that the generated solutions are diversified to adapt to different task requirements and environmental conditions.

[0067] Finally, the generated multiple planning schemes are evaluated, taking into account multiple dimensions such as schedulability, economy, maintainability, and efficiency to form an evaluation score. The best scheme is selected from the evaluation and fine-tuned as needed to ensure that it adapts to current task requirements and environmental changes.

[0068] The embodiments of the present application can quickly adapt to environmental changes and improve the flexibility of detection tasks by collecting and fusing the status information of detection nodes and targets in real time. By using a large multimodal model for task understanding, complex detection tasks can be understood more accurately, and the accuracy and efficiency of task execution can be improved. By calling multiple professional models in parallel to generate planning plans, multiple feasible plans can be generated in a short time, improving the efficiency of detection tasks. By comprehensively considering the feasibility, economy and collaborative requests of the plan, the optimal plan can be selected to improve the execution efficiency of the detection task.

[0069] The interactive situation detection resource perception and planning method provided in the embodiment of the present application significantly improves the flexibility and efficiency of the detection task through real-time data fusion, application of multimodal large models, parallel processing of professional models, selection of preferred solutions and implementation of manual intervention.

[0070] Optionally, a multimodal large model is used to call multiple different professional models in parallel to generate planning schemes according to the task type of the detection task, and the large model is used to control the deduction and optimization of each planning scheme, including:

[0071] In combination with the first collaborative request, the nodes in the other clusters are interacted with, and based on the principles of internal resource coordination and external resource operation and request, a multimodal large model is used to call a dynamic programming model and a reinforcement learning model in parallel to generate planning schemes corresponding to different models;

[0072] Score and rank the planning schemes generated by each professional model from multiple dimensions such as schedulability, economy, maintainability and efficiency, and select the preferred scheme based on the score ranking;

[0073] The first collaborative request is a collaborative request for standardized detection resources from other clusters or nodes.

[0074] Specifically, the first collaborative request in this embodiment comes from other clusters or nodes, and the specific information includes the quantity and type of standardized detection resources and the required task characteristics (for example, detection time, range, accuracy, etc.).

[0075] According to the nature of the requested detection task, the multimodal large model is called, and the dynamic programming model and reinforcement learning model are integrated on it. The specific steps are as follows:

[0076] Dynamic programming model: mainly used to solve resource allocation and task allocation problems. By constructing state space and action space, it provides the optimal allocation path for each resource.

[0077] Reinforcement learning model: Through interaction with the environment, the model continuously learns and adjusts strategies to maximize long-term rewards, enabling the model to adapt to dynamically changing environments and demands.

[0078] In order to ensure that the selected plan is the best, the generated planning plans need to be scored and ranked from multiple dimensions:

[0079] Schedulability: Evaluate whether the solution can be scheduled and completed within a given time frame, taking into account factors such as resource availability and task urgency.

[0080] Economic feasibility: Analyze the resource cost of the solution, including time cost, actual resource consumption and the potential benefits it brings.

[0081] Maintainability: Evaluate the solution’s response speed, maintenance difficulty, and cost when a failure occurs.

[0082] Efficiency: Evaluate the efficiency of the solution in resource utilization and scheduling execution in actual detection tasks.

[0083] According to the above score ranking, the scheme with the highest score is selected as the preferred scheme. This process requires comprehensive consideration of the trade-offs between different objectives, such as finding the best balance between schedulability, economy, maintainability and efficiency.

[0084] The embodiment of the present application ensures that resources are used optimally through effective internal coordination and external resource requests. Through the first collaborative request mechanism, collaboration between clusters is promoted and resources from all parties are effectively integrated. The combination of dynamic programming and reinforcement learning ensures that no matter how the task changes, it can respond quickly and provide effective solutions. The monitoring and feedback mechanism enables the system to dynamically adjust according to actual conditions, improving the execution effect and flexibility of the task.

[0085] The embodiment of the present application not only realizes the effective combination of multi-modal large models and professional models, but also ensures the efficient execution of detection tasks through fine resource management and comprehensive solution evaluation. It can flexibly respond to task changes, optimize resource allocation, and realize intelligent collaborative detection goals.

[0086] Optionally, it also includes:

[0087] A second collaboration request is output through the multimodal large model, where the second collaboration request is a collaboration request for other visible clusters or nodes, and the second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

[0088] Specifically, the second collaboration request in the embodiment of the present application is based on the output of the multimodal large model, and is intended to strengthen resource coordination and task collaboration between clusters or nodes. It is mainly used to request support from other nodes to achieve more efficient resource utilization and task execution.

[0089] The second collaboration request in this application is a collaboration request for other visible clusters or nodes, that is, other clusters or nodes that are identifiable in the current cluster and can provide necessary resources or support. Visibility is generally determined based on information such as network connections, task requirements, and resource status.

[0090] With the power of the multimodal large model, data from different sources can be processed, including sensor information, task descriptions, resource status, etc. The model generates a second collaborative request by analyzing this information to ensure the accuracy and appropriateness of the request.

[0091] Through the second collaborative request, other nodes can quickly identify their own schedulable resources and optimize resource allocation. This enables collaborative execution of tasks. For example, different clusters or nodes can jointly complete a complex detection task. In this process, collaborative actions are reasonably allocated to improve the overall efficiency of task completion.

[0092] Optionally, inputting the input information into a multimodal large model for task understanding includes:

[0093] Inputting the input information into a multimodal large model for unified encoding to obtain an encoding vector;

[0094] Using the multi-layer neural network encoder of the multimodal large model, compressing and adjusting the encoding vector to obtain a compression feature;

[0095] Based on the compressed feature model analysis, task understanding of multimodal large models can be achieved.

[0096] Furthermore, the method further comprises:

[0097] receiving a first input from a user, wherein the first input is a natural language intervention task instruction;

[0098] The compressed features are combined with natural language intervention tasks to achieve task understanding of the multimodal large model.

[0099] In this embodiment, first, various input information is input into the model through a multimodal large model for unified encoding. This encoding stage can convert data of different modalities into a unified representation, usually a high-dimensional encoding vector. This process ensures the consistency of information, so that multimodal features can be compared and analyzed on the same dimension.

[0100] The generated encoding vector is processed using a multi-layer neural network encoder of a multimodal large model. Through the multi-layer structure of the network, higher-level features can be extracted, which can more effectively express the core semantics of the input information.

[0101] The encoder compresses and adjusts the encoded vector through nonlinear transformation and dimensionality reduction operations to generate compressed features. These compressed features are usually low-dimensional and more representative features, retaining the key information of the original input while eliminating redundancy.

[0102] Using the generated compressed features, the model can understand the task and identify the task objectives and requirements hidden in the input data. The compressed features can be further input into deep learning models, such as classifiers or regression models, to perform specific task understanding analysis. For example, determine which category the input task belongs to, or predict the execution result of the task.

[0103] Furthermore, by introducing user interaction, the system can receive the user's first input, usually in the form of natural language intervention task instructions. Users can describe their needs in natural language and provide specific task instructions. Combining compressed features with the user's natural language instructions further enhances the model's ability to understand the task. This process is usually implemented through text embedding and feature fusion techniques to ensure that the user's intentions are effectively integrated into the model's task understanding process. Combining user input can provide additional contextual information to help large multimodal models understand and perform tasks more accurately.

[0104] The embodiment of the present application utilizes a multimodal large model, and effectively realizes the understanding of complex tasks through unified encoding of input information, extraction of compressed features, and combination of natural language instructions of users. This not only improves the intelligence level of the model, but also enhances the interaction between users and the system, making the execution of detection tasks more flexible and efficient.

[0105] Reference Figure 2 , the complete process diagram of this application includes:

[0106] S1. Collect and detect resource status inside and outside the cluster in real time;

[0107] S2. Real-time perception of the status of the detection task target;

[0108] S3. Calculate the relationship between detection resources and mission objectives;

[0109] S4. Combine artificial natural language intervention to understand large-scale model planning tasks;

[0110] S5. The big model controls multiple professional models to generate planning solutions in parallel;

[0111] S6. Combine collaborative request intervention to conduct large-scale model control planning scheme deduction and optimization.

[0112] Reference Figure 3 The present application also provides an interactive situation detection resource perception and planning system, including:

[0113] The node information acquisition module 310 is used to obtain node status information of detection nodes inside and outside the cluster node;

[0114] A perception module 320 is used to obtain situation information of the detection task and perceive the target state information of the detection target in the detection task in real time according to the situation information;

[0115] A relationship information acquisition module 330, used to determine various relationship information between the detection node and the detection target according to the node state information and the target state information;

[0116] A model understanding module 340 is used to combine the detection task, node state information, target state information, relationship information with the user input instruction and the first collaborative request as input information, and input the input information into the multimodal large model for task understanding;

[0117] The solution generation module 350 is used to use the multimodal large model to call multiple different professional models according to the task type of the detection task, generate planning solutions in parallel based on the professional models, and select a preferred solution from each of the planning solutions.

[0118] Optionally, a multimodal large model is used to call multiple different professional models in parallel to generate planning schemes according to the task type of the detection task, and the large model is used to control the deduction and optimization of each planning scheme, including:

[0119] In combination with the first collaborative request, the nodes in the other clusters are interacted with, and based on the principles of internal resource coordination and external resource operation and request, a multimodal large model is used to call a dynamic programming model and a reinforcement learning model in parallel to generate planning schemes corresponding to different models;

[0120] Score and rank the planning schemes generated by each professional model from multiple dimensions such as schedulability, economy, maintainability and efficiency, and select the preferred scheme based on the score ranking;

[0121] The first collaborative request is a collaborative request for standardized detection resources from other clusters or nodes.

[0122] Optionally, a collaborative request module is further included, for:

[0123] A second collaboration request is output through the multimodal large model, where the second collaboration request is a collaboration request for other visible clusters or nodes, and the second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

[0124] Optionally, the node status information is collected in real time from detection nodes inside and outside the cluster node according to a fixed time period; the node status information includes time location information, detection means, detection range, performance parameters and cruising range information.

[0125] Optionally, sensing target state information of the detection task in real time according to the situation information includes:

[0126] Real-time perception of the time position information and multi-modal information of the detection target according to a fixed time period;

[0127] According to the node status information and the target status information, various relationship information between the detection node and the detection target is determined, including:

[0128] According to the time position information of the detection node and the time position information of the detection target, the distance and detection capability matching degree between the detection node and the detection target are determined.

[0129] Optionally, inputting the input information into a multimodal large model for task understanding includes:

[0130] Inputting the input information into a multimodal large model for unified encoding to obtain an encoding vector;

[0131] Using the multi-layer neural network encoder of the multimodal large model, compressing and adjusting the encoding vector to obtain a compression feature;

[0132] Based on the compressed feature model analysis, task understanding of multimodal large models can be achieved.

[0133] Optionally, an intervention module is further included, for:

[0134] receiving a first input from a user, wherein the first input is a natural language intervention task instruction;

[0135] The compressed features are combined with natural language intervention tasks to achieve task understanding of the multimodal large model.

[0136] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0137] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0138] Reference Figure 4 Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor (processor) 410, a communication interface (CommunicationsInterface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the method in the above embodiment.

[0139] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0140] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0141] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0142] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0143] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0144] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0145] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0146] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An interactive situation detection resource perception and planning method, characterized in that: include: Collect and organize detection resources inside and outside the cluster in real time; Acquire situation information of the detection task and perceive target state information of the detection target in the detection task in real time according to the situation information; Calculate and determine various relationship information between detection nodes and detection targets based on node status information and target status information; Taking the detection task, node status information, target status information, relationship information, user input instructions, and the first collaborative request as input information, and inputting the input information into the multimodal large model for task understanding; A multimodal large model is used to call multiple different professional models to generate planning schemes in parallel according to the task type of the detection task, and the large model is used to control the deduction and optimization of each planning scheme.

2. The situation detection resource perception and planning method according to claim 1, characterized in that: The multimodal big model is used to call multiple different professional models to generate planning schemes in parallel according to the task type of the detection task, and the big model is used to control the deduction and optimization of each planning scheme, including: Combined with the first collaborative request, interact with other cluster nodes, and based on the principles of internal resource coordination and external resource operation and request, use a multi-modal large model to call widely used multi-type dynamic programming, heuristic search and deep reinforcement learning professional models in parallel, so as to generate a variety of different alternative planning plans; Using the big model in a prompt learning way, the planning schemes generated by each professional model are deduced from multiple dimensions such as schedulability, economy, maintainability and efficiency. The schemes are ranked according to the deduction scores and the optimal scheme is recommended. The first collaborative request is a collaborative request for standardized detection resources from other clusters or nodes.

3. The situation detection resource perception and planning method according to claim 1, characterized in that: Also includes: A second collaboration request is output through the multimodal large model, where the second collaboration request is a collaboration request for other visible clusters or nodes, and the second collaboration request is used by other nodes to achieve resource optimization and task collaboration.

4. The situation detection resource perception and planning method according to claim 1, characterized in that: The node status information is collected in real time from detection nodes inside and outside the cluster nodes according to a fixed time period; the node status information includes time location information, detection means, detection range, performance parameters and cruising range information.

5. The situation detection resource perception and planning method according to claim 1, characterized in that: The target state information of the detection task is perceived in real time according to the situation information, including: Real-time perception of the time position information and multi-modal information of the detection target according to a fixed time period; According to the node status information and the target status information, various relationship information between the detection node and the detection target is determined, including: According to the time position information of the detection node and the time position information of the detection target, the distance and detection capability matching degree between the detection node and the detection target are determined.

6. The situation detection resource perception and planning method according to claim 1, characterized in that: Inputting the input information into the multimodal large model for task understanding includes: Inputting the input information into a multimodal large model for unified encoding to obtain an encoding vector; Using the multi-layer neural network encoder of the multimodal large model, compressing and adjusting the encoding vector to obtain a compression feature; Based on the compressed feature model analysis, task understanding of multimodal large models can be achieved.

7. The situation detection resource perception and planning method according to claim 6, characterized in that: The method further comprises: receiving a first input from a user, wherein the first input is a natural language intervention task instruction; The compressed features are combined with natural language intervention tasks to achieve task understanding of the multimodal large model.

8. An interactive situation detection resource perception and planning system, characterized in that: include: The node information acquisition module is used to obtain the node status information of the detection nodes inside and outside the cluster node; A perception module is used to obtain situation information of the detection task and perceive the target state information of the detection target in the detection task in real time according to the situation information; A relationship information acquisition module is used to determine various relationship information between the detection node and the detection target according to the node status information and the target status information; A model understanding module, used to combine the detection task, node status information, target status information, relationship information with the user input instruction and the first collaborative request as input information, and input the input information into the multimodal large model for task understanding; The scheme generation module is used to use the multimodal large model to call multiple different professional models according to the task type of the detection task, generate planning schemes in parallel based on the professional models, and select the preferred scheme from each of the planning schemes.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images