Unmanned system control method and device based on multistage world model, and medium

Abstract

The invention discloses an unmanned system control method and device based on a multistage world model and a medium, and can be widely applied to the technical field of unmanned control. According to the invention, after the to-be-trained large-world model is trained through the multi-modal data including the image data, the video data or the business text data to obtain the cloud-side large-world model, the side middle-world model and the end-side small-world model are obtained through distillation according to the cloud-side large-world model; inputting the natural language instruction of the target object and the intelligent agent state of the end-side small-world model into the side-side middle-world model to predict and obtain a working instruction, inputting the working instruction into the end-side small-world model to predict and obtain a working state adjusting instruction of the unmanned system, and then adjusting the unmanned system according to the working state adjusting instruction. Therefore, the dependence on human experts is reduced, the generalization ability on diversified and changing task scenes is improved, and the patrol task of a specific scene can be timely and effectively completed.

Description

Technical Field

[0001] The present application relates to the field of unmanned control technology, and in particular to an unmanned system control method, device and medium based on a multi-level world model. Background Art

[0002] Among related technologies, unmanned systems (such as unmanned vehicles and drones) have been widely used. However, the scheduling, coordination and control functions of current unmanned systems mainly rely on direct control and manual program arrangement by technicians, resulting in limited generalization capabilities of unmanned systems in the face of diverse and changing mission scenarios, making it impossible to complete inspection tasks in specific scenarios in a timely and effective manner.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an unmanned system control method, device and medium based on a multi-level world model, which can complete the inspection tasks of specific scenarios in a timely and effective manner.

[0005] To achieve the above object, an embodiment of the present application provides an unmanned system control method based on a multi-level world model, the method comprising the following steps:

[0006] Acquiring multimodal data, wherein the multimodal data includes image data, video data or business text data;

[0007] The large world model to be trained is trained according to the multimodal data to obtain a cloud-side large world model;

[0008] According to the cloud-side large-world model, a side-medium-world model and a terminal-side small-world model are obtained by distillation;

[0009] Acquire a natural language instruction of a target object and an agent state of the terminal-side small-world model;

[0010] Inputting the natural language instruction and the agent state into the side world model to predict the work instruction;

[0011] Inputting the work instruction into the terminal-side small-world model to predict and obtain a work state adjustment instruction of the unmanned system;

[0012] The unmanned system is adjusted according to the working status adjustment instruction.

[0013] In some embodiments, training the large world model to be trained according to the multimodal data includes:

[0014] When the multimodal data is the business text data, aggregating the business text data to obtain a first data set, wherein the first data set includes data pairs of instructions and texts;

[0015] According to the first data set, model parameters of the large-world model to be trained are adjusted by a first preference optimization algorithm.

[0016] In some embodiments, training the large world model to be trained according to the multimodal data includes:

[0017] When the multimodal data includes image data or video data, the image data or the video data is aggregated to obtain a second data set;

[0018] According to the second data set, the model parameters of the large world model to be trained are adjusted by a second preference optimization algorithm and a future frame prediction algorithm.

[0019] In some embodiments, training the large world model to be trained according to the multimodal data includes:

[0020] When the multimodal data includes image and text pair data or video and text pair data, data construction is performed according to the image and text pair data or the video and text pair data to obtain a third data set, wherein the third data set includes image data and description information of the image data, and video data and description information of the video data;

[0021] The model parameters of the large-world model to be trained are adjusted according to the third data set.

[0022] In some embodiments, the model parameter amount of the cloud-side big world model is greater than the model parameter amount of the edge-side medium-world model, and the model parameter amount of the edge-side medium-world model is greater than the model parameter amount of the terminal-side small world model; the cloud-side big world model has language knowledge reasoning capabilities and multimodal knowledge aggregation and updating capabilities, and the edge-side medium-world model and the terminal-side small world model both have language knowledge reasoning capabilities.

[0023] In some embodiments, the step of inputting the natural language instruction and the agent state into the side world model to predict the work instruction includes:

[0024] Recognize the intention information of the target object according to the natural language instruction;

[0025] Arrange the work tasks of the agent cluster according to the intention information and the agent status;

[0026] Generate the work instruction corresponding to the work task.

[0027] In some embodiments, inputting the working instruction into the terminal-side small-world model to predict and obtain the working state adjustment instruction of the unmanned system includes:

[0028] Acquiring working environment information corresponding to the unmanned system;

[0029] According to the work instruction and the work environment information, the work state adjustment instruction of the unmanned system is obtained by predicting the terminal-side small-world model.

[0030] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides an unmanned system control device based on a multi-level world model, the device comprising:

[0031] A first module is used to obtain multimodal data, where the multimodal data includes image data, video data or business text data;

[0032] The second module is used to train the large world model to be trained according to the multimodal data to obtain a cloud-side large world model;

[0033] The third module is used to obtain an edge-side medium-world model and a terminal-side small-world model according to the cloud-side large-world model distillation;

[0034] A fourth module is used to obtain a natural language instruction of a target object and an agent state of the terminal-side small-world model;

[0035] A fifth module is used to input the natural language instruction and the agent state into the side world model to predict the work instruction;

[0036] A sixth module is used to input the work instruction into the terminal-side small-world model to predict and obtain the work state adjustment instruction of the unmanned system;

[0037] The seventh module is used to adjust the unmanned system according to the working status adjustment instruction.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides a computer device, the device comprising:

[0039] at least one processor;

[0040] at least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0042] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0043] The embodiments of the present application include at least the following beneficial effects: The present application provides an unmanned system control method, device and medium based on a multi-level world model. The scheme trains a large world model to be trained with multimodal data including image data, video data or business text data to obtain a cloud-side large world model, and then obtains an edge-side medium-world model and a terminal-side small-world model according to the distillation of the cloud-side large world model. The natural language instructions of the target object and the agent state of the terminal-side small-world model are input into the edge-side medium-world model to predict the work instructions, and the work instructions are input into the terminal-side small-world model to predict the work state adjustment instructions of the unmanned system, and then the unmanned system is adjusted according to the work state adjustment instructions, thereby reducing dependence on human experts and improving the generalization ability of diverse and changing task scenarios, so that the inspection tasks of specific scenarios can be completed in a timely and effective manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of an unmanned system control method based on a multi-level world model provided in an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of an application scenario of an unmanned system control method based on a multi-level world model provided in an embodiment of the present application;

[0046] Figure 3 It is a structural schematic diagram of an unmanned system control device based on a multi-level world model provided in an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0051] World model: A virtual environment built within an AI system that simulates the dynamics, physical laws, and interactions between objects in the real world. This model enables AI to learn and make decisions in a controlled environment without directly interacting with the real world. This is particularly important for high-cost or high-risk experiments that are difficult to perform in the real world. The key function of the world model is to perform counterfactual reasoning, that is, to explore the question "What if this happens?" in a simulated environment. This model is mainly used in reinforcement learning, robotics, and simulation. There is currently no unified definition of the world model. Its research purpose is to design a neural network module that can update the state, to memorize and model the environment, to input the current observation (image, state, etc.) and the action to be taken, and to predict the next possible observation (image, state) and action based on the model's memory and understanding of the world, and to use the difference between the actual observation and the predicted observation at the next moment after the action is taken.

[0052] In related technologies, unmanned systems have been widely used, such as in traffic maintenance, unmanned delivery and other scenarios. However, the scheduling, coordination and control of unmanned systems currently rely mainly on direct control by technicians and manual program arrangement control. This control method has the following problems: First, the application threshold is high and it is difficult to promote on a large scale; second, the application procedures are relatively cumbersome, and tasks cannot be quickly issued through simple text instructions when facing different scenarios; third, unmanned systems such as drones and unmanned vehicles have weak perception of environmental information, resulting in insufficient robustness; fourth, the coordination capabilities within and between unmanned clusters are limited, making it difficult to form efficient coordination and creating potential safety hazards.

[0053] In view of this, an unmanned system control method, device and medium based on a multi-level world model are provided in an embodiment of the present application. After the embodiment of the present application trains a large world model to be trained with multimodal data including image data, video data or business text data to obtain a cloud-side large world model, an edge-side medium-world model and a terminal-side small-world model are obtained according to distillation of the cloud-side large world model. The natural language instructions of the target object and the agent state of the terminal-side small-world model are input into the edge-side medium-world model to predict the work instructions, and the work instructions are input into the terminal-side small-world model to predict the work state adjustment instructions of the unmanned system. Then, the unmanned system is adjusted according to the work state adjustment instructions, thereby reducing dependence on human experts and improving the generalization ability of diverse and changing task scenarios, so as to complete the inspection tasks of specific scenarios in a timely and effective manner.

[0054] The unmanned system control method based on the multi-level world model provided in the embodiment of the present application relates to the field of unmanned control technology. The unmanned system control method based on the multi-level world model provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements the unmanned system control method based on the multi-level world model, etc., but is not limited to the above forms.

[0055] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0056] Figure 1 is an optional flow chart of the unmanned system control method based on the multi-level world model provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S110 to S170:

[0057] Step S110: acquiring multimodal data, wherein the multimodal data includes image data, video data or business text data;

[0058] Step S120: training the large world model to be trained according to the multimodal data to obtain a cloud-side large world model;

[0059] Step S130: Obtain an edge-side medium-world model and a terminal-side small-world model according to the cloud-side large-world model through distillation;

[0060] Step S140: Acquire the natural language instruction of the target object and the agent state of the small-world model on the terminal side;

[0061] Step S150, inputting the natural language instruction and the agent state into the side world model to predict the work instruction;

[0062] Step S160: input the work instruction into the small-world model on the terminal side, and predict the work state adjustment instruction of the unmanned system;

[0063] Step S170: adjust the unmanned system according to the working status adjustment instruction.

[0064] It is understandable that if Figure 2 As shown, the embodiments of the present application involve three levels of world models, namely the cloud-side big world model, the edge-side medium world model, and the terminal-side small world model. Among them, the cloud-side big world model is the intelligence center of the entire drone system. The model is built based on the big world model to be trained, and is mainly responsible for centralized training of model knowledge and capability upgrades. The cloud-side big world model is a general large model with multimodal data processing capabilities. By aggregating multimodal data such as images, videos, texts, etc. collected on the edge and training them centrally, the cloud-side big world knowledge is continuously accumulated and updated, and the counterfactual reasoning ability of the big world model is continuously improved.

[0065] In an embodiment of the present application, when the multimodal data is business text data, the business text data is aggregated, for example, rules and regulations, operating specifications, instruction sets, etc. are aggregated to form "instruction-answer" data pairs and saved in the first data set, and then the model parameters of the large world model to be trained are adjusted in combination with the first preference optimization algorithm to improve the text understanding and knowledge answering capabilities of the cloud-side large world model. When the multimodal data includes image data or video data, the image data or video data is aggregated to obtain a second data set, such as shooting surveillance images, historical event data, etc., and the model parameters of the large world model to be trained are adjusted in combination with the image / video content understanding direct preference optimization algorithm (second preference optimization algorithm) and the future frame prediction algorithm to improve the image or video content understanding capabilities of the cloud-side large world model. When the multimodal data includes image and text pair data or video and text pair data, data construction is performed according to the image and text pair data or the video and text pair data, such as constructing historical work order data, traffic event data, etc., and a third data set of image data and image data description information, video data and video data description information, and then the model parameters of the large world model to be trained are adjusted in combination with the image-text alignment algorithm to improve the image-text alignment capability of the cloud-side large world model. It can be seen from this that this embodiment continuously fine-tunes the model parameters of the cloud-side large world model through multimodal knowledge aggregation, so that it can respond to a variety of specific tasks, such as knowledge question and answer (effectively answering the instructions proposed by the user according to the aggregated knowledge), image or video content understanding (understanding and analyzing the image or video content, and further adjusting and understanding the image or video content according to the target object instruction content), video content prediction (predicting subsequent content based on the input image or continuous video), and other matters.

[0066] In the embodiment of the present application, the world model on the edge is responsible for the understanding, allocation and scheduling of tasks, and can be combined with the business middle station. The world model on the edge is distilled from the big world model on the cloud side, and also belongs to a large language model. It is constructed by applying the "teacher-student" model distillation method, and the natural language knowledge reasoning ability of the big world model on the cloud side is fixed with a model structure with relatively low parameter quantity, and its multimodal knowledge aggregation and update ability is abandoned, which can greatly reduce the computing resource threshold of the world model on the edge side, so that it can run on the edge side device. It can be seen that the model parameter quantity of the big world model on the cloud side is greater than the model parameter quantity of the world model on the edge side, and the big world model on the cloud side has language knowledge reasoning ability and multimodal knowledge aggregation and update ability, and the world model on the edge side has language knowledge reasoning ability. When responding to a task, the world model on the edge side can receive the natural language instructions of the target object, and can identify the intention information of the target object through the natural language instructions of the target object, and according to the intention information and the agent state of the small world model, reasonably arrange the work tasks of the agent cluster, generate work instructions for scheduling agents and save them in a dedicated command set.

[0067] In the embodiment of the present application, the end-side small world model is an intelligent entity (such as an unmanned vehicle or drone) equipped with a language model with certain analysis and decision-making capabilities. The end-side small world model receives the work instructions arranged by the edge-side medium world model, analyzes the work state adjustment instructions corresponding to the task execution environment through the matching work environment information, and adjusts the execution process of the unmanned system. Compared with the edge-side medium world model, the end-side small world model has a further reduced number of parameters. Through the "teacher-student" model distillation method, the environmental perception and understanding ability of the cloud-side large world model is fixed, and the multimodal information perception and analysis ability of the end-side small world model is trained, further reducing the computing resource threshold of the edge-side small world model so that it can run on the end-side device. It can be seen that the model parameter amount of the edge-side medium world model is greater than the model parameter amount of the end-side small world model, and the end-side small world model has language knowledge reasoning ability. In the task response process, the end-side small world model can perceive the work environment information through computer vision, various sensors and other devices, and adjust the work state adjustment instructions according to the work environment information, so that the execution process adapts to the actual situation.

[0068] It can be understood that the interactive responses of world models at different levels of this embodiment are as follows: the cloud-side big world model achieves capability updates and improves counterfactual reasoning capabilities by aggregating edge-side knowledge data; the edge-side medium world model is distilled and trained from the cloud-side big world model, has the ability to understand natural language instructions, orchestrates work tasks and issues instructions to the end-level small world model, and at the same time aggregates knowledge to the cloud-side big world model; the end-side small world model is also distilled and trained from the cloud-side big world model, has environmental perception and analysis capabilities, and can adaptively execute tasks issued by the edge model according to the actual environment.

[0069] It is understandable that the method of the embodiment of the present application can realize the function of quickly dispatching the unmanned system to complete the work based on natural language commands. For example, in the highway scene, the highway inspection staff issues a text instruction to the side world model "Please complete the inspection of the slope of XX highway on XX / XX / XX", and the side world model understands the staff's intention information, and then completes the special work "slope inspection" of the specific object "XX section of XX highway" at a specific time of XX / XX / XX, and compiles a work plan, and outputs a special instruction through the side world model, and issues it to the intelligent body (drone) equipped with the small world model through the unmanned system middle platform (which can be integrated with the existing one, or the middle world model can be used as the middle platform). After receiving the flight control instruction, the intelligent body begins to perform its respective tasks, and relies on the corresponding environmental perception equipment (such as cameras, radars, etc.) to perceive the environmental status (weather, obstacles, etc.) of the operating area during the task execution, and performs autonomous obstacle avoidance through the end-side small world model to complete the corresponding work tasks.

[0070] For example, in an unmanned express delivery system, the express delivery staff issues a text instruction to the side middle-world model, "Please complete the express delivery work in XX community on XX / XX / XX." The side middle-world model understands the staff's intention information, and then completes the special task of "express delivery" for the specific object "XX community" on a specific time of XX / XX / XX, and compiles a work plan. It also outputs special instructions through the side middle-world model, and issues them to the intelligent agent (unmanned vehicle) equipped with the small-world model through the unmanned system middle platform. After receiving the driving instructions, the intelligent agent begins to perform its respective tasks, and relies on corresponding environmental perception equipment (such as cameras, radars, etc.) to perceive the environmental status of the operating area (weather, obstacles, etc.) during the task execution, and performs autonomous obstacle avoidance through the end-side small-world model to complete the corresponding work tasks.

[0071] In summary, the method of the embodiment of the present application can effectively improve the application generalization of the multi-agent cluster in different regions and scenarios through the interconnected and collaborative method of the cloud-edge multi-agent cluster. Moreover, during the application of the method provided by the present application, the multi-agent cluster can autonomously execute various tasks including highway road slope bridge and tunnel inspection, highway rescue material distribution, daily formation tasks, etc. through only simple text instructions, providing a more efficient and intelligent solution for the intelligent transformation of industries such as transportation and logistics.

[0072] Reference Figure 3 , an embodiment of the present application provides an unmanned system control device based on a multi-level world model, the device comprising:

[0073] A first module 310 is used to obtain multimodal data, wherein the multimodal data includes image data, video data or business text data;

[0074] The second module 320 is used to train the big world model to be trained according to the multimodal data to obtain a cloud-side big world model;

[0075] The third module 330 is configured to obtain an edge-side medium-world model and a terminal-side small-world model according to the cloud-side large-world model distillation;

[0076] The fourth module 340 is used to obtain the natural language instruction of the target object and the agent state of the terminal-side small-world model;

[0077] The fifth module 350 is used to input the natural language instruction and the agent state into the side world model to predict the work instruction;

[0078] The sixth module 360 ​​is used to input the work instruction into the end-side small-world model to predict the work state adjustment instruction of the unmanned system;

[0079] The seventh module 370 is used to adjust the unmanned system according to the working status adjustment instruction.

[0080] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0081] The embodiment of the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned unmanned system control method based on a multi-level world model when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0082] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0083] See also Figure 4 , Figure 4 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0084] The processor 410 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0085] The memory 420 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 420 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 420, and the processor 410 calls and executes the unmanned system control method based on the multi-level world model in the embodiment of this application;

[0086] Input / output interface 430, used to implement information input and output;

[0087] Communication interface 440, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0088] bus 450 , which transmits information between the various components of the device (e.g., processor 410 , memory 420 , input / output interface 430 , and communication interface 440 );

[0089] The processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 are connected to each other in communication within the device via the bus 450 .

[0090] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned unmanned system control method based on the multi-level world model.

[0091] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0092] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0093] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0094] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0095] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0096] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0097] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0098] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0103] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for controlling an unmanned system based on a multi-level world model, characterized in that: The method comprises the following steps: Acquiring multimodal data, wherein the multimodal data includes image data, video data or business text data; The large world model to be trained is trained according to the multimodal data to obtain a cloud-side large world model; According to the cloud-side large-world model, a side-medium-world model and a terminal-side small-world model are obtained by distillation; Acquire a natural language instruction of a target object and an agent state of the terminal-side small-world model; Inputting the natural language instruction and the agent state into the side world model to predict the work instruction; Inputting the work instruction into the terminal-side small-world model to predict and obtain a work state adjustment instruction of the unmanned system; The unmanned system is adjusted according to the working status adjustment instruction.

2. The method according to claim 1, characterized in that The step of training the large world model to be trained according to the multimodal data includes: When the multimodal data is the business text data, aggregating the business text data to obtain a first data set, wherein the first data set includes data pairs of instructions and texts; According to the first data set, model parameters of the large-world model to be trained are adjusted by a first preference optimization algorithm.

3. The method according to claim 2, characterized in that The step of training the large world model to be trained according to the multimodal data includes: When the multimodal data includes image data or video data, the image data or the video data is aggregated to obtain a second data set; According to the second data set, the model parameters of the large world model to be trained are adjusted by a second preference optimization algorithm and a future frame prediction algorithm.

4. The method according to claim 3, characterized in that The step of training the large world model to be trained according to the multimodal data includes: When the multimodal data includes image and text pair data or video and text pair data, data construction is performed according to the image and text pair data or the video and text pair data to obtain a third data set, wherein the third data set includes image data and description information of the image data, and video data and description information of the video data; The model parameters of the large-world model to be trained are adjusted according to the third data set.

5. The method according to claim 1, characterized in that The model parameter quantity of the cloud-side big world model is greater than the model parameter quantity of the edge-side medium-world model, and the model parameter quantity of the edge-side medium-world model is greater than the model parameter quantity of the terminal-side small world model; the cloud-side big world model has language knowledge reasoning capability and multimodal knowledge aggregation and updating capability, and the edge-side medium-world model and the terminal-side small world model both have language knowledge reasoning capability.

6. The method according to claim 1, characterized in that The step of inputting the natural language instruction and the agent state into the side world model to predict the work instruction includes: Recognize the intention information of the target object according to the natural language instruction; Arrange the work tasks of the agent cluster according to the intention information and the agent status; Generate the work instruction corresponding to the work task.

7. The method according to claim 1, characterized in that The step of inputting the working instruction into the terminal-side small-world model to predict and obtain the working state adjustment instruction of the unmanned system includes: Acquiring working environment information corresponding to the unmanned system; According to the work instruction and the work environment information, the work state adjustment instruction of the unmanned system is obtained by predicting the terminal-side small-world model.

8. An unmanned system control device based on a multi-level world model, characterized in that: The device comprises: A first module is used to obtain multimodal data, where the multimodal data includes image data, video data or business text data; The second module is used to train the large world model to be trained according to the multimodal data to obtain a cloud-side large world model; The third module is used to obtain an edge-side medium-world model and a terminal-side small-world model according to the cloud-side large-world model distillation; A fourth module is used to obtain a natural language instruction of a target object and an agent state of the terminal-side small-world model; A fifth module is used to input the natural language instruction and the agent state into the side world model to predict the work instruction; A sixth module is used to input the work instruction into the terminal-side small-world model to predict and obtain the work state adjustment instruction of the unmanned system; The seventh module is used to adjust the unmanned system according to the working status adjustment instruction.

9. A computer device, characterized in that: The device comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images