Fire control method, system and device, computer equipment and medium
Through the combination of multimodal encoder and action decoder, the fire fighting machine can make independent decisions and perform fire fighting tasks, solving the problem of fire fighting robots lacking independent fire fighting capabilities and improving fire fighting efficiency.
Patent Information
- Application Number
- CN202411331822.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2024-12-20
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Firefighting robots lack independent fire fighting capabilities, resulting in low fire fighting efficiency.
Multimodal encoder, target encoder and action decoder are used to collect fire-fighting images, convert them into feature vectors, and generate action sequences with multimodal information to realize the independent decision-making and execution of the fire-fighting machine.
It improves the learning, reasoning and decision-making capabilities of fire fighting machines, and can more accurately understand and adapt to the fire fighting environment, thereby improving fire fighting efficiency.
Smart Images
Figure CN119169601A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of fire protection, and in particular to a fire protection control method, system, device, computer equipment and medium. Background Art
[0002] At present, with the rapid development of social economy and the particularity of construction and enterprise production processes, the potential accidents such as leakage of dangerous chemicals and radioactive substances, fire, explosion, collapse, etc. are becoming increasingly prominent. When a fire occurs, firefighters face a severe environment - high temperature, darkness, toxic gas and dense fog. If they blindly enter the fire without protective measures, they will not only be unable to perform their tasks, but may even cause additional casualties.
[0003] In the prior art, firefighting robots are usually used to replace manual labor and are competent for rescue in dangerous environments and difficult-to-reach areas, such as disaster sites in tunnels, underground buildings or other narrow spaces. With the rapid advancement of firefighting robot technology, its application in the field of firefighting and rescue has become more and more extensive. In firefighting and rescue, robots can play a role in firefighting, reduce the risk of casualties of rescue personnel, effectively reduce the temperature of the fire scene, and reduce smoke, thereby reducing casualties and property losses, which is crucial to improving rescue safety. At present, mobile firefighting robots mainly use remote control, and the robots do not have the ability to judge the situation on the scene and use firefighting tools at the fire scene. Therefore, the prior art has the problem of low firefighting efficiency due to the lack of autonomous firefighting capabilities of firefighting robots. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a fire control method, system, device, computer equipment and medium, which aims to solve the problem of low firefighting efficiency caused by the lack of autonomous firefighting capability of firefighting robots.
[0005] In a first aspect, an embodiment of the present application provides a fire control method, which is applied to a first fire machine, wherein the first fire machine includes a multimodal encoder, a target encoder, and an action decoder, and the method includes:
[0006] Upon receiving first multimodal information for instructing to perform a first firefighting action related to a first object, acquiring a firefighting image of the first object;
[0007] The firefighting image is converted into a target feature vector through a target encoder;
[0008] Inputting the first multimodal information into a multimodal encoder to obtain a multimodal prompt feature vector;
[0009] Inputting the multimodal prompt feature vector and the target feature vector into an action decoder to obtain an action sequence of a first firefighting action;
[0010] Execute the action sequence.
[0011] In some embodiments, after inputting the multimodal cue feature vector and the target feature vector, the action decoder performs the following steps:
[0012] extracting key sequences and value sequences from the multimodal prompt feature vector;
[0013] According to the target feature vector, the target historical action sequence is retrieved in the trajectory history sequence;
[0014] Based on the target historical action sequence, the key sequence and the value sequence, an action sequence of the first firefighting action is generated.
[0015] In some implementations, the action sequence is generated by the following formula:
[0016]
[0017] Among them, H′ is the action sequence, softmax is the normalized exponential function, Q H is the target historical action sequence, is the key sequence, V p is a sequence of values, and d is the embedding dimension.
[0018] In some embodiments, the first multimodal information is generated by the second fire fighting machine by inputting the second multimodal information into the multimodal big model;
[0019] The second multimodal information is used to indicate the firefighting tasks at the fire scene.
[0020] In some embodiments, the multimodal large model includes a multimodal encoder, an input projector, a large language model backbone network, an output projector, and a multimodal generator; the second fire fighting machine generates the first multimodal information by inputting the second multimodal information into the multimodal large model, including:
[0021] The multimodal encoder is used to obtain a multimodal feature vector according to the second multimodal information input;
[0022] The input projector is used to align the multimodal feature vector to the text modality space to obtain the input projection vector of the multimodal feature vector;
[0023] The large language model backbone network is used to obtain the instruction text and the signal feature vector according to the input projection vector and the feature vector of the instruction text;
[0024] The output projector is used to map the signal feature vector to the feature space of the multimodal generator to obtain an output projection vector of the signal feature vector;
[0025] The multimodal generator is used to obtain first multimodal information according to the output projection vector.
[0026] In a second aspect, an embodiment of the present application provides a fire control system, the system comprising:
[0027] The second fire-fighting machine is used to input the second multimodal information into the multimodal large model to generate the first multimodal information, and send the first multimodal information to the first fire-fighting machine; wherein the second multimodal information is used to indicate the fire-fighting task at the fire scene, and the first multimodal information indicates the first fire-fighting action related to the first object of the fire-fighting task.
[0028] The first fire-fighting machine communicates with the second fire-fighting machine, and includes a multimodal encoder, a target encoder and an action decoder, which is used to collect a fire-fighting image of the first object when receiving first multimodal information for indicating the execution of a first fire-fighting action related to the first object; convert the fire-fighting image into a target feature vector through the target encoder; input the first multimodal information into the multimodal encoder to obtain a multimodal prompt feature vector; input the multimodal prompt feature vector and the target feature vector into the action decoder to obtain an action sequence of the first fire-fighting action; and execute the action sequence.
[0029] In some embodiments, the multimodal large model includes a multimodal encoder, an input projector, a large language model backbone network, an output projector, and a multimodal generator; the second fire fighting machine is also used to:
[0030] Inputting the second multimodal information into the multimodal encoder to obtain a multimodal feature vector;
[0031] Aligning the multimodal feature vector to the text modality space through an input projector to obtain an input projection vector of the multimodal feature vector;
[0032] Input the input projection vector and the feature vector of the instruction text into the large language model backbone network to obtain the instruction text and signal feature vectors;
[0033] Mapping the signal feature vector to the feature space of the multimodal generator through the output projector to obtain an output projection vector of the signal feature vector;
[0034] The output projection vector is input into the multimodal generator to obtain the first multimodal information.
[0035] In a third aspect, an embodiment of the present application provides a fire control device, the device comprising:
[0036] A collection module, configured to collect a firefighting image of the first object upon receiving first multimodal information for instructing to perform a first firefighting action related to the first object;
[0037] A conversion module, used for converting the firefighting image into a target feature vector through a target encoder;
[0038] A multimodal prompt feature vector generation module, used for inputting the first multimodal information into a multimodal encoder to obtain a multimodal prompt feature vector;
[0039] An action sequence generation module, used for inputting the multimodal prompt feature vector and the target feature vector into an action decoder to obtain an action sequence of a first firefighting action;
[0040] The execution module is used to execute the action sequence.
[0041] In a fourth aspect, an embodiment of the present application provides a computer device, the device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0042] Memory, used to store computer programs;
[0043] The processor is used to implement the steps of the fire control method provided in the first aspect of the embodiment of the present application when executing the program stored in the memory.
[0044] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the fire control method provided in the first aspect of the embodiment of the present application are implemented.
[0045] In the embodiment of the present application, when the first fire-fighting machine receives the first multimodal information for instructing to execute the first fire-fighting action related to the first object, the fire-fighting image of the first object is first collected, and then the fire-fighting image is converted into a target feature vector through the target encoder, and the first multimodal information is input into the multimodal encoder to obtain the multimodal prompt feature vector, and then the multimodal prompt feature vector and the target feature vector are input into the action decoder to obtain the action sequence of the first fire-fighting action, and finally the action sequence is executed. In this way, the fire-fighting machine can have certain learning, reasoning and decision-making capabilities, and can more accurately understand and adapt to the current environment, thereby improving the efficiency of firefighting. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0047] Figure 1 It is a flowchart of a fire control method provided in an embodiment of the present application;
[0048] Figure 2 This is a schematic diagram of a flow chart of action sequence generation provided by an embodiment of the present application;
[0049] Figure 3 It is a schematic diagram of the architecture of a multimodal large model provided in an embodiment of the present application;
[0050] Figure 4 It is a schematic diagram of a fire control method provided in a specific embodiment of the present application;
[0051] Figure 5 It is a structural schematic diagram of a fire control system provided in an embodiment of the present application;
[0052] Figure 6 It is a structural schematic diagram of a fire control device provided in an embodiment of the present application;
[0053] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0055] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0056] The cloud computing platform management method, system, device, computer equipment and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0057] A first aspect of an embodiment of the present application provides a fire control method, which is applied to a first fire-fighting machine. The first fire-fighting machine is a machine equipment that can perform fire-fighting tasks at a fire scene. It has high reliability and can work stably in various harsh environments, such as high temperature, humidity, smoke and other environments.
[0058] In the embodiment of the present application, the first firefighting machine may include a robot dog and a robot. Among them, the robot dog is a firefighting machine with bionic quadruped walking ability, which can easily pass through various complex terrains. It is also equipped with panoramic cameras, remote communication modules and other equipment, which can obtain images of the fire scene and replace fire investigators to perform detection and monitoring tasks. The robot is an autonomous device that can respond to firefighting scenes such as fire, rescue and emergency tasks, and can flexibly use on-site tools such as fire extinguishers, fire hydrants, etc. to perform tasks such as fire extinguishing and rescue.
[0059] See also Figure 1 , is a flow chart of a fire control method provided in an embodiment of the present application. Figure 1 As shown, the fire control method may include the following steps:
[0060] Step 110: upon receiving first multimodal information for instructing to perform a first firefighting action associated with a first object, collecting a firefighting image of the first object.
[0061] In an embodiment of the present application, the first multimodal information is information for instructing the execution of a first firefighting action related to a first object, that is, the specific content of a firefighting task. When the first firefighting machine receives the first multimodal message, a firefighting image of the first object is collected. For example, the first multimodal information is the use of a nearby fire extinguisher to deal with a fire on a waterproof board trolley. Then the firefighting image collected by the first firefighting machine is an image of the fire on the waterproof board trolley. It should be noted that the number of first firefighting machines and first multimodal information can be multiple.
[0062] Step 120: Convert the firefighting image into a target feature vector through a target encoder.
[0063] Since the feature vector is the basis for the model to process images, the collected fire images need to be converted into feature vectors. For the target encoder, each collected fire image is an object, and each object is represented by a bounding box and a cropped image. The target encoder first divides the fire image into a series of tiles, and then converts each tile into a target feature vector to form an input sequence.
[0064] Step 130: Input the first multimodal information into a multimodal encoder to obtain a multimodal prompt feature vector.
[0065] A multimodal encoder is a neural network model used to process multiple different types of data, namely multimodal data (such as text, images, audio, etc.), with the aim of converting data of different modalities into a unified and comparable feature representation.
[0066] In an embodiment of the present application, after the first multimodal information is input into the multimodal encoder, for each modality of data, the multimodal encoder interleaves the data between different modalities while maintaining the original spatial order of the data of various modalities, thereby outputting a multimodal prompt feature vector. Among them, the original spatial order of the data of various modalities refers to the word order in the text, the pixel or feature position in the image, etc. In this way, the comprehensiveness and accuracy of information processing can be improved.
[0067] Step 140: Input the multimodal prompt feature vector and the target feature vector into the action decoder to obtain the action sequence of the first firefighting action.
[0068] At the fire scene, the essence of the first fire machine automatically performing rescue operations based on multimodal information is to transform the robot's execution of tasks into a sequence model construction process. The method of generating action sequences by combining the action decoder with the multimodal prompt feature vector and the target feature vector can strengthen the connection between the fire status, the fire machine action and the multimodal prompt, thereby improving the efficiency of fire task execution. The language + image multimodal prompt is more intuitive and expressive than pure text instructions. For the interaction of new objects, the first fire machine can also quickly learn basic concepts and supplement visual constraints through simple image examples to avoid accidentally touching dangerous objects.
[0069] Step 150: Execute the action sequence.
[0070] The actions or instructions in the action sequence are arranged in a certain order, and the execution of each action or instruction depends on the completion of the previous action or instruction. The first fire-fighting machine executes the actions in the action sequence one by one in the order of arrangement to complete the fire-fighting task.
[0071] Through the above steps, when the first fire-fighting machine receives the first multimodal information for instructing to execute the first fire-fighting action related to the first object, the fire-fighting image of the first object is first collected, and then the fire-fighting image is converted into a target feature vector through the target encoder, and the first multimodal information is input into the multimodal encoder to obtain a multimodal prompt feature vector, and then the multimodal prompt feature vector and the target feature vector are input into the action decoder to obtain the action sequence of the first fire-fighting action, and finally the action sequence is executed. In this way, the fire-fighting machine can have certain learning, reasoning and decision-making capabilities, and can more accurately understand and adapt to the current environment, thereby improving the level of cooperation and improving fire-fighting efficiency.
[0072] See also Figure 2 , is a flow chart of an action sequence generation process provided by an embodiment of the present application. Figure 2 As shown, in some embodiments, after inputting the multimodal prompt feature vector and the target feature vector, the action decoder may perform the following steps:
[0073] extracting key sequences and value sequences from the multimodal prompt feature vector;
[0074] According to the target feature vector, the target historical action sequence is retrieved in the trajectory history sequence;
[0075] Based on the target historical action sequence, the key sequence and the value sequence, an action sequence of the first firefighting action is generated.
[0076] The action decoder consists of multiple alternating cross-attention layers and self-attention layers. The method of generating action sequences by processing the multimodal prompt feature vector and the target feature vector through the action decoder can strengthen the connection between the fire status, the fire machine action and the multimodal prompt, thereby improving the efficiency of firefighting task execution.
[0077] The first fire fighting machine is obtained by training a multimodal agent using multimodal data and historical actions. Among them, the multimodal data used for training contains data sets of multiple modalities such as text, images, videos, audio, etc., covering various scenarios and tasks that the agent needs to understand and process. Multimodal agents (MAAs) are a class of systems that generate effective actions in a specific environment based on the understanding of multimodal sensory input. During the training process, the optimization goal is to minimize the negative log-likelihood loss of the predicted action. For a trajectory containing T steps, the function to be optimized is as follows:
[0078]
[0079] Where θ is the model weight of the robot controller, P is the multimodal prompt sequence, and H is the trajectory history sequence. t represents the action in the P,H state at step t, π θ (a t |P,H) represents the action strategy of the first fire fighting machine.
[0080] After inputting the multimodal prompt feature vector and the target feature vector, the action decoder first extracts the key sequence from the multimodal prompt feature vector Sum value sequence V p , and retrieve the target historical action sequence Q in the trajectory history sequence according to the target feature vector H Then, based on the extracted target historical action sequence, key sequence, and value sequence, an output sequence H is generated at each cross attention ′ , which is the action sequence of the first firefighting action. The action sequence consists of multiple action feature vectors.
[0081] In some implementations, the action sequence may be generated by the following formula:
[0082]
[0083] Among them, H′ is the action sequence, softmax is the normalized exponential function, Q H is the target historical action sequence, is the key sequence, V p is a sequence of values, and d is the embedding dimension.
[0084] In some embodiments, the first multimodal information is generated by the second fire fighting machine by inputting the second multimodal information into the multimodal big model;
[0085] The second multimodal information is used to indicate the firefighting tasks at the fire scene.
[0086] Specifically, the second multimodal information is a primary firefighting plan formulated by the manual terminal according to factors such as the terrain, conditions, and distribution of rescue personnel at the fire scene. The manual terminal may be a firefighter at the fire scene, or a human being who communicates remotely with the fire scene and understands the disaster status of the fire scene.
[0087] The second fire-fighting machine is a fire-fighting machine used to command according to the fire-fighting tasks at the fire scene. The second fire-fighting machine integrates a large multimodal model, and can generate first multimodal information based on the received second multimodal information. The first multimodal information is more specific information obtained by further planning on the basis of the second multimodal information. In one example, the second multimodal information is: a fire broke out on the second floor of a building, and the burning objects, personnel and fire situation are unknown. Please enter the fire scene according to the planned route and perform the following operations: 1. Investigate the burning objects, personnel and fire; 2. Develop a fire rescue plan. The first multimodal information is instruction information that instructs a specific robot or robot dog to arrive at a certain location on the second floor of the building to perform specific actions.
[0088] It should be noted that the second fire-fighting machine can be a machine that directly performs fire-fighting tasks at the fire scene, that is, the first fire-fighting machine, or can be other fire-fighting machines other than the first fire-fighting machine. When the second fire-fighting machine and the first fire-fighting machine are the same machine, the first fire-fighting machine directly generates the first modal information based on the second multimodal information at the fire scene.
[0089] See also Figure 3 , is a schematic diagram of the architecture of a multi-modal large model provided in an embodiment of the present application. Figure 3 As shown, in some embodiments, the multimodal large model includes a multimodal encoder, an input projector, a large language model backbone network, an output projector, and a multimodal generator; the second fire fighting machine generates the first multimodal information by inputting the second multimodal information into the multimodal large model, which may include:
[0090] The multimodal encoder is used to obtain a multimodal feature vector according to the second multimodal information input;
[0091] The input projector is used to align the multimodal feature vector to the text modality space to obtain the input projection vector of the multimodal feature vector;
[0092] The large language model backbone network is used to obtain the instruction text and the signal feature vector according to the input projection vector and the feature vector of the instruction text;
[0093] The output projector is used to map the signal feature vector to the feature space of the multimodal generator to obtain an output projection vector of the signal feature vector;
[0094] The multimodal generator is used to obtain first multimodal information according to the output projection vector.
[0095] Specifically, the core function of the multimodal large model is to integrate multiple modal information, achieve data collaborative reasoning, and ultimately produce output that conforms to human values. In the embodiment of the present application, a large language model is introduced as the prior knowledge and driving force of the multimodal large model to improve model performance and reduce computing load.
[0096] The multimodal large model is mainly composed of five parts: multimodal encoder, input projector, large language model backbone network, output projector and multimodal generator. The process of pre-training the multimodal large model can be divided into multimodal understanding (covering three links: multimodal encoding, input projection and large model backbone) and multimodal generation (including two stages: output projection and multimodal generation). During the entire training process, the parameters of the multimodal encoder, generator and large language model are frozen, and the main optimization tasks focus on the input projection and output projection links, and the parameters of the two account for about 2% of the total parameters.
[0097] like Figure 3 As shown, the multimodal encoder first converts the input second multimodal information I x Mapped into a multimodal feature vector F x , the calculation formula is as follows:
[0098] F x =ME x (I x )
[0099] Among them, ME x Represents encoders corresponding to different modalities. For example, for the video modality, five video frames are uniformly sampled and then the ViT model is used to encode the input in image format.
[0100] After obtaining the multimodal feature vector F x After that, the input projector transforms the multimodal feature vector F xAlign to the text modal space where the large language model is good at, and obtain the input projection vector P of the multimodal feature vector x .
[0101] As the core of the multimodal large model, the large language model backbone network is mainly involved in pattern expression, semantic understanding, reasoning and decision output. x and the feature vector F of the input instruction text t t Produce direct text output, that is, output command text t ′ , and the signal feature vector S used to guide the multimodal generator to generate output x , the calculation formula is as follows:
[0102] t′,S x =LLM(P x ,F t )
[0103] The multimodal feature vector F x The optimization goal in the process of aligning to the text modality space where the large language model is good at is to minimize the text generation loss L corresponding to the modality x txt-gen , the calculation formula is as follows:
[0104] argmin L txt-gen LLM((P x ,F t ),t)
[0105] The specific content of the text is the instruction text related to fire rescue. In one example, the input text t is "Please use a fire extinguisher to put out the fire, and judge whether the fire in the picture is extinguished every 10 seconds to proceed to the next step", and the output text t ′ "Based on the pictures, the fire has not been extinguished yet, please continue to put out the fire."
[0106] The output projector is used to transform the signal feature vector S x Mapped to the feature space of the multimodal generator, the output projection vector H of the signal feature vector is obtained x The multimodal generator integrates a potential diffusion model, through which various modal information MG can be generated and output in various modalities. x (t) is the first multimodal information. For example, based on the current fire scene photos, the Stable Diffusion model can generate pictures after the fire is extinguished, thereby guiding the robot to perform the fire extinguishing task.
[0107] According to the output projection vector H x The optimization goal of the first multimodal information mapping process is to minimize the output projection H x Multimodal Generation MG under Text t Constraints x(t) The distance between MSE , the calculation formula is as follows:
[0108] argmin L MSE (H x ,MG x (t))
[0109] For ease of understanding, a specific embodiment is used as an example for illustration:
[0110] See also Figure 4 , is a schematic diagram of a fire control method provided by a specific embodiment of the present application. Figure 4 As shown in the figure, the firefighters formulate a firefighting plan based on factors such as the on-site terrain, fire conditions, and the distribution of people to be rescued, and send high-level multimodal instructions to the command robot 1. After receiving the multimodal instructions, the command robot 1 plans precise rescue and firefighting tasks and sends them to robots 2 and 3. The robot dog then carries out the exploration of people and fire conditions on the designated floor according to the instructions of the firefighters.
[0111] In this specific embodiment, Robot 2 and Robot 3 are the first firefighting machines, and Robot 1 is the second firefighting machine. The firefighter first writes a high-level multimodal prompt based on the multimodal information of the fire scene, and then sends it to Robot 1. Robot 1 then plans specific firefighting tasks, i.e., fine-grained multimodal prompts, based on the high-level multimodal prompts written by the firefighter. The specific firefighting tasks can be divided into firefighting tasks and rescue tasks. Robot 1 sends each specific firefighting task to Robot 2, and the rescue task to Robot 3.
[0112] During the task execution, the robot 2 can collect multimodal information at the scene, and use firefighting tools to perform firefighting tasks based on real-time multimodal information and received fine-grained prompts. After the task is completed, the task execution result is returned, which can include data such as on-site images and videos. In this way, multimodal information, execution actions, and environmental feedback data can be summarized to further train the firefighting and rescue capabilities of robots and robot dogs.
[0113] In this way, through human-machine collaboration, firefighters are responsible for dealing with complex problems and making decisions, while robots and robot dogs take on repetitive, tedious or dangerous tasks, improving the efficiency and quality of the entire firefighting work and reducing personnel risks.
[0114] See also Figure 5 , is a structural diagram of a fire control system provided in an embodiment of the present application. In a third aspect, an embodiment of the present application provides a fire control system, the system comprising:
[0115] The second fire fighting machine 510 is used to input the second multimodal information into the multimodal large model to generate the first multimodal information, and send the first multimodal information to the first fire fighting machine; wherein the second multimodal information is used to indicate the fire fighting task at the fire fighting scene, and the first multimodal information indicates the first fire fighting action related to the first object of the fire fighting task;
[0116] The first fire-fighting machine 520 communicates with the second fire-fighting machine 510, and includes a multimodal encoder, a target encoder and an action decoder, which is used to collect a fire-fighting image of the first object when receiving first multimodal information for indicating the execution of a first fire-fighting action related to the first object; convert the fire-fighting image into a target feature vector through the target encoder; input the first multimodal information into the multimodal encoder to obtain a multimodal prompt feature vector; input the multimodal prompt feature vector and the target feature vector into the action decoder to obtain an action sequence of the first fire-fighting action; and execute the action sequence.
[0117] In some embodiments, the multimodal large model includes a multimodal encoder, an input projector, a large language model backbone network, an output projector, and a multimodal generator; the second fire fighting machine is also used to:
[0118] Inputting the second multimodal information into the multimodal encoder to obtain a multimodal feature vector;
[0119] Aligning the multimodal feature vector to the text modality space through an input projector to obtain an input projection vector of the multimodal feature vector;
[0120] Input the input projection vector and the feature vector of the instruction text into the large language model backbone network to obtain the instruction text and signal feature vectors;
[0121] Mapping the signal feature vector to the feature space of the multimodal generator through the output projector to obtain an output projection vector of the signal feature vector;
[0122] The output projection vector is input into the multimodal generator to obtain the first multimodal information.
[0123] See also Figure 6 , is a schematic diagram of the structure of a fire control device provided in an embodiment of the present application. In a third aspect, an embodiment of the present application provides a fire control device 600, and the device 600 includes:
[0124] The acquisition module 610 is used to acquire a firefighting image of the first object when receiving the first multimodal information for instructing to perform a first firefighting action related to the first object;
[0125] A conversion module 620, for converting the firefighting image into a target feature vector through a target encoder;
[0126] A multimodal prompt feature vector generating module 630, configured to input the first multimodal information into a multimodal encoder to obtain a multimodal prompt feature vector;
[0127] An action sequence generation module 640, used for inputting the multimodal prompt feature vector and the target feature vector into an action decoder to obtain an action sequence of a first firefighting action;
[0128] The execution module 650 is used to execute the action sequence.
[0129] The fire control device 600 provided in the third aspect of the embodiment of the present application can implement each process implemented in the above-mentioned method embodiment and achieve the same beneficial effects. To avoid repetition, it will not be described here.
[0130] See also Figure 7 , is a structural diagram of a computer device provided in an embodiment of the present application. The fourth aspect of the embodiment of the present application provides a computer device 700, including a processor 710 and a memory 720, the memory 720 stores machine executable instructions that can be executed by the processor 710, and the processor 710 can execute the machine executable instructions to implement the above-mentioned fire control method.
[0131] A fifth aspect of an embodiment of the present application provides a machine-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor implements the above-mentioned fire control method.
[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0133] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0134] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0135] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0136] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0137] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0138] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
[0139] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.
Claims
1. A fire control method, characterized in that: Applied to a first fire-fighting machine, the first fire-fighting machine includes a multimodal encoder, a target encoder, and an action decoder, the method includes: Upon receiving first multimodal information for instructing to perform a first firefighting action related to a first object, acquiring a firefighting image of the first object; The firefighting image is converted into a target feature vector by the target encoder; Inputting the first multimodal information into the multimodal encoder to obtain a multimodal prompt feature vector; Inputting the multimodal prompt feature vector and the target feature vector into the action decoder to obtain an action sequence of the first firefighting action; Execute the sequence of actions.
2. The method according to claim 1, characterized in that After inputting the multimodal prompt feature vector and the target feature vector, the action decoder performs the following steps: extracting a key sequence and a value sequence from the multimodal prompt feature vector; Retrieving a target historical action sequence in a trajectory history sequence according to the target feature vector; An action sequence of the first firefighting action is generated based on the target historical action sequence, the key sequence and the value sequence.
3. The method according to claim 2, characterized in that The action sequence is generated by the following formula: Among them, H′ is the action sequence, softmax is the normalized exponential function, Q H is the target historical action sequence, is the key sequence, V p is a sequence of values, and d is the embedding dimension.
4. The method according to claim 1, characterized in that: The first multimodal information is generated by the second fire fighting machine by inputting the second multimodal information into the multimodal large model; The second multimodal information is used to indicate the firefighting tasks at the fire scene.
5. The method according to claim 4, characterized in that The multimodal large model includes a multimodal encoder, an input projector, a large language model backbone network, an output projector, and a multimodal generator; the second fire fighting machine generates the first multimodal information by inputting the second multimodal information into the multimodal large model, including: The multimodal encoder is used to obtain a multimodal feature vector according to the second multimodal information input; The input projector is used to align the multimodal feature vector to the text modal space to obtain an input projection vector of the multimodal feature vector; The large language model backbone network is used to obtain the instruction text and the signal feature vector according to the input projection vector and the feature vector of the instruction text; The output projector is used to map the signal feature vector to the feature space of the multimodal generator to obtain an output projection vector of the signal feature vector; The multimodal generator is used to obtain first multimodal information according to the output projection vector.
6. A fire control system, characterized in that: The system comprises: The second fire-fighting machine is used to input the second multimodal information into the multimodal large model to generate the first multimodal information, and send the first multimodal information to the first fire-fighting machine; wherein the second multimodal information is used to indicate the fire-fighting task of the fire scene, and the first multimodal information indicates the first fire-fighting action related to the first object of the fire-fighting task; The first fire-fighting machine communicates with the second fire-fighting machine, and includes a multimodal encoder, a target encoder and an action decoder, which is used to collect a fire-fighting image of the first object when receiving first multimodal information for indicating the execution of a first fire-fighting action related to the first object; convert the fire-fighting image into a target feature vector through the target encoder; input the first multimodal information into the multimodal encoder to obtain a multimodal prompt feature vector; input the multimodal prompt feature vector and the target feature vector into the action decoder to obtain an action sequence of the first fire-fighting action; and execute the action sequence.
7. The system according to claim 6, characterized in that The multimodal large model includes a multimodal encoder, an input projector, a large language model backbone network, an output projector and a multimodal generator; the second fire fighting machine is also used for: Inputting the second multimodal information into the multimodal encoder to obtain a multimodal feature vector; Aligning the multimodal feature vector to the text modal space through the input projector to obtain an input projection vector of the multimodal feature vector; Inputting the input projection vector and the feature vector of the instruction text into the large language model backbone network to obtain the instruction text and the signal feature vector; Mapping the signal feature vector to the feature space of the multimodal generator through the output projector to obtain an output projection vector of the signal feature vector; The output projection vector is input into the multimodal generator to obtain first multimodal information.
8. A fire control device, characterized in that: The device comprises: a collection module, configured to collect a firefighting image of a first object upon receiving first multimodal information for instructing to perform a first firefighting action related to the first object; A conversion module, used for converting the firefighting image into a target feature vector through the target encoder; a multimodal prompt feature vector generating module, configured to input the first multimodal information into the multimodal encoder to obtain a multimodal prompt feature vector; An action sequence generation module, used for inputting the multimodal prompt feature vector and the target feature vector into the action decoder to obtain an action sequence of the first firefighting action; An execution module is used to execute the action sequence.
9. A computer device, characterized in that: The computer device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the fire control method described in any one of claims 1 to 5 when executing the program stored in the memory.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the fire control method according to any one of claims 1 to 5 are implemented.