Fire-fighting task execution method, device and equipment based on fire-fighting robot and medium

Abstract

The embodiment of the invention discloses a fire-fighting task execution method and device based on a fire-fighting robot, equipment and a medium. The method comprises the following steps: performing text conversion on voice information for indicating a fire-fighting task to obtain text information; performing task decomposition on the text information based on a pre-trained fire-fighting task decomposition model to obtain a decomposition instruction; the fire-fighting task decomposition model is a model subjected to retrieval enhancement generation processing based on a fire-fighting task instruction library; and performing action control on the current fire-fighting robot step by step based on the decomposition instruction so as to enable the current fire-fighting robot to complete a fire-fighting task. According to the technical scheme, task decomposition is performed by retrieving the enhanced fire-fighting task decomposition model, and the decomposition instruction which can be executed by the fire-fighting robot is obtained, so that the fire-fighting robot can comprehensively understand and execute the fire-fighting task.

Description

Technical Field

[0001] The present invention relates to the technical field of firefighting robots, and in particular to a firefighting task execution method, device, equipment and medium based on a firefighting robot. Background Art

[0002] With the advancement of science and technology, the application of robot technology in emergency rescue has gradually attracted attention. At present, there are some robot products for emergency rescue on the market, such as firefighting robots for fire rescue.

[0003] Existing firefighting robots can usually only perform a single task when performing firefighting tasks, such as controlling the water outlet of a fire hydrant to aim at the fire source. In complex and changeable disaster scenarios, they cannot fully understand and execute firefighting tasks. Summary of the invention

[0004] The present invention provides a firefighting task execution method, device, equipment and medium based on a firefighting robot, which can decompose a received firefighting task and execute the decomposed firefighting task.

[0005] According to one aspect of the present invention, a method for executing a firefighting task based on a firefighting robot is provided, the method comprising:

[0006] Performing text conversion on the voice information for indicating the firefighting task to obtain text information;

[0007] Based on a pre-trained fire task decomposition model, the text information is decomposed into tasks to obtain decomposition instructions; the fire task decomposition model is a model generated after retrieval enhancement based on a fire task instruction library;

[0008] Based on the decomposed instructions, the action of the current fire-fighting robot is gradually controlled so that the current fire-fighting robot can complete the fire-fighting task.

[0009] According to another aspect of the present invention, a fire fighting task execution device based on a fire fighting robot is provided, comprising:

[0010] A text conversion module, used for converting the voice information used to indicate the firefighting task into text information to obtain text information;

[0011] A task decomposition module, used to perform task decomposition on the text information based on a pre-trained fire task decomposition model to obtain decomposition instructions; the fire task decomposition model is a model generated after retrieval enhancement based on a fire task instruction library;

[0012] The action control module is used to gradually control the action of the current fire-fighting robot based on the decomposed instructions so that the current fire-fighting robot can complete the fire-fighting task.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the firefighting task execution method based on the firefighting robot described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the firefighting task execution method based on a firefighting robot described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present application includes: converting the voice information used to indicate the firefighting task into text information; performing task decomposition on the text information based on a pre-trained firefighting task decomposition model to obtain decomposition instructions; the firefighting task decomposition model is a model that has been retrieved, enhanced, and generated based on a firefighting task instruction library; and gradually controlling the actions of the current firefighting robot based on the decomposition instructions so that the current firefighting robot completes the firefighting task. The technical solution decomposes the task by retrieving the enhanced firefighting task decomposition model, and obtains the decomposition instructions that the firefighting robot can execute, so that the firefighting robot can fully understand and execute the firefighting task.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flowchart of a fire fighting task execution method based on a fire fighting robot provided according to the first embodiment of the present application;

[0022] Figure 2It is a flowchart of a fire fighting task execution method based on a fire fighting robot provided according to the second embodiment of the present application;

[0023] Figure 3 It is a structural schematic diagram of a fire-fighting task execution device based on a fire-fighting robot provided according to the third embodiment of the present application;

[0024] Figure 4 It is a structural schematic diagram of an electronic device for implementing a firefighting task execution method based on a firefighting robot according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 A flowchart of a method for executing a firefighting task based on a firefighting robot is provided for the first embodiment of the present application. The embodiment of the present application can be applied to the situation where a firefighting robot executes a firefighting task. The method can be executed by a firefighting task execution device based on a firefighting robot. The firefighting task execution device based on a firefighting robot can be implemented in the form of hardware and / or software. The firefighting task execution device based on a firefighting robot can be configured in an electronic device with data processing capability, and the electronic device can be located in the firefighting robot. Figure 1 As shown, the method includes:

[0029] S110, converting the voice information for indicating the firefighting task into text to obtain text information.

[0030] The technical solution of the embodiment of the present application can be executed by a firefighting robot, and the technical solution can be applied to the situation where a firefighting task is executed according to the voice information of the firefighting task in a firefighting scene. The voice information for indicating the firefighting task can be the voice information of the firefighter, and the voice information can be the voice information sent by the firefighter to the firefighting robot when a fire occurs.

[0031] Specifically, if a fire occurs in a building, firefighters can send a voice message to the fire robot, and the voice message is used to indicate the firefighting task. In this case, if a voice message indicating the firefighting task is received, the voice message is converted into text to obtain text information, so as to perform subsequent task decomposition and execution according to the text information.

[0032] S120, performing task decomposition on the text information based on a pre-trained fire task decomposition model to obtain decomposition instructions.

[0033] Among them, the fire task decomposition model can understand the meaning of the fire task in the text information and perform task decomposition. The fire task decomposition model can be constructed based on a large language model and a task decomposition (HTN) method. The fire task decomposition model is a model that has been retrieval-enhanced and generated based on a fire task instruction library.

[0034] Specifically, since the firefighting task decomposition model is used to decompose the firefighting task, and the decomposed instructions obtained need to be instructions that can be executed by the firefighting robot, the firefighting task decomposition model is retrieved and enhanced in advance so that the decomposed instructions obtained are instructions that can be executed by the firefighting robot.

[0035] In an embodiment of the present application, when performing retrieval enhancement processing on the fire task decomposition model, retrieval enhancement can be performed based on the fire task instruction library. Since the fire task instruction library reflects the operations that the fire robot can perform and the targets that may be involved when performing the task, the decomposition instructions obtained by the fire task decomposition model after the retrieval enhancement processing are instructions executable by the fire robot.

[0036] S130, based on the decomposed instructions, gradually controlling the actions of the current firefighting robot so that the current firefighting robot completes the firefighting task.

[0037] Specifically, after receiving the decomposition instruction, the decomposition instruction can be gradually executed by the current fire-fighting robot until the fire-fighting task is completed.

[0038] Exemplarily, in one case, the decomposition instruction may be to find a fire extinguisher, move from the fire extinguisher location to the fire source location, and use the fire extinguisher to extinguish the fire source; in this case, the current firefighting robot can generate a path from the current position to the fire extinguisher location through a path planning algorithm, and then the current firefighting robot controls the lower limbs to walk to the fire extinguisher location through the control module, and then picks up the fire extinguisher through the control module, controls the current firefighting robot to move to the fire source location and extinguishes the fire through the fire extinguisher.

[0039] The technical solution of the embodiment of the present application includes: converting the voice information used to indicate the firefighting task into text information; performing task decomposition on the text information based on a pre-trained firefighting task decomposition model to obtain decomposition instructions; the firefighting task decomposition model is a model that has been retrieved, enhanced, and generated based on a firefighting task instruction library; and gradually controlling the actions of the current firefighting robot based on the decomposition instructions so that the current firefighting robot completes the firefighting task. The technical solution decomposes the task by retrieving the enhanced firefighting task decomposition model, and obtains the decomposition instructions that the firefighting robot can execute, so that the firefighting robot can fully understand and execute the firefighting task.

[0040] Embodiment 2

[0041] Figure 2 A flowchart of a method for executing a firefighting task based on a firefighting robot is provided in Embodiment 2 of the present application. The embodiment of the present application is optimized based on the above embodiment.

[0042] like Figure 2 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0043] S210, converting the voice information for indicating the firefighting task into text to obtain text information.

[0044] Exemplarily, through automatic speech recognition (ASR) technology, voice information indicating firefighting tasks is converted into text information to facilitate subsequent intelligent processing and decision-making.

[0045] S220, performing task decomposition on the text information based on a pre-trained fire task decomposition model to obtain decomposition instructions.

[0046] In the embodiment of the present application, when processing text information, the text information can be understood based on the large language model and task decomposition (HTN) technology, the firefighting task can be determined, and the firefighting task can be decomposed to obtain decomposed instructions that can be executed by the firefighting robot. That is, the firefighting task decomposition model can be constructed based on the large language model and task decomposition.

[0047] Specifically, since the firefighting task reflected in the text information is usually to extinguish the fire source, in this case, if the firefighting task decomposition model has not undergone retrieval enhancement processing, the decomposition instructions obtained by its decomposition may be to dial a fire alarm number, etc. This situation is not suitable for execution by a firefighting robot. Therefore, the embodiment of the present application performs retrieval enhancement processing on the firefighting task decomposition model through a firefighting task instruction library, so that the decomposition instructions are instructions executable by the firefighting robot.

[0048] The firefighting task decomposition model is a model generated after retrieval enhancement based on a firefighting task instruction library; in an embodiment of the present application, optionally, the firefighting task instruction library includes: a firefighting robot action library, a firefighting target library and a firefighting map library; the firefighting robot action library includes the firefighting robot's lower limb walking information and upper limb control information; the firefighting target library reflects the firefighting equipment information that can be controlled by the firefighting robot; the firefighting map library reflects the spatial information of the firefighting robot's action area.

[0049] Among them, the fire robot action library reflects the action operations that the fire robot can perform, which may include the walking information of the lower limbs of the fire robot and the control information of the upper limbs. The walking of the lower limbs may include: moving forward, backward, and climbing stairs, etc.; the upper limb control information may include: taking the fire extinguisher / fire hydrant outlet, using the fire extinguisher, etc. The fire target library reflects the fire equipment information that the fire robot can control, and the fire target library includes: fire extinguishers, fire hydrants, and fire blankets, etc.; the fire map library reflects the spatial information of the action area of ​​the fire robot, such as warehouses, stairwells, etc.

[0050] In an embodiment of the present application, optionally, the process of determining the fire task decomposition model after retrieval enhancement generation includes: indexing and constructing the fire task instruction library; if the fire task decomposition model receives input data, searching the fire task instruction library according to the input data; if index content of the fire robot action library, the fire target library or the fire map library is retrieved in the fire task instruction library, the index content is integrated into the fire task decomposition model to obtain the fire task decomposition model after retrieval enhancement generation.

[0051] Specifically, after determining the fire task instruction library, an index of the fire task instruction library is established. After the fire task decomposition model receives the input data, it searches the fire task instruction library according to the information in the input data. If it is determined that the index content is retrieved, the features corresponding to the index content can be fused with the features corresponding to the input data, so that when the fire task decomposition model outputs the decomposition instructions, it can obtain the decomposition instructions associated with the fire task instruction library.

[0052] This scheme is configured so that after task decomposition based on the fire task decomposition model, the decomposition result obtained is that the fire robot can execute the decomposition instructions, thereby avoiding the problem that the fire robot may not be able to execute the decomposition instructions and improving the intelligence level of the fire robot.

[0053] In an embodiment of the present application, in one case, the fire-fighting robot can construct a three-dimensional model based on the stereoscopic visual information of the area corresponding to the fire-fighting task, and the three-dimensional model can reflect the information of all areas corresponding to the fire-fighting task. In this case, the environmental targets determined in the three-dimensional model based on the semantic segmentation (CLIP) technology are the targets that may be involved in the process of executing the fire-fighting task. Therefore, the environmental targets can be input into the fire-fighting task decomposition model together with the text information to obtain decomposition instructions suitable for the current environment.

[0054] In an embodiment of the present application, optionally, the method also includes: obtaining a three-dimensional model of the area corresponding to the firefighting task; determining the environmental target in the three-dimensional model based on semantic segmentation technology; accordingly, performing task decomposition on the text information based on a pre-trained firefighting task decomposition model to obtain decomposition instructions, including: inputting the environmental target and the text information into a pre-trained firefighting task decomposition model to obtain decomposition instructions for the firefighting environment target; accordingly, based on the decomposition instructions, gradually controlling the action of the current firefighting robot so that the current firefighting robot completes the firefighting task, including: determining the firefighting environment target in the three-dimensional model; determining the movement trajectory of the firefighting robot based on the position information of the firefighting environment target in the three-dimensional model and the decomposition instructions; and controlling the movement of the current firefighting robot based on the movement trajectory of the firefighting robot so that the current firefighting robot completes the movement task in the firefighting task.

[0055] Specifically, the depth information can be obtained based on the stereoscopic vision of the area corresponding to the firefighting task, and a three-dimensional model can be constructed based on the depth information and other scanning information. Then, the environmental targets that may be associated with the firefighting task can be determined in the three-dimensional model through semantic segmentation technology. Since the firefighting task decomposition model is constructed based on the large language model, the environmental targets and text information are input into the firefighting task decomposition model. The model can understand the firefighting task in the current scenario and make task decomposition to obtain decomposition instructions. Exemplarily, if the environmental targets include: fire extinguishers, fire sources, gates, etc., and the text information is: use the fire extinguisher at the entrance to extinguish the fire, then the decomposition instructions output by the firefighting task decomposition model can be: determine the path to the fire extinguisher at the entrance, pick up the fire extinguisher at the entrance, determine the path from the fire extinguisher position to the fire extinguishing position, and extinguish the fire with the fire extinguisher at the fire extinguishing position.

[0056] Furthermore, in firefighting tasks, moving tasks are usually involved, such as avoiding a carpet to reach a fire extinguisher. In this case, the decomposition instructions may involve firefighting environment targets (i.e., the environmental targets involved in the decomposition instructions) carpets and fire extinguishers. Therefore, the firefighting environment targets can be determined in the three-dimensional model; the firefighting robot movement trajectory is generated based on the position information of the firefighting environment targets in the three-dimensional model and the decomposition instructions; when generating the movement trajectory, the shortest path can be planned through the A* algorithm to guide the robot to perform autonomous navigation.

[0057] In an embodiment of the present application, optionally, the movement of the current firefighting robot is controlled based on the movement trajectory of the firefighting robot so that the current firefighting robot completes the movement task in the firefighting mission, including: if it is determined that there is an obstacle on the current route during the movement of the current firefighting robot, then an adjustment trajectory of the firefighting robot is generated based on the position of the obstacle and the remaining movement trajectory; the movement of the current firefighting robot is controlled based on the adjustment trajectory of the firefighting robot so that the current firefighting robot completes the movement task in the firefighting mission.

[0058] For example, in a firefighting scenario, as the fire develops, objects may collapse and the originally passable roads may be affected. In response to the appearance of obstacles on the current route, an adjustment trajectory of the firefighting robot can be generated based on the position of the obstacle and the remaining movement trajectory (the remaining movement trajectory is the movement trajectory of the firefighting robot minus the part of the road that the firefighting robot has passed).

[0059] The present scheme is configured in such a way that, in the process of performing firefighting tasks, the firefighting robot can generate a movement trajectory suitable for the current environment and firefighting tasks based on the three-dimensional model, the location information of the firefighting environment target and the decomposed instructions. In addition, the firefighting robot can adaptively generate and adjust the movement trajectory during the movement, which greatly improves the firefighting robot's ability to perform firefighting tasks and provides support for fast and accurate firefighting.

[0060] In an embodiment of the present application, optionally, the method also includes: obtaining a current three-dimensional model that has been constructed; determining a current target in the current three-dimensional model based on semantic segmentation technology; correspondingly, performing task decomposition on the text information based on a pre-trained fire task decomposition model to obtain decomposition instructions, including: inputting the current target and the text information into a pre-trained fire task decomposition model to obtain decomposition instructions; correspondingly, gradually controlling the action of the current fire robot based on the decomposition instructions so that the current fire robot completes the fire task, including: if the decomposition instructions contain necessary fire targets that are not in the current target, then during the movement of the current fire robot, constructing a three-dimensional model of the surrounding environment through stereo vision technology to obtain an updated three-dimensional model; determining the movement trajectory of the fire robot based on the position information of each target in the updated three-dimensional model and the decomposition instructions; and controlling the movement of the current fire robot based on the movement trajectory of the fire robot so that the current fire robot completes the movement task in the fire task.

[0061] Specifically, in another case, the fire-fighting robot obtains part of the environmental information through three-dimensional reconstruction, that is, the current three-dimensional model is obtained by scanning the fire-fighting robot, and there may be some areas that are not scanned. In this case, the current target and the text information can be input into a pre-trained fire-fighting task decomposition model to obtain a decomposition instruction; if the decomposition instruction contains a necessary fire-fighting target that is not in the current target (for example, the decomposition instruction contains a situation where a fire extinguisher is found, but there is no fire extinguisher in the current target, and the fire extinguisher is the necessary fire-fighting target at this time), then during the current movement of the fire-fighting robot, the three-dimensional model of the surrounding environment is continuously constructed through stereo vision technology, so that the current three-dimensional model is improved and an updated three-dimensional model is obtained; if there is a necessary fire-fighting target in the updated three-dimensional model, the movement trajectory of the fire-fighting robot is determined based on the position information of each target and the decomposition instruction.

[0062] This scheme is set up so that the fire-fighting robot can perform fire-fighting tasks even in unfamiliar environments, greatly improving the scene adaptability.

[0063] Specifically, when executing decomposition instructions, it will also involve identifying the corresponding target in the 3D model according to the target information corresponding to the decomposition instructions. The semantic alignment (ALIGN) technology can be used to match the image and text information to help the robot better understand the environment and tasks. In the fire-fighting scenario, ALIGN can help the robot align the location of the fire source with the location of the fire extinguisher to improve the efficiency of fire-fighting.

[0064] Exemplarily, the firefighting robot executes decomposition instructions through ROS (Robot Operating System), for example, it can walk to the fire extinguisher through the lower limbs, pick up the fire extinguisher through the upper limbs, perform fire extinguishing operations according to the fire extinguishing strategy, and spray the fire extinguishing agent.

[0065] S230: If a mid-course voice message indicating a firefighting task is received, the mid-course voice message is converted into text to obtain a task adjustment text.

[0066] In the embodiment of the present application, when the firefighting robot performs a firefighting task, it is actually executing the decomposed instructions one by one. During the execution process, it can provide real-time progress feedback to the firefighters, such as communicating with the firefighters through voice. Voice output is an important way for robots to interact with people. Through text-to-speech (TTS) technology, text information is converted into voice signals to provide real-time voice feedback and instructions. In the fire-fighting scenario, the main tasks of voice output include feedback on the current task status, prompting possible dangers, and communicating with other rescuers. In the fire-fighting scenario, TTS can help the robot feedback the current task status, such as "fire extinguisher found", "moving to the fire source", etc.

[0067] Specifically, when the firefighting robot communicates with firefighters, there may be a situation where the firefighters adjust the firefighting task. That is, if a mid-way voice message for indicating a firefighting task is received, the mid-way voice message is converted into text to obtain a task adjustment text.

[0068] S240, processing the adjustment text and the executed decomposition tasks based on the pre-trained firefighting task decomposition model to obtain an update decomposition instruction.

[0069] S250, based on the updated decomposition instruction, gradually controlling the action of the current firefighting robot so that the current firefighting robot completes the firefighting task.

[0070] In the embodiment of the present application, since the firefighting robot can interact with firefighters, it can obtain task adjustment text in a timely manner, and then generate updated decomposition instructions suitable for the latest firefighting tasks based on the executed decomposition tasks (i.e., decomposition instructions) and the adjustment text, thereby greatly improving the flexibility of the firefighting robot in performing firefighting tasks, and can make adaptive adjustments to emergencies.

[0071] Embodiment 3

[0072] Figure 3 This is a schematic diagram of the structure of a firefighting task execution device based on a firefighting robot provided in the third embodiment of the present application. The device can execute the firefighting task execution method based on a firefighting robot provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 3As shown, the device comprises:

[0073] A text conversion module 310, used for converting the voice information for indicating the firefighting task into text information;

[0074] The task decomposition module 320 is used to perform task decomposition on the text information based on a pre-trained fire task decomposition model to obtain decomposition instructions; the fire task decomposition model is a model generated after retrieval enhancement based on a fire task instruction library;

[0075] The action control module 330 is used to gradually control the action of the current fire-fighting robot based on the decomposed instructions so that the current fire-fighting robot can complete the fire-fighting task.

[0076] The technical solution of the embodiment of the present application includes: a text conversion module 310, which is used to convert the voice information used to indicate the firefighting task into text information; a task decomposition module 320, which is used to decompose the text information based on a pre-trained firefighting task decomposition model to obtain decomposition instructions; the firefighting task decomposition model is a model generated after retrieval enhancement based on a firefighting task instruction library; an action control module 330, which is used to gradually control the action of the current firefighting robot based on the decomposition instructions, so that the current firefighting robot completes the firefighting task. This technical solution decomposes the task by retrieving the enhanced firefighting task decomposition model, and obtains the decomposition instructions that the firefighting robot can execute, so that the firefighting robot can fully understand and execute the firefighting task.

[0077] In the embodiment of the present application, optionally, the firefighting task instruction library includes: a firefighting robot action library, a firefighting target library and a firefighting map library;

[0078] The fire-fighting robot action library includes the fire-fighting robot's lower limb walking information and upper limb control information;

[0079] The fire target library reflects the fire equipment information that can be controlled by the fire robot;

[0080] The firefighting map library reflects the spatial information of the action area of ​​the firefighting robot.

[0081] In the embodiment of the present application, optionally, the device further includes: a determination module of a fire task decomposition model, specifically used to:

[0082] Indexing and constructing the firefighting task instruction library;

[0083] If the fire task decomposition model receives input data, searching the fire task instruction library according to the input data;

[0084] If the index content of the fire robot action library, the fire target library or the fire map library is retrieved in the fire task instruction library, the index content is integrated into the fire task decomposition model to obtain the fire task decomposition model generated after retrieval enhancement.

[0085] In the embodiment of the present application, optionally, the device further includes:

[0086] A three-dimensional model acquisition module is used to obtain a three-dimensional model of the area corresponding to the firefighting task;

[0087] An environmental target determination module, used for determining environmental targets in the three-dimensional model based on semantic segmentation technology;

[0088] Accordingly, the task decomposition module 320 includes:

[0089] A task decomposition unit, used for inputting the environmental target and the text information into a pre-trained fire task decomposition model to obtain a decomposition instruction for the fire environment target;

[0090] Accordingly, the action control module 330 includes:

[0091] A fire environment target determination unit, used to determine a fire environment target in the three-dimensional model;

[0092] A movement trajectory determination unit, used to determine the movement trajectory of the fire-fighting robot based on the position information of the fire-fighting environment target in the three-dimensional model and the decomposition instruction;

[0093] The mobile control unit is used to control the movement of the current fire-fighting robot based on the movement trajectory of the fire-fighting robot, so that the current fire-fighting robot can complete the movement task in the fire-fighting task.

[0094] In an embodiment of the present application, optionally, the mobile control unit includes:

[0095] The adjustment trajectory generation subunit is used to generate an adjustment trajectory of the fire-fighting robot based on the position of the obstacle and the remaining movement trajectory if it is determined that there is an obstacle on the current travel route during the current movement of the fire-fighting robot;

[0096] The mobile control subunit is used to control the movement of the current fire-fighting robot based on the fire-fighting robot's adjusted trajectory, so that the current fire-fighting robot can complete the movement task in the fire-fighting task.

[0097] In the embodiment of the present application, optionally, the device further includes:

[0098] The current three-dimensional model acquisition module is used to acquire the constructed current three-dimensional model;

[0099] A current target determination module, used for determining a current target in the current three-dimensional model based on a semantic segmentation technique;

[0100] Accordingly, the task decomposition module 320 includes:

[0101] A task decomposition unit, used for inputting the current target and the text information into a pre-trained fire task decomposition model to obtain a decomposition instruction;

[0102] Accordingly, the action control module 330 includes:

[0103] an updated three-dimensional model construction unit, configured to construct a three-dimensional model of the surrounding environment by using stereo vision technology during the movement of the current fire-fighting robot, if the decomposition instruction contains a necessary fire-fighting target that is not included in the current target, to obtain an updated three-dimensional model;

[0104] A movement trajectory determination unit, used to determine the movement trajectory of the fire-fighting robot based on the position information of each target in the updated three-dimensional model and the decomposition instruction;

[0105] The mobile control unit is used to control the movement of the current fire-fighting robot based on the movement trajectory of the fire-fighting robot, so that the current fire-fighting robot can complete the movement task in the fire-fighting task.

[0106] In the embodiment of the present application, optionally, the action control module 330 includes:

[0107] A task adjustment text determination unit, configured to, if receiving midway voice information for indicating a firefighting task, convert the midway voice information into text to obtain a task adjustment text;

[0108] An update decomposition instruction determination unit is used to process the adjustment text and the executed decomposition task based on a pre-trained fire task decomposition model to obtain an update decomposition instruction;

[0109] The action control unit is used to gradually control the action of the current fire-fighting robot based on the updated decomposition instructions so that the current fire-fighting robot can complete the fire-fighting task.

[0110] A firefighting task execution device based on a firefighting robot provided in an embodiment of the present application can execute a firefighting task execution method based on a firefighting robot provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method.

[0111] Embodiment 4

[0112] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0113] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a firefighting task execution method based on a firefighting robot.

[0116] In some embodiments, the firefighting task execution method based on the firefighting robot can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the firefighting task execution method based on the firefighting robot described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the firefighting task execution method based on the firefighting robot by any other appropriate means (for example, by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0122] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A firefighting task execution method based on a firefighting robot, characterized in that: include: Performing text conversion on the voice information for indicating the firefighting task to obtain text information; Based on a pre-trained fire task decomposition model, the text information is decomposed into tasks to obtain decomposition instructions; the fire task decomposition model is a model generated after retrieval enhancement based on a fire task instruction library; Based on the decomposed instructions, the current fire-fighting robot is gradually controlled to perform actions so that the current fire-fighting robot can complete the fire-fighting task.

2. The method according to claim 1, characterized in that The firefighting task instruction library includes: a firefighting robot action library, a firefighting target library and a firefighting map library; The fire-fighting robot action library includes the fire-fighting robot's lower limb walking information and upper limb control information; The fire target library reflects the fire equipment information that can be controlled by the fire robot; The firefighting map library reflects the spatial information of the action area of ​​the firefighting robot.

3. The method according to claim 2, characterized in that The process of determining the firefighting task decomposition model after retrieval enhancement generation includes: Indexing and constructing the firefighting task instruction library; If the fire task decomposition model receives input data, searching the fire task instruction library according to the input data; If the index content of the fire robot action library, the fire target library or the fire map library is retrieved in the fire task instruction library, the index content is integrated into the fire task decomposition model to obtain the fire task decomposition model generated after retrieval enhancement.

4. The method according to claim 1, characterized in that: The method further comprises: Obtain a three-dimensional model of the area corresponding to the firefighting mission; Determining environmental targets in the three-dimensional model based on semantic segmentation technology; Accordingly, the text information is decomposed into tasks based on the pre-trained fire task decomposition model to obtain decomposition instructions, including: Inputting the environmental target and the text information into a pre-trained fire task decomposition model to obtain decomposition instructions for the fire environment target; Accordingly, based on the decomposed instructions, the current fire-fighting robot is gradually controlled to perform actions so that the current fire-fighting robot completes the fire-fighting task, including: determining a fire environment target in the three-dimensional model; Determine the movement trajectory of the firefighting robot based on the position information of the firefighting environment target in the three-dimensional model and the decomposition instruction; The movement of the current fire-fighting robot is controlled based on the movement trajectory of the fire-fighting robot, so that the current fire-fighting robot completes the movement task in the fire-fighting task.

5. The method according to claim 4, characterized in that The mobile control of the current fire-fighting robot is performed based on the mobile trajectory of the fire-fighting robot, so that the current fire-fighting robot completes the mobile task in the fire-fighting task, including: If it is determined that there is an obstacle on the current route during the movement of the fire-fighting robot, an adjustment trajectory of the fire-fighting robot is generated based on the position of the obstacle and the remaining movement trajectory; Based on the fire-fighting robot adjustment trajectory, the current fire-fighting robot is controlled to move so that the current fire-fighting robot completes the moving task in the fire-fighting task.

6. The method according to claim 1, characterized in that The method further comprises: Get the current 3D model that has been constructed; Determine a current target in the current three-dimensional model based on semantic segmentation technology; Accordingly, the text information is decomposed into tasks based on the pre-trained fire task decomposition model to obtain decomposition instructions, including: Inputting the current target and the text information into a pre-trained fire task decomposition model to obtain a decomposition instruction; Accordingly, based on the decomposed instructions, the current fire-fighting robot is gradually controlled to perform actions so that the current fire-fighting robot completes the fire-fighting task, including: If the decomposition instruction contains a necessary firefighting target that is not included in the current target, during the current movement of the firefighting robot, a three-dimensional model of the surrounding environment is constructed by using stereo vision technology to obtain an updated three-dimensional model; Determine the movement trajectory of the fire-fighting robot based on the position information of each target in the updated three-dimensional model and the decomposition instruction; The movement of the current fire-fighting robot is controlled based on the movement trajectory of the fire-fighting robot, so that the current fire-fighting robot completes the movement task in the fire-fighting task.

7. The method according to claim 1, characterized in that Based on the decomposed instructions, the current fire-fighting robot is gradually controlled to perform actions so that the current fire-fighting robot completes the fire-fighting task, including: If a mid-course voice message for indicating a firefighting task is received, the mid-course voice message is converted into text to obtain a task adjustment text; Processing the adjustment text and the executed decomposition tasks based on a pre-trained firefighting task decomposition model to obtain an updated decomposition instruction; Based on the updated decomposition instructions, the current fire-fighting robot is gradually controlled to perform actions so that the current fire-fighting robot can complete the fire-fighting task.

8. A fire-fighting task execution device based on a fire-fighting robot, characterized in that: include: A text conversion module, used for converting the voice information used to indicate the firefighting task into text information to obtain text information; A task decomposition module, used to perform task decomposition on the text information based on a pre-trained fire task decomposition model to obtain decomposition instructions; the fire task decomposition model is a model generated after retrieval enhancement based on a fire task instruction library; The action control module is used to gradually control the action of the current fire-fighting robot based on the decomposed instructions so that the current fire-fighting robot can complete the fire-fighting task.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the firefighting task execution method based on the firefighting robot as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the firefighting task execution method based on a firefighting robot according to any one of claims 1 to 7 when executed.

Images