Search and rescue robot collaborative decision-making method based on language guidance

Abstract

The invention discloses a search and rescue robot collaborative decision-making method based on language guidance, and relates to the technical field of robotics.The search and rescue robot collaborative decision-making method based on language guidance comprises the steps that a real-time video, an allowable action set and search and rescue task cues are obtained, the real-time video is input into a video text coding model, a video description text is obtained, and the video description text is input into a search and rescue robot; and inputting the video description text, the allowed action set and the search and rescue task cue word into a natural language guidance model to obtain an action sequence decision result, the natural language guidance model being obtained by training a basic model based on a video text sample and an action sequence sample. According to the application, the real-time video is converted into the corresponding text through the video text coding model, and the action sequence required by the current task is determined through the natural language guidance model according to the video text, the cue word and the allowed action set, so that the search and rescue robot is not allocated to the task not required by the current rescue environment; therefore, the rescue effect can be improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a language-guided collaborative decision-making method for search and rescue robots. Background Technology

[0002] With the rapid development of technologies such as automatic control, artificial intelligence, 5G, and high-performance computing, using rescue robots for disaster relief has become an important direction for disaster relief efforts. To improve the effectiveness of disaster relief, efficient decision-making methods are needed to enable rescue robots with different strengths to collaborate.

[0003] Current decision-making methods for rescue robots can only assign search and rescue robots to tasks they have been trained for, while assigning them to tasks that are not necessary for the current rescue environment. This results in poor rescue effectiveness when using current methods for post-disaster relief. Summary of the Invention

[0004] The main purpose of this application is to provide a language-guided collaborative decision-making method for search and rescue robots, which aims to solve the technical problem that current methods have poor rescue effectiveness in post-disaster relief.

[0005] To achieve the above objectives, this application proposes a language-guided collaborative decision-making method for search and rescue robots, the method comprising: Acquire real-time video collected by the search and rescue robot, the set of permitted actions of the search and rescue robot, and search and rescue mission prompts; The real-time video is input into a preset video text encoding model to obtain the video description text corresponding to the real-time video. The video text encoding model is trained on a preset model to be trained based on preset search and rescue image samples and image text samples. The video description text, the allowed action set, and the search and rescue task prompts are input into a preset natural language guidance model to obtain the action sequence decision results of the search and rescue robot. The natural language guidance model is trained on a preset base model based on preset video text samples and action sequence samples corresponding to the video text samples.

[0006] In one embodiment, before the step of inputting the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model to obtain the action sequence decision result of the search and rescue robot, the method further includes: Obtain the base model, the video text samples, the action sequence samples, and the allowed action set, wherein the base model is a pre-trained large language model; The video text sample is input into the base model. Based on the base model, multiple target actions are determined from the set of allowed actions and a sequence of target actions is formed. Based on the target action sequence and the action sequence samples, the base model is trained using a preset reinforcement learning training paradigm to obtain the natural language guidance model.

[0007] In one embodiment, the step of training the base model based on the target action sequence and the action sequence samples using a preset reinforcement learning training paradigm to obtain the natural language guidance model includes: Calculate the sequence similarity between the target action sequence and the action sequence sample; If the sequence similarity is higher than a preset similarity threshold, then the reward value of the target action sequence is determined based on the sequence similarity. If the sequence similarity is lower than the similarity threshold, then the penalty value of the target action sequence is determined based on the sequence similarity. Based on the reward value or the penalty value, the parameters of the base model are adjusted to obtain the natural language guidance model.

[0008] In one embodiment, before the step of inputting the real-time video into a preset video text encoding model to obtain the video description text corresponding to the real-time video, the method further includes: Acquire search and rescue image samples, image text samples corresponding to the search and rescue image samples, and the model to be trained, wherein the model to be trained is obtained by pre-training based on a preset visual dataset; The search and rescue image samples and the image text samples are encoded to obtain the search and rescue image codes corresponding to the search and rescue image samples and the image text codes corresponding to the image text samples; Select a target search and rescue image code from the search and rescue image codes, select a target image text code from the image text samples, and calculate a loss value based on the target search and rescue image code, the target image text code, and a preset average loss function; Based on the loss value, the parameters of the model to be trained are updated to obtain the video text encoding model.

[0009] In one embodiment, the step of calculating the loss value based on the target search and rescue image encoding and the target image text encoding includes: The target image text encoding is transposed, and the transposed target image text encoding is logarithmically divided to obtain the logarithmic image text encoding. The first loss value is obtained by multiplying the target search and rescue image code and the logarithmic image text code. The target search and rescue image code is transposed, and the logarithm of the transposed target search and rescue image code is taken to obtain the logarithmic search and rescue image code. The second loss value is obtained by multiplying the target search and rescue image code and the logarithmic search and rescue image code. The average of the first loss value and the second loss value is calculated to obtain the loss value.

[0010] In one embodiment, the stage task prompts include a stage task description and a task object description. The step of inputting the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model to obtain the action sequence decision results of the search and rescue robot includes: The video description text, the allowed action set, and the search and rescue mission prompts are input into a preset natural language guidance model. Based on the stage task description and the task object description, the execution stages of the search and rescue mission and the task objects of each execution stage are determined through a preset thought chain. Based on the execution stage and the task object, determine the execution actions corresponding to each execution stage and the execution order of each execution action from the set of allowed actions; The execution actions are combined based on the execution order to obtain the action sequence decision result.

[0011] Furthermore, to achieve the above objectives, this application also proposes a language-guided collaborative decision-making device for search and rescue robots, which includes: The data acquisition module is used to acquire real-time video collected by the search and rescue robot, the set of permitted actions of the search and rescue robot, and search and rescue mission prompts. The text generation module is used to input the real-time video into a preset video text encoding model to obtain the video description text corresponding to the real-time video. The video text encoding model is trained on a preset model to be trained based on preset search and rescue image samples and image text samples. The decision generation module is used to input the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model to obtain the action sequence decision results of the search and rescue robot. The natural language guidance model is trained on a preset base model based on preset video text samples and action sequence samples corresponding to the video text samples.

[0012] Furthermore, to achieve the above objectives, this application also proposes a language-guided collaborative decision-making device for search and rescue robots. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the language-guided collaborative decision-making method for search and rescue robots as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the language-guided collaborative decision-making method for search and rescue robots as described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the language-guided collaborative decision-making method for search and rescue robots as described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: The search and rescue robot acquires real-time video collected by the robot, its permitted action set, and search and rescue task prompts. The real-time video is input into a preset video text encoding model to obtain video description text corresponding to the real-time video. The video text encoding model is trained on a preset training model based on preset search and rescue image samples and image text samples. The video description text, the permitted action set, and the search and rescue task prompts are then input into a preset natural language guidance model to obtain the robot's action sequence decision results. The natural language guidance model is trained on a preset base model based on preset video text samples and corresponding action sequence samples.

[0016] To address the problem that current language-guided collaborative decision-making methods for search and rescue robots can only assign them to pre-trained tasks, sometimes placing them in tasks unsuitable for the current rescue environment, resulting in poor disaster relief effectiveness, this application utilizes language for decision-making in search and rescue robots, thus improving their rescue efficiency. Specifically, since the video-to-text encoding model in this application is trained on a pre-set model based on pre-defined search and rescue image samples and image-to-text samples, it can convert the video acquired by the search and rescue robot into corresponding text. Furthermore, this application trains the base model using video text and corresponding action sequence samples, enabling the natural language guidance model to determine the action sequence required for the current task based on the video text, prompts, and allowed action sets. Therefore, this application does not assign tasks based on pre-trained tasks, but rather determines the action sequence to be executed based on the real-time images acquired by the robot using natural language, avoiding the assignment of search and rescue robots to tasks unsuitable for the current rescue environment and improving the effectiveness of disaster relief. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the language-guided collaborative decision-making method for search and rescue robots in this application. Figure 2 This is a schematic diagram of the overall process of the first embodiment of the language-guided collaborative decision-making method for search and rescue robots in this application; Figure 3 This is a schematic diagram of the model structure provided in Embodiment 1 of the language-guided collaborative decision-making method for search and rescue robots in this application; Figure 4 A schematic diagram showing the success rate comparison of the first embodiment of the language-guided collaborative decision-making method for search and rescue robots in this application; Figure 5 A comparative diagram of average time provided for Embodiment 1 of the language-guided collaborative decision-making method for search and rescue robots in this application; Figure 6This is a diagram showing the comparison of the number of calculations provided in Embodiment 1 of the language-guided collaborative decision-making method for search and rescue robots in this application; Figure 7 This is a schematic diagram showing the average success rate comparison of the first embodiment of the language-guided collaborative decision-making method for search and rescue robots in this application. Figure 8 This is a flowchart illustrating Embodiment 2 of the language-guided collaborative decision-making method for search and rescue robots in this application. Figure 9 This is a schematic diagram of the reward and punishment process provided in Embodiment 2 of the language-guided collaborative decision-making method for search and rescue robots in this application; Figure 10 This is a flowchart illustrating Embodiment 3 of the language-guided collaborative decision-making method for search and rescue robots in this application. Figure 11 This is a schematic diagram of the prompt words provided in Embodiment 3 of the Language-Guided Collaborative Decision-Making Method for Search and Rescue Robots in this application; Figure 12 This is a schematic diagram of the module structure of the language-guided collaborative decision-making device for search and rescue robots according to an embodiment of this application; Figure 13 This is a schematic diagram of the hardware operating environment involved in the language-guided collaborative decision-making method for search and rescue robots in the embodiments of this application; Figure 14 This is a schematic diagram illustrating the data acquisition consent process involved in the language-guided collaborative decision-making method for search and rescue robots in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a language-guided search and rescue robot collaborative decision-making device. The following description uses a language-guided search and rescue robot collaborative decision-making device as an example to illustrate this embodiment and the subsequent embodiments.

[0024] Since the beginning of the 21st century, natural disasters have occurred frequently around the world. Major natural disasters cause casualties and severe damage to social production and daily life. Traditional disaster relief work mainly relies on manpower, which is costly and carries the risk of death for rescuers. Therefore, unmanned and intelligent systems are important development directions for disaster relief. With the rapid development of technologies such as automatic control, artificial intelligence, 5G, and high-performance computing, more and more rescue robots for disaster relief are being developed.

[0025] Currently, most search and rescue robots operate as single agents, with designers focusing on specific structural features to better enable them to perform particular rescue tasks. To improve the effectiveness of disaster relief, efficient decision-making methods are needed to allow rescue robots with different strengths to collaborate.

[0026] Currently, collaborative decision-making methods for search and rescue robots can be divided into centralized task allocation methods and distributed task allocation methods. Centralized task allocation avoids the difficulties caused by communication between distributed systems, but it also suffers from problems such as excessive computational load on the central processing module and inability to effectively cope with dynamically changing search and rescue environments. Distributed task allocation methods do not require a central system to coordinate the robots; instead, task allocation is achieved through information and communication received by each robot. However, these systems suffer from problems such as excessively high communication quality requirements and difficulty in coping with highly dynamic environments.

[0027] Therefore, as can be seen from the above, most existing technologies can only cope with static or low-dynamic environments. They are unable to quickly and effectively coordinate sub-search and rescue robots to deal with new situations in the face of rapidly changing rescue scenes. Moreover, the information received and processed by current methods is mostly sensor data. The decision-making system needs to perform accurate and intensive numerical calculations on this data before it can output a decision, which results in excessive computational costs. Furthermore, the information received and processed cannot directly provide assistance to rescue personnel.

[0028] Furthermore, current decision-making methods for rescue robots can only assign search and rescue robots to tasks they have been trained for, which may result in them being assigned to tasks that are not necessary for the current rescue environment. This leads to poor rescue effectiveness when using current methods for post-disaster relief.

[0029] Based on this, embodiments of this application provide a language-guided collaborative decision-making method for search and rescue robots, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the language-guided collaborative decision-making method for search and rescue robots according to this application.

[0030] In this embodiment, the language-guided collaborative decision-making method for search and rescue robots includes steps S10 to S30: Step S10: Obtain the real-time video collected by the search and rescue robot, the set of permitted actions of the search and rescue robot, and the search and rescue task prompts; It should be noted that real-time video refers to the dynamic image data stream continuously collected and transmitted by the search and rescue robot through its onboard cameras or LiDAR and other external devices at any given moment. This image data stream can be RGB images, depth maps, or radar scan images. The permissible action set refers to the set of all legal actions that the search and rescue robot can perform in its current state, determined by its hardware structure, mobility capabilities, and current environmental constraints.

[0031] It should also be noted that the search and rescue robot in this embodiment includes sub-robots with certain mobility capabilities. In practical applications, the working robotic arms of these sub-robots can be customized or adjusted according to the needs of the site. Before using the decision-making system of this invention, the working sub-robots need to be adjusted to a level where they can successfully complete macro-movements. For example, the grasping robot can complete the action through the macro-movement command "grab + object", and the training and encapsulation of macro-movements are completed locally on the sub-robot.

[0032] In this embodiment, there is a good communication connection between the sub-robots. Before using the decision system, the decision system model and the signal receiving part of each sub-robot can be placed in the same wireless network with low latency, or a physical network cable connection can be used.

[0033] Step S20: Input the real-time video into a preset video text encoding model to obtain the video description text corresponding to the real-time video. The video text encoding model is trained on a preset model to be trained based on preset search and rescue image samples and image text samples. It should be noted that a video text encoding model refers to a trained multimodal machine learning model capable of mapping input video data into natural language text describing the video content. Search and rescue image samples refer to image data collected from actual or simulated search and rescue environments that exhibit typical disaster or rescue scene characteristics. Image text samples refer to naturally language descriptive text, manually annotated or automatically generated, that matches specific image content.

[0034] It is understood that the video text encoding model in this embodiment is trained using search and rescue image samples specifically designed for search and rescue scenarios and their corresponding image text samples. It has learned the mapping relationship between typical visual elements in disaster environments and natural language descriptions, enabling the model to generate more accurate and task-relevant video description text when processing real-time video transmitted back by search and rescue robots. Compared with general video description models, the video text encoding model in this embodiment can significantly improve the ability to identify and express key information in search and rescue scenarios.

[0035] Step S30: Input the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model to obtain the action sequence decision result of the search and rescue robot. The natural language guidance model is trained on a preset basic model based on preset video text samples and action sequence samples corresponding to the video text samples.

[0036] It should be noted that the Natural Language Guidance Model (NLP) refers to a trained intelligent decision-making model capable of receiving environmental descriptions, task instructions, and a set of optional actions in natural language form, and outputting a sequence of actions that conforms to the task objectives. Video text samples refer to the input data used to train the NLP, containing natural language descriptions of the search and rescue scene video content. Action sequence samples refer to manually annotated robot action execution sequences corresponding to the video text samples, representing the reasonable action flow to be taken under specific visual semantics and task conditions. The overall flow of this embodiment can be referred to... Figure 2 .

[0037] Since the natural language guidance model in this embodiment is trained on a large number of paired video text samples and corresponding action sequence samples, it can output an action sequence that is both feasible for robot actions and suitable for rescue tasks after receiving video description text generated from real scenes, the current set of allowed actions and task prompts, enabling search and rescue robots to make decisions in complex and dynamic disaster environments.

[0038] Furthermore, this embodiment enhances the decision-making system's understanding of the environment through language guidance, thereby significantly improving the system's decision-making capabilities in highly dynamic environments and more effectively organizing multiple search and rescue robots to complete rescue operations. Moreover, by collecting and processing dynamic information in the search and rescue environment in the form of images and text, it generates data that is more intuitive, concise, and valuable for search and rescue personnel.

[0039] In one feasible implementation, the specific implementation of inputting the real-time video into a preset video text encoding model to obtain the video description text corresponding to the real-time video can also be: The process involves acquiring search and rescue image samples, corresponding image text samples, and a model to be trained. The model is pre-trained based on a preset visual dataset. The search and rescue image samples and the image text samples are encoded to obtain search and rescue image codes and image text codes corresponding to the image text samples. A target search and rescue image code is selected from the search and rescue image codes, and a target image text code is selected from the image text samples. A loss value is calculated based on the target search and rescue image code, the target image text code, and a preset average loss function. The parameters of the model to be trained are updated based on the loss value to obtain the video text encoding model.

[0040] It should be noted that visual datasets refer to large-scale public multimodal datasets containing a large number of general image-text pairs; search and rescue image encoding refers to the vector representation obtained after feature extraction of search and rescue image samples by the model to be trained; and image-text encoding refers to the vector representation obtained after encoding the image-text samples corresponding to the search and rescue image samples by the model to be trained.

[0041] Target search and rescue image coding refers to a specific image coding selected from a batch of search and rescue image codings during the training process, used for similarity calculation with image text coding. Target image text coding refers to a specific image coding selected from a batch of image text codings, used for similarity calculation with search and rescue image coding.

[0042] It should also be noted that the model to be trained in this embodiment can be a CLIP (Contrastive Language–Image Pretraining) model. The CLIP model is an image understanding model that uses a dataset consisting of paired images and annotations as training data to learn the semantics contained in each element of the image. After sufficient training, it can accurately describe the things and events contained in the image. The CLIP model structure in this embodiment can be referred to... Figure 3 This embodiment, based on the CLIP model, employs a contrastive learning method to calculate the cosine similarity between the encoded image and text, and then fine-tunes the model. The formula for calculating the cosine similarity is:

[0043] in, Here, I represents cosine similarity, and T represents image encoding.

[0044] It is understandable that this embodiment, based on the pre-training of existing general visual datasets, further fine-tunes the model using image and text samples from the search and rescue field, and optimizes the model parameters by using the similarity loss between the target search and rescue image encoding and the target image text encoding. As a result, the final video text encoding model not only inherits the general multimodal representation capability, but also enhances the accuracy of understanding and describing the unique visual elements of the search and rescue scene.

[0045] In one feasible implementation, the specific implementation of calculating the loss value based on the target search and rescue image encoding and the target image text encoding can also be: The target image text encoding is transposed, and the logarithm of the transposed target image text encoding is taken to obtain a logarithmic image text encoding. The product of the target search and rescue image encoding and the logarithmic image text encoding is calculated to obtain a first loss value. The target search and rescue image encoding is transposed, and the logarithm of the transposed target search and rescue image encoding is taken to obtain a logarithmic search and rescue image encoding. The product of the target search and rescue image encoding and the logarithmic search and rescue image encoding is calculated to obtain a second loss value. The average of the first loss value and the second loss value is calculated to obtain the loss value.

[0046] It should be noted that logarithmic image text encoding refers to the vector representation obtained by performing a logarithmic operation on the transposed target image text encoding, and logarithmic search and rescue image encoding refers to the vector representation obtained by performing a logarithmic operation on the transposed target search and rescue image encoding.

[0047] It should also be noted that the formula for calculating the first loss value in this embodiment is as follows:

[0048] in, The first loss value, It is the transpose of the text encoding.

[0049] The formula for calculating the second loss value in this embodiment is as follows:

[0050] in, It is the second loss value. It is the transpose of the image encoding.

[0051] The average loss function is:

[0052] in, It is the loss value calculated using the average loss function.

[0053] Understandably, this embodiment does not directly employ traditional similarity measures such as cosine similarity. Instead, it constructs a bidirectional loss function by transposing the target image encoding and text encoding respectively, taking the logarithm of each element, and then multiplying them with the encoding of another modality. The logarithmic operation enhances the response sensitivity of key semantic dimensions, while the method of transposing the target image encoding and text encoding respectively, taking the logarithm of each element, and then multiplying them with the encoding of another modality to calculate similarity ensures that the model learns the numerical distribution characteristics of the image encoding itself while learning the image-text matching relationship, thus avoiding excessively large dynamic range or sparsity issues during training.

[0054] In one embodiment, the language-guided collaborative decision-making method for search and rescue robots of this embodiment was used to compare its effectiveness with current methods in several different types of search and rescue tasks based on multiple concatenation indicators such as search and rescue mission, computational cost, and mission scope.

[0055] Specifically, in the success rate and average time test of search and rescue missions, this embodiment built multiple mission environments and designed four types of tasks by setting different targets and terrains: navigation tasks, obstacle clearing tasks, search tasks, and rescue tasks. Specifically: navigation tasks require the decision-making system to control a designated sub-robot to reach a designated location; obstacle clearing tasks require the decision-making system to control a suitable sub-robot to reach the obstacle location and use appropriate tools to clear the obstacle; search tasks require the decision-making system to control a sub-robot to search for a specific target on a map; and rescue tasks require the decision-making system to control a sub-robot to search for and rescue simulated wounded personnel to a designated location. Each type of task contains 20 sub-tasks, with 50 independent tests conducted on each task. The task success rate is calculated based on the average success rate across all types of tasks, and the task time is also calculated based on the average time across all types of tasks.

[0056] For specific comparison results, please refer to Figure 4 and Figure 5 ,from Figure 4 and 5 As can be seen, compared with the most advanced collective decision-making methods and distributed decision-making methods, the decision-making method in this embodiment can not only effectively improve the success rate of the task, but also significantly reduce the task time and improve the efficiency of the search and rescue task.

[0057] In the cost consumption test, this embodiment uses floating-point operations (FLOPs) as the metric to compare the average number of floating-point operations of the decision-making system designed in this invention with those of aggregate and distributed systems when completing a task cycle. Specific comparison results can be found in [reference needed]. Figure 6 This embodiment not only completes the search and rescue mission more efficiently, but also has a lower number of floating-point calculations and consumes less computing resources.

[0058] In the applicable scope testing, this embodiment adds some new tasks not used for training to each category of tasks to test the applicability and generalization performance of each decision system to new tasks or scenarios. The testing method is to directly apply each decision system after training to the new tasks and calculate their average success rate after solving the new tasks independently multiple times.

[0059] For detailed comparison results, please refer to Figure 7 This embodiment maintains a high success rate in all categories of new tasks, compared to trained tasks ( Figure 5 The performance of the decision-making system only decreases slightly, while existing aggregate and distributed systems cannot maintain good performance on new tasks that have not been trained. Therefore, the method in this embodiment has better adaptability and can complete more diverse search and rescue tasks.

[0060] In summary, current language-guided collaborative decision-making methods for search and rescue robots can only assign them to pre-trained tasks, potentially placing them in tasks unsuitable for the current rescue environment, leading to poor disaster relief results. This embodiment addresses this issue by utilizing language for search and rescue robot decision-making, thus improving rescue effectiveness. Specifically, the video-to-text encoding model in this embodiment is trained on a pre-set model based on preset search and rescue image samples and image-to-text samples, converting the video acquired by the search and rescue robot into corresponding text. Furthermore, this embodiment trains the basic model using video text and corresponding action sequence samples, enabling the natural language guidance model to determine the action sequence required for the current task based on the video text, prompts, and allowed action sets. Therefore, this embodiment does not assign tasks based on pre-trained tasks but instead determines the action sequence to be executed based on the real-time images acquired by the robot using natural language, preventing the search and rescue robot from being assigned to tasks unsuitable for the current rescue environment and improving the effectiveness of disaster relief.

[0061] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 8 Before step S10, the language-guided collaborative decision-making method for search and rescue robots further includes steps S01 to S03: Step S01: Obtain the base model, the video text sample, the action sequence sample, and the allowed action set, wherein the base model is a pre-trained large language model; It should be noted that, in this embodiment, the base model refers to a large language model pre-trained on a large-scale general dataset. In this embodiment, the base model can be invoked using an online network call interface API (Application Programming Interface), or it can be an open-source large model pre-downloaded locally. The base large language model can be obtained through both online and offline methods.

[0062] Step S02: Input the video text sample into the base model, and based on the base model, determine multiple target actions from the allowed action set and form a target action sequence; Understandably, the base model is pre-trained on large-scale language data, possessing strong understanding and generalization capabilities for task prompts, environmental descriptions, and other contextual information. Therefore, even in unfamiliar search and rescue scenarios, it can infer reasonable action combinations based on linguistic logic. Furthermore, this embodiment effectively avoids the problem of the model outputting illegal or infeasible instructions by limiting the source of actions to a set of allowed actions, thus improving the safety and practicality of action decision-making. Therefore, this embodiment utilizes the base model, which already possesses powerful semantic reasoning capabilities, to directly select actions and construct target action sequences within a restricted set of allowed actions. This ensures that the generated actions not only conform to the semantics of the scene described in natural language but also remain within the physically executable range of the robot, thereby improving the accuracy of action decision-making.

[0063] Step S03: Based on the target action sequence and the action sequence samples, the base model is trained using a preset reinforcement learning training paradigm to obtain the natural language guidance model.

[0064] In one feasible implementation, the specific implementation of training the base model based on the target action sequence and the action sequence samples using a preset reinforcement learning training paradigm to obtain the natural language guidance model can also be: Calculate the sequence similarity between the target action sequence and the action sequence sample. If the sequence similarity is higher than a preset similarity threshold, determine the reward value of the target action sequence based on the sequence similarity. If the sequence similarity is lower than the similarity threshold, determine the penalty value of the target action sequence based on the sequence similarity. Adjust the parameters of the base model based on the reward value or the penalty value to obtain the natural language guidance model.

[0065] It should be noted that sequence similarity refers to a quantitative indicator used to measure the consistency of action content and sequence order between a target action sequence and an action sequence sample. The similarity threshold refers to a preset numerical boundary used to determine whether the degree of matching between the target action sequence and the action sequence sample reaches an acceptable level. If it is higher than the threshold, it is considered reasonable behavior and rewarded; if it is lower than the threshold, it is considered a large deviation and punished.

[0066] It should also be noted that in this embodiment, a set of standard and correct action sequences is first provided manually, and these correct sequences are assigned higher reward values. If, during fine-tuning training of the large model, the generated action sequences are closer to the manually designed standard sequences, the reward value is larger and closer to +1. Conversely, if the generated action sequences differ significantly from the manually designed standard sequences, a penalty is imposed, with the penalty being closer to -1 for larger differences. The process of comparing the action sequences with manually labeled action sequence samples to determine the reward and penalty in this embodiment can be referred to [reference needed]. Figure 9 .

[0067] Understandably, this embodiment calculates the sequence similarity between the target action sequence and the action sequence samples, and dynamically allocates reward or penalty values ​​accordingly, thus providing a clear and quantifiable optimization direction for model training. When determining reward and penalty values, this embodiment uses a similarity threshold to obtain gradient feedback during partial matching, avoiding the learning instability caused by a judgment method requiring complete matching. Furthermore, since the reward and penalty in this embodiment are directly related to the overall sequence structure, rather than focusing solely on individual actions, the model tends to learn the temporal logic and task coherence between actions. This allows the resulting natural language guidance model to generate action sequences with reasonable action flows when faced with complex search and rescue instructions, thereby improving the accuracy and efficiency of search and rescue mission execution.

[0068] In summary, this embodiment selects robot actions by allowing action sets to be performed, and designs a threshold-based reward / penalty mechanism based on the sequence similarity between action sequence samples and target action sequences. This ensures that the trained natural language guidance model not only possesses strong language understanding capabilities but also outputs action sequences that conform to the robot's motion capability limitations. Furthermore, since the entire training process is driven by a reinforcement learning paradigm, the model can learn the temporal logic and task consistency between actions from sequence-level feedback, avoiding the local optima problem caused by relying solely on action-by-action supervision. This enables the model to generate action sequences with reasonable action flows, thereby improving the accuracy and efficiency of search and rescue mission execution.

[0069] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 10Step S30, the language-guided collaborative decision-making method for search and rescue robots further includes steps S31-S33: Step S31: Input the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model. Based on the stage task description and the task object description, determine the execution stage of the search and rescue task and the task object of each execution stage through a preset thought chain. It should be noted that the phase task description refers to the natural language description of each execution phase of the search and rescue mission, divided in time or logically, used to represent the sub-task objectives that should be completed at the current stage. The task object description refers to the natural language description of the specific entities or areas that need to be focused on or manipulated in a particular execution phase.

[0070] It should also be noted that the search and rescue mission prompts in this embodiment further include reasoning examples, heuristic statements, and fillable video descriptions. The fillable video descriptions are used to supplement the real-time video. Since the reasoning process of the large model is based on the principle of context filling, adding reasoning examples and heuristic statements to the prompts in this embodiment is more conducive to the large model's reasonable reasoning. This can be referred to... Figure 11 .

[0071] Understandably, this embodiment utilizes a pre-defined thought chain mechanism, combining stage task descriptions and task object descriptions to hierarchically parse the input multimodal semantic information. This enables the natural language guidance model to explicitly distinguish the temporal and logical stages of the task and their corresponding operational focuses. This embodiment, through this structured understanding approach of thought chains, provides clear contextual anchors for the generation of subsequent action sequences, thereby improving the interpretability of the decision-making process, the coherence of the stages, and the accuracy of action sequence generation.

[0072] Step S32: Based on the execution stage and the task object, determine the execution actions corresponding to each execution stage and the execution order of each execution action from the set of allowed actions; It is understood that, based on a clearly defined execution stage and task object, this embodiment dynamically selects and sorts actions from a limited set of allowed actions to ensure that the generated action sequence not only conforms to the semantic requirements of the current subtask but also strictly satisfies the robot's physical motion constraints. Since the selection and sorting of actions are related to the current task execution stage, and the task execution stage is obtained through stage prompts, this embodiment improves the temporal rationality of the action sequence through the above steps, thereby enhancing the rescue effectiveness.

[0073] Step S33: Combine the execution actions based on the execution order to obtain the action sequence decision result.

[0074] It is understood that the actions and sequence of combination in this embodiment are determined from the set of allowed actions based on phased prompts. This can avoid logical conflicts or redundant operations between actions during combination, so that the generated decision results can not only efficiently advance the search and rescue mission, but also adapt to the actual movement capabilities of the robot and improve the rescue effect.

[0075] In one feasible implementation, the specific implementation method after determining the execution stages of the search and rescue mission and the task objects of each execution stage through a preset thought chain can also be: Based on the execution phase and the task object, spatial semantic clues related to the task object are extracted from the video description text, and a relative orientation description of the task object relative to the search and rescue robot is generated based on the spatial semantic clues.

[0076] It should be noted that spatial semantic cues refer to the linguistic information implicitly or explicitly expressed in the video description text, which is used to indicate the positional relationship of the task object in three-dimensional space. Relative orientation description refers to the structured expression of the direction and rough distance of the task object in natural language with the search and rescue robot as the coordinate origin, which is used to guide the direction selection of subsequent actions.

[0077] It is understood that, based on the already determined execution stage and task object, this embodiment further performs fine-grained semantic analysis on the video description text to identify and extract spatial semantic clues related to the task object. Based on the spatial semantic clues, a relative orientation description with the search and rescue robot as the reference frame is generated, thereby providing concrete spatial guidance information for subsequent action selection.

[0078] This embodiment addresses the characteristics of search and rescue scenarios where the target object is often partially obscured, its location is ambiguous, and its position is inferred from linguistic clues. It actively extracts spatial semantic clues from video description text and transforms them into structured relative orientation descriptions, thereby providing reliable spatial guidance for action planning even in disaster environments lacking precise sensor positioning.

[0079] Specifically, general-purpose robot systems typically rely on geometric maps or point clouds for localization, while this embodiment uses a language model to understand the unstructured description of space and transform it into action guidance information, thus achieving navigation under conditions of limited perception.

[0080] Furthermore, when selecting robot actions, this embodiment can dynamically filter out actions that do not match the orientation through spatial semantics, thereby significantly improving the directional accuracy and search efficiency of the action sequence.

[0081] In one feasible implementation, the specific implementation of extracting spatial semantic cues related to the task object from the video description text based on the execution stage and the task object, and generating a relative orientation description of the task object relative to the search and rescue robot based on the spatial semantic cues, may also be: If the video description text does not explicitly contain spatial semantic clues, a default relative orientation description that conforms to the task logic of the current stage is generated based on the typical behavior patterns of the execution stage and the directional actions in the allowed action set.

[0082] It should be noted that directional actions refer to executable actions that explicitly include directional information within the action set. Typical behavior patterns refer to the standard operating paradigms that are typically required at a specific execution stage. Default relative orientation description: refers to a reasonable hypothetical orientation description inferred by the system based on the typical behavior patterns and available directional actions at the current execution stage, used to guide action selection when explicit spatial semantic cues are lacking.

[0083] Understandably, this embodiment first determines whether the video description text contains spatial semantic clues that can be used to locate the task object. If no explicit clues are detected, it combines the typical behavior patterns corresponding to the current execution stage and analyzes the directional actions available in the allowed action set. Based on this, a default relative location description that conforms to the task logic and is feasible is generated. Thus, even under extreme search and rescue conditions where there is insufficient perception information or ambiguous language description, the continuity of the decision-making process and the rationality of action generation can still be maintained.

[0084] In summary, the actions performed in this embodiment are always limited to the set of allowed actions, and their order is dynamically determined by the task stage and the object, thus avoiding infeasible or out-of-sequence actions that the model might generate. Furthermore, since the identification of stages and objects in this embodiment is based on environmental semantics and task objectives, the system can adaptively adjust its behavior strategy to cope with dynamically changing search and rescue scenarios, thereby expanding the range of tasks that the search and rescue robot can perform while ensuring the physical feasibility of the actions.

[0085] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the language-guided collaborative decision-making method for search and rescue robots in this application. Any simple modifications based on this technical concept are within the scope of protection of this application.

[0086] This application also provides a language-guided collaborative decision-making device for search and rescue robots; please refer to [reference needed]. Figure 12 The language-guided collaborative decision-making device for search and rescue robots includes: The data acquisition module 10 is used to acquire real-time video collected by the search and rescue robot, the set of permitted actions of the search and rescue robot, and search and rescue task prompts. The text generation module 20 is used to input the real-time video into a preset video text encoding model to obtain the video description text corresponding to the real-time video. The video text encoding model is trained on a preset model to be trained based on preset search and rescue image samples and image text samples. The decision generation module 30 is used to input the video description text, the allowed action set, and the search and rescue task prompt words into a preset natural language guidance model to obtain the action sequence decision result of the search and rescue robot. The natural language guidance model is trained on a preset basic model based on preset video text samples and action sequence samples corresponding to the video text samples.

[0087] In one embodiment, the language-guided search and rescue robot collaborative decision-making device includes: The guidance sample acquisition module is used to acquire the base model, the video text samples, the action sequence samples, and the allowed action set, wherein the base model is a pre-trained large language model; An action sequence generation module is used to input the video text samples into the base model, and based on the base model, determine multiple target actions from the allowed action set and form a target action sequence. The guidance model training module is used to train the base model based on the target action sequence and the action sequence samples through a preset reinforcement learning training paradigm to obtain the natural language guidance model.

[0088] In one embodiment, the guided model training module includes: A sequence similarity calculation submodule is used to calculate the sequence similarity between the target action sequence and the action sequence sample; The reward value determination submodule is used to determine the reward value of the target action sequence based on the sequence similarity if the sequence similarity is higher than a preset similarity threshold. The punisher determination submodule is used to determine the punishment value of the target action sequence based on the sequence similarity if the sequence similarity is lower than the similarity threshold; The guidance module training submodule is used to adjust the parameters of the base model based on the reward value or the penalty value to obtain the natural language guidance model.

[0089] In one embodiment, the language-guided search and rescue robot collaborative decision-making device includes: The encoding sample acquisition module is used to acquire search and rescue image samples, image text samples corresponding to the search and rescue image samples, and the model to be trained, wherein the model to be trained is obtained by pre-training based on a preset visual dataset; The encoding module is used to encode the search and rescue image samples and the image text samples to obtain the search and rescue image code corresponding to the search and rescue image sample and the image text code corresponding to the image text sample. The loss calculation module is used to select a target search and rescue image code from the search and rescue image codes, select a target image text code from the image text samples, and calculate a loss value based on the target search and rescue image code, the target image text code, and a preset average loss function; The encoding model training module is used to update the parameters of the model to be trained based on the loss value, so as to obtain the video text encoding model.

[0090] In one embodiment, the loss calculation module includes: The image text encoding processing submodule is used to transpose the target image text encoding and take the logarithm of the transposed target image text encoding to obtain the logarithmic image text encoding. The first loss calculation submodule is used to calculate the product of the target search and rescue image code and the logarithmic image text code to obtain the first loss value; The search and rescue image encoding processing submodule is used to transpose the target search and rescue image encoding and take the logarithm of the transposed target search and rescue image encoding to obtain the logarithmic search and rescue image encoding. The second loss calculation submodule calculates the product of the target search and rescue image code and the logarithmic search and rescue image code to obtain the second loss value; The loss value calculation submodule is used to calculate the average of the first loss value and the second loss value to obtain the loss value.

[0091] In one embodiment, the decision generation module includes: The phase determination submodule is used to input the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model. Based on the phase task description and the task object description, the execution phase of the search and rescue task and the task objects of each execution phase are determined through a preset thought chain. The sequence determination submodule is used to determine the execution actions corresponding to each execution stage and the execution order of each execution action from the set of allowed actions, based on the execution stage and the task object. The combination submodule is used to combine the execution actions based on the execution order to obtain the action sequence decision result.

[0092] The language-guided collaborative decision-making device for search and rescue robots provided in this application employs the language-guided collaborative decision-making method for search and rescue robots described in the above embodiments, which can solve the technical problem of poor rescue effectiveness in current methods for post-disaster rescue. Compared with the prior art, the beneficial effects of the language-guided collaborative decision-making device for search and rescue robots provided in this application are the same as those of the language-guided collaborative decision-making method for search and rescue robots provided in the above embodiments, and other technical features in the language-guided collaborative decision-making device for search and rescue robots are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0093] This application provides a language-guided collaborative decision-making device for search and rescue robots. The language-guided collaborative decision-making device for search and rescue robots includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the language-guided collaborative decision-making method for search and rescue robots in the above embodiment 1.

[0094] The following is for reference. Figure 13 The diagram illustrates a structural schematic of a language-guided collaborative decision-making device for search and rescue robots suitable for implementing embodiments of this application. The language-guided collaborative decision-making device for search and rescue robots in this application can include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 13 The language-guided collaborative decision-making device for search and rescue robots shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0095] like Figure 13As shown, the language-guided search and rescue robot collaborative decision-making device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1002 or programs loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the language-guided search and rescue robot collaborative decision-making device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the language-guided search and rescue robot collaborative decision-making device to communicate wirelessly or wiredly with other devices to exchange data. Although a language-guided search and rescue robot collaborative decision-making device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0096] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0097] The language-guided collaborative decision-making device for search and rescue robots provided in this application employs the language-guided collaborative decision-making method for search and rescue robots described in the above embodiments, which can solve the technical problem of poor rescue effectiveness in current methods for post-disaster rescue. Compared with the prior art, the beneficial effects of the language-guided collaborative decision-making device for search and rescue robots provided in this application are the same as those of the language-guided collaborative decision-making method for search and rescue robots provided in the above embodiments, and other technical features of this language-guided collaborative decision-making device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0098] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0100] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the language-guided collaborative decision-making method for search and rescue robots in the above embodiments.

[0101] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0102] The aforementioned computer-readable storage medium may be included in the language-guided collaborative decision-making device for search and rescue robots; or it may exist independently and not be assembled into the language-guided collaborative decision-making device for search and rescue robots.

[0103] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a language-guided search and rescue robot collaborative decision-making device, cause the language-guided search and rescue robot collaborative decision-making device to execute the aforementioned language-guided search and rescue robot collaborative decision-making method.

[0104] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0107] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described language-guided collaborative decision-making method for search and rescue robots. This addresses the technical problem of poor rescue effectiveness in current disaster relief methods. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the language-guided collaborative decision-making method for search and rescue robots provided in the above embodiments, and will not be elaborated upon here.

[0108] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the language-guided collaborative decision-making method for search and rescue robots as described above.

[0109] The computer program product provided in this application can solve the technical problem of poor rescue effectiveness in current disaster relief methods. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the language-guided collaborative decision-making method for search and rescue robots provided in the above embodiments, and will not be repeated here.

[0110] All user-related data involved in this application was obtained with the user's permission or consent, as per [reference]. Figure 14 In other words, when this application is applied to a specific product or technology, user permission is required to acquire and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.

[0111] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A language-guided collaborative decision-making method for search and rescue robots, characterized in that, The method includes: Acquire real-time video collected by the search and rescue robot, the set of permitted actions of the search and rescue robot, and search and rescue mission prompts; The real-time video is input into a preset video text encoding model to obtain the video description text corresponding to the real-time video. The video text encoding model is trained on a preset model to be trained based on preset search and rescue image samples and image text samples. The video description text, the allowed action set, and the search and rescue task prompts are input into a preset natural language guidance model to obtain the action sequence decision results of the search and rescue robot. The natural language guidance model is trained on a preset base model based on preset video text samples and action sequence samples corresponding to the video text samples.

2. The method as described in claim 1, characterized in that, Before the step of inputting the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model to obtain the action sequence decision result of the search and rescue robot, the method further includes: Obtain the base model, the video text samples, the action sequence samples, and the allowed action set, wherein the base model is a pre-trained large language model; The video text sample is input into the base model. Based on the base model, multiple target actions are determined from the set of allowed actions and a sequence of target actions is formed. Based on the target action sequence and the action sequence samples, the base model is trained using a preset reinforcement learning training paradigm to obtain the natural language guidance model.

3. The method as described in claim 2, characterized in that, The step of training the base model based on the target action sequence and the action sequence samples using a preset reinforcement learning training paradigm to obtain the natural language guidance model includes: Calculate the sequence similarity between the target action sequence and the action sequence sample; If the sequence similarity is higher than a preset similarity threshold, then the reward value of the target action sequence is determined based on the sequence similarity. If the sequence similarity is lower than the similarity threshold, then the penalty value of the target action sequence is determined based on the sequence similarity. Based on the reward value or the penalty value, the parameters of the base model are adjusted to obtain the natural language guidance model.

4. The method as described in claim 1, characterized in that, Before the step of inputting the real-time video into a preset video text encoding model to obtain the video description text corresponding to the real-time video, the method further includes: Acquire search and rescue image samples, image text samples corresponding to the search and rescue image samples, and the model to be trained, wherein the model to be trained is obtained by pre-training based on a preset visual dataset; The search and rescue image samples and the image text samples are encoded to obtain the search and rescue image codes corresponding to the search and rescue image samples and the image text codes corresponding to the image text samples; Select a target search and rescue image code from the search and rescue image codes, select a target image text code from the image text samples, and calculate a loss value based on the target search and rescue image code, the target image text code, and a preset average loss function; Based on the loss value, the parameters of the model to be trained are updated to obtain the video text encoding model.

5. The method as described in claim 4, characterized in that, The step of calculating the loss value based on the target search and rescue image encoding and the target image text encoding includes: The target image text encoding is transposed, and the transposed target image text encoding is logarithmically divided to obtain the logarithmic image text encoding. The first loss value is obtained by multiplying the target search and rescue image code and the logarithmic image text code. The target search and rescue image code is transposed, and the logarithm of the transposed target search and rescue image code is taken to obtain the logarithmic search and rescue image code. The second loss value is obtained by multiplying the target search and rescue image code and the logarithmic search and rescue image code. The average of the first loss value and the second loss value is calculated to obtain the loss value.

6. The method as described in claim 1, characterized in that, The search and rescue mission prompts include stage mission prompts, which in turn include stage mission descriptions and mission object descriptions. The step of inputting the video description text, the allowed action set, and the search and rescue mission prompts into a preset natural language guidance model to obtain the action sequence decision results of the search and rescue robot includes: The video description text, the allowed action set, and the search and rescue mission prompts are input into a preset natural language guidance model. Based on the stage task description and the task object description, the execution stages of the search and rescue mission and the task objects of each execution stage are determined through a preset thought chain. Based on the execution stage and the task object, determine the execution actions corresponding to each execution stage and the execution order of each execution action from the set of allowed actions; The execution actions are combined based on the execution order to obtain the action sequence decision result.

7. A language-guided collaborative decision-making device for search and rescue robots, characterized in that, The device includes: The data acquisition module is used to acquire real-time video collected by the search and rescue robot, the set of permitted actions of the search and rescue robot, and search and rescue mission prompts. The text generation module is used to input the real-time video into a preset video text encoding model to obtain the video description text corresponding to the real-time video. The video text encoding model is trained on a preset model to be trained based on preset search and rescue image samples and image text samples. The decision generation module is used to input the video description text, the allowed action set, and the search and rescue task prompts into a preset natural language guidance model to obtain the action sequence decision results of the search and rescue robot. The natural language guidance model is trained on a preset base model based on preset video text samples and action sequence samples corresponding to the video text samples.

8. A language-guided collaborative decision-making device for search and rescue robots, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the language-guided collaborative decision-making method for search and rescue robots as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the language-guided collaborative decision-making method for search and rescue robots as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the language-guided collaborative decision-making method for search and rescue robots as described in any one of claims 1 to 6.

Images