Search direction autonomous decision-making method fusing prior probability and large language model

Abstract

The invention discloses a search direction autonomous decision-making method fusing prior probability and a large language model, which belongs to the field of autonomous decision-making, and comprises the following steps of: firstly, aiming at a scene traversal problem caused by an unknown target position, performing scene search sequence decision-making through logical reasoning based on environment prior knowledge; generating an optimal search sequence as global guidance; and then, scene matching is realized through prior probability analysis and big language model reasoning based on visual observation, a region of interest is obtained based on semantic correlation evaluation of a big language model, and autonomous decision-making in the search direction is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of autonomous decision-making, and particularly relates to a search direction autonomous decision-making method fusing prior probability and a large language model. BACKGROUND

[0002] In search tasks in unknown indoor environments, unmanned aerial vehicles (UAVs) face the challenge of how to efficiently search and understand the environment to quickly locate the target.

[0003] Due to the unknown environment information, the UAV cannot obtain the specific location of the target in advance, and therefore must establish cognition of the environment through step-by-step search and continuously adjust the strategy during the search process until the target is found. However, the lack of prior information about the target location makes it difficult for the UAV to quickly determine the search direction, often requiring extensive and purposeless search, which not only reduces the search efficiency but also significantly increases the time and energy cost of the task. SUMMARY

[0004] To solve the above technical problems, the present application provides a search direction autonomous decision-making method fusing prior probability and a large language model, comprising:

[0005] generating a global search order using a large language model according to a list of search areas and a target name to be searched;

[0006] generating a list of object names in different directions according to the global search order and the visual observation results of the UAV;

[0007] calculating the matching degree scores of the search areas in different directions according to the list of object names in different directions and the list of search areas;

[0008] determining the final search direction according to the matching degree scores of the search areas in different directions and the global search order.

[0009] Preferably, the process of generating a global search order comprises:

[0010] designing a role description and an example description for the large language model according to the list of search areas and the target name to be searched;

[0011] inputting the list of search areas and the target name to be searched into the large language model, and generating a global search order according to the role description and the example description.

[0012] Preferably, the role description comprises a task scenario description and a capability boundary limitation, and the example description comprises an input specification and an output format.

[0013] Preferably, the process of generating a list of object names in different directions comprises:

[0014] According to the visual observation results of the unmanned aerial vehicle in different directions, objects in the scene are identified by using a visual language model, and a list of object names is generated;

[0015] According to the global search sequence, the lists of object names in different directions are sorted.

[0016] Preferably, the process of calculating the matching degree score of the search area in different directions comprises:

[0017] According to the list of object names in different directions, a region category-item name prior probability list constructed in advance is combined to calculate the matching degree score of the search area in different directions.

[0018] According to the matching degree score, the search areas in different directions are sorted.

[0019] Preferably, the region category-item name prior probability list is constructed from a public data set containing multiple images and their recognition results by a statistical method.

[0020] Preferably, the method further comprises:

[0021] According to the final search direction, the sensing range of the sensor is combined to generate a sub-target point position.

[0022] According to the sub-target point position, the unmanned aerial vehicle is guided to move gradually to the target direction through local planning of a search path.

[0023] Preferably, the final search direction is obtained based on weighted fusion of the prior probability score and the large language model score.

[0024] In another aspect, the present application also provides an electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0025] In another aspect, the present application also provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the method.

[0026] Compared with the prior art, the present application has the following advantages and technical effects:

[0027] The present invention provides a method for autonomously deciding search directions by integrating prior probabilities and a large language model. This method can determine the search direction in unknown environments based on the search area category, the name of the target to be searched, and the visual observation results of a drone. Large language model technology can also be used to determine the global search order, avoiding inefficient traversal searches. Furthermore, by integrating prior probabilities, autonomous search direction decisions can be made in unknown indoor environments. By determining the global search order and autonomously deciding the search direction, this application improves the efficiency of autonomous searches in unknown indoor environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0029] Figure 1 This is a schematic diagram of a semantic map according to an embodiment of the present invention;

[0030] Figure 2 A conversion diagram from a pixel coordinate system to a world coordinate system according to an embodiment of the present invention;

[0031] Figure 3 Flowchart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Example 1

[0035] This embodiment provides a method for autonomously deciding a search direction by integrating prior probabilities and a large language model, including:

[0036] Based on the list of areas to be searched and the names of the targets to be searched, a global search order is generated using a large language model;

[0037] Generate a list of object names in different directions based on the global search order and drone visual observation results;

[0038] According to the object name list and the list of areas to be searched in different directions, the matching scores of the areas to be searched in different directions are calculated;

[0039] According to the matching degree score of the to-be-searched region in different directions and the global search order, the final search direction is determined.

[0040] Specifically:

[0041] Based on the to-be-searched region list and the to-be-searched target name, a global search order is generated based on large language model technology.

[0042] Based on the visual observation results of the unmanned aerial vehicle in different directions, an object name list in different directions is generated through visual language model technology, and a semantic map is established by combining coordinate system conversion.

[0043] Based on a public dataset containing 3000 pictures and their recognition results, a "region category-item name" prior probability list is constructed, and the prior probability of the object appearing in a specific region is calculated through statistical methods.

[0044] Based on the object name list in different directions and the to-be-searched region list, higher information content items are screened out through entropy value calculation, the matching degree score of the to-be-searched region in different directions is calculated through posterior probability, and the matching degree ranking is obtained.

[0045] Based on the object name list in different directions and the to-be-searched region list, a large language model technology is used to generate a matching degree ranking of the to-be-searched region in different directions.

[0046] Based on the object name list in different directions and the to-be-searched target name, a large language model technology is used to generate a relevance score in different directions.

[0047] Based on the matching degree score and ranking of the to-be-searched region in different directions obtained from the prior probability, and the matching degree ranking and relevance score obtained from the large language model technology, the final matching degree ranking and relevance score of the to-be-searched region are obtained through weighted fusion.

[0048] Based on the global search order, the matching degree ranking and relevance score of the to-be-searched region in different directions, the final search direction is obtained.

[0049] Step 1: Determine the global search order;

[0050] To effectively guide the large language model to generate a search order that meets the actual situation, a prompt is designed according to the characteristics of indoor search tasks. The prompt design includes two key elements: role description and example description, and adopts a single interaction mode without the need to establish a continuous dialogue mechanism. The construction method of the prompt is as follows:

[0051] (1) Role description design;

[0052] The role description includes task scenario description and ability boundary limitation, which clearly limits the scope of responsibility and decision-making authority of the unmanned aerial vehicle system.

[0053] 1) Task_Scene description;

[0054] Task_Scene contains two elements: role definition and application scenario.

[0055] Task_Scene={R Des ,A Sce};

[0056] In the formula, R Des is the role definition, which clearly defines the function of large language model in the task; A Sce is the application scenario, which defines the specific experimental environment.

[0057] For the task requirement of generating global search order, the role definition is sorting assistant; the application scenario is oriented to indoor environment, and the spatial distribution possibility of search target is evaluated.

[0058] 2) Capability boundary definition;

[0059] Capability boundary can effectively guide the behavior of large language model, prevent potential risks, and provide institutional guarantee for reliable operation of the system. A complete capability boundary definition should include the following core contents:

[0060] Capability_Boundaries={O Per ,O Pro};

[0061] In the formula, O Per is the authorized operation, which clearly defines the operation behavior that large language model can call when solving the target task, to ensure that all operations meet the system safety specifications and task requirements. O Pro is the prohibited operation, which strictly limits the operation behavior that may cause security risks.

[0062] For the task requirement of generating global search order, the authorized operation is limited to using the pre-defined area label in the input list, and the output result must be limited within the input area label set and include all labels; the prohibited operation prohibits generating any area label outside the list, omitting the pre-defined area label or exceeding the hypothetical reasoning of environment description. This design not only fully utilizes the reasoning advantage of large language model, but also effectively avoids its potential "illusion" risk.

[0063] (2) Example description design;

[0064] Example description clearly defines the interaction paradigm between user and large language model by designing interaction form. A complete interaction form should include the following core contents:

[0065] Interaction_Protocol={ISpe , I For};

[0066] Where, I Spe For input specifications, clearly define the types of requirements contained in user instructions to ensure that the instructions are unambiguous. For The output format serves the task requirements.

[0067] To meet the requirements of generating a global search order, the input specification specifies the format and definition of the input elements, enabling the large language model to distinguish between the target name to be searched and the list of region labels to be searched in the input elements of this method. The output format serves automated parsing. To facilitate region label extraction and reduce the possibility of program errors, an easy-to-extract output format is provided to guide the output of the large model. The output format example is as follows:

[0068] Area label 1 - Area label 2 - ... - Area label n.

[0069] After generating a search order, the drone will search each area in order of priority. Within each area, the drone uses its visual perception module to identify the target object. If the target is not found in the current area, the drone will move to the next higher priority area according to the search order and continue searching.

[0070] Step 2: Detection results and semantic map construction based on visual language model technology;

[0071] (1) Recognition and detection based on visual language model technology;

[0072] Since the target objects and task scenarios involved in the task instructions are unknown, and traditional deep learning methods are limited by the number of predefined labels, they may not be able to cover all potential targets. Therefore, to achieve the above capabilities, two localized visual language models are deployed:

[0073] 1) Recognize-anything model;

[0074] This model can identify all objects in the input image and return a list of their corresponding labels. This example uses this model to construct a scene description, representing the scene through a list of object names. This is used for subsequent tasks such as region label matching and search direction decisions. Its input is an RGB image, and its output is a list of the names of all detected objects in the image.

[0075] 2) GroundingDINO model;

[0076] The model can locate the corresponding object in the input image according to the given prompt and add an identification anchor frame to it, thereby achieving accurate positioning of the object. This function provides important support for subsequent semantic map construction.

[0077] By combining the above two models, the system can obtain the name of the object in the scene and its anchor frame information. The specific implementation is to obtain the object label in the original image through the Recognize-anything model, and then use it as the prompt word of the GroundingDINO model to generate the identification anchor frame. The data in the depth image is used for coordinate conversion, so as to calculate the specific position of the object.

[0078] (2) Semantic map construction;

[0079] Semantic map construction is an important part of target search, aiming to combine the geometric information and semantic information in the environment to construct an environment map containing high-level semantic information. Unlike traditional geometric maps, semantic maps not only contain the spatial position of objects in the environment, but also contain the identification and classification information of objects in the environment, thereby providing robots with richer environmental understanding capabilities. Target recognition and detection provide identification and classification information of objects in the environment, identification anchor frames provide geometric information, and through pixel coordinate conversion to world coordinates, the spatial position of objects in the environment is obtained.

[0080] The schematic diagram of semantic map construction is shown in Figure 1 When the unmanned aerial vehicle moves in an unknown environment, it first needs to ensure flight safety, so it will observe the surrounding environment in real time. By detecting and identifying objects in the scene, the system establishes a vector representation for each object and matches it with a semantic label, thereby providing support for local path planning for search. The local path planning for search can avoid obstacles identified during operation and guide the unmanned aerial vehicle to move forward. In addition, the system will perform target observation at each flight path point and record the observation results for subsequent simplified information representation and secondary search. For example, at position p1, the recorded information is {p1:obj1, obj4}, indicating that objects obj1 and obj4 are observed at this position.

[0081] The target detection can obtain the pixel points s corresponding to the target, and the pixel points can obtain the position in the pixel coordinate system. In this embodiment, the position of the target in the world coordinate system is obtained through coordinate transformation, thereby obtaining the semantic map based on the world coordinate system. This process involves three times of conversion between pixel coordinate system, camera coordinate system, body coordinate system and world coordinate system. As Figure 2As shown, W, B and C represent the world coordinate system, the body coordinate system and the camera coordinate system respectively, and point q is a point in the world coordinate system with coordinates The projection point in the pixel coordinate system is T WB and T BC represent the conversion relationship between the world coordinate system and the body coordinate system, and the conversion relationship between the body coordinate system and the camera coordinate system.

[0082] The conversion relationship from the pixel coordinate to the world coordinate is as follows:

[0083]

[0084] In the formula, is the augmented matrix of s p WB is the position of the unmanned aerial vehicle in the world coordinate system.

[0085] Based on the above formula, the geometric information of the objects in the scene can be obtained, and the semantic information obtained by fusing the target detection and recognition algorithm is used to construct a semantic map for the next behavior decision.

[0086] Step three: autonomous decision of search direction based on prior probability;

[0087] (1) Construction of "region category-item name" prior probability list;

[0088] Based on a public dataset containing 3000 pictures and their recognition results, a "region category-item name" prior probability list is constructed. The list covers most common region names (such as kitchen, bedroom, living room, etc.) and item names (such as refrigerator, bed, sofa, etc.), and calculates the prior probability of the appearance of an item in a specific region, i.e. the probability p(a|b) of the appearance of item a in region b, through statistical methods.

[0089] (2) Information entropy screening method;

[0090] After obtaining the scene description in different directions, in order to reduce the influence of common items (such as doors and windows) on scene recognition, the entropy value of each object category is calculated, and the object with the largest amount of information is extracted through the measurement result of the entropy value, and the scene posterior probability is calculated based on this. The formula for calculating the entropy value is as follows:

[0091]

[0092] In the formula, o i ∈L o represents the target item category (such as refrigerator, bed, etc.), t j ∈L T represents the scene category (such as kitchen, bedroom, etc.). p(tj |o i ) represents the target object o i The conditional probability appearing in the scene t j is obtained by normalizing the current scene categories.

[0093] (3) Region matching degree ranking generation;

[0094] After screening out a number of key objects with the lowest information entropy (i.e., the strongest discriminative), the likelihood score of each region category, i.e., the posterior probability, is calculated through the conditional probability distribution thereof:

[0095]

[0096] In the formula, {o1, o2,..., o K} is the key object set obtained by information entropy screening. Since the number of objects detected by different perspectives may be different, the following formula is used to normalize the posterior probability P(t j ) to obtain the final likelihood score S(t j ), and the region matching degree probability ranking po1, po2, po3 under the three perspectives is obtained through the likelihood score.

[0097]

[0098] Step four: search direction autonomous decision based on large language model;

[0099] (1) Region label matching degree ranking;

[0100] 1) Role description design;

[0101] (a) Task scenario description;

[0102] The task scenario includes two elements of role definition and application scenario. For the task requirement of generating region label matching ranking, the role definition is a sorting assistant; the application scenario faces scenario matching, and the matching degree of the scene description and the region label is evaluated.

[0103] (b) Ability boundary limitation;

[0104] The ability boundary includes two elements of authorized operation and prohibited operation. For the task requirement of generating region label matching ranking, the authorized operation is limited to using the pre-defined region label in the input list, and the output result must be limited within the input region label set and include all labels; the prohibited operation prohibits generating any region label outside the list and omitting the pre-defined region label.

[0105] 2) Example description design;

[0106] The example description includes input specification and output format, which clearly defines the interaction form with the large language model. For the task requirement of generating region label matching ranking, the input specification increases the definition of input elements, enabling the large language model to distinguish between the scene description list and the region label list in the input elements of this method. The output format serves the downstream task automatic parsing and provides an output format convenient for extraction. The output format example is as follows:

[0107] Direction n: Region Label 1-Region Label 2-......-Region Label n.

[0108] (2) Correlation score;

[0109] 1) Role description design;

[0110] (a) Task scene description;

[0111] The task scene includes two elements: role definition and application scenario. For the task requirement of generating correlation measurement scores, the role definition is a scoring assistant; the application scenario measures the relevance of the searched target to the current scene. After the UAV arrives at the region, when the region label in each direction is consistent, the score is used to determine the next search direction of the UAV.

[0112] (b) Ability boundary limitation;

[0113] The ability boundary includes two elements: authorized operation and prohibited operation. For the task requirement of generating correlation measurement scores, authorized operation is limited to scoring; prohibited operation prohibits the use of any form other than scoring for measurement.

[0114] 2) Example description design;

[0115] The example description includes input specification and output format, which clearly defines the interaction form with the large language model. For the task requirement of generating correlation measurement scores, the input specification enables the large language model to distinguish between the target object name and the scene description of different directions in the input elements of this method; the output format serves the automatic parsing and is measured by a score between [0, 10]. The output format example is as follows:

[0116] Direction n: x, x ∈ [0, 10];

[0117] Step four: fusion decision method design;

[0118] The method based on prior probability can provide accurate numerical basis to lay the foundation for regional label determination. When encountering scenes beyond the range of the prior probability list, the method based on large language model can provide effective supplementary judgment. More importantly, this fusion method can provide relevance measurement for search tasks in the same scene, thereby providing a basis for determining the next moving direction. The regional label fusion determination score calculation formula is as follows:

[0119] s = a * area_score_p + b * area_score_l;

[0120] In the formula, s is the fusion possibility score, a and b are fixed values, area_score_p is the possibility score s(t j ) in step two, and area_score_l is the assignment score based on the matching degree ranking obtained in step three.

[0121] The pseudo code of the fusion strategy is shown in Table 1. This method aims to generate regional labels and their relevance scores for each direction. The input includes the ranking and its corresponding score generated based on prior probability and large language model, and the output is the fused ranking and score. Specifically, when the labels determined by prior probability and large language model are consistent, the current label is directly adopted. If they are not consistent, the fusion possibility score of each regional label is calculated for ranking, and the label with the highest score is selected as the final label for the current direction. Since the output result of the large language model does not include the matching degree score, this embodiment assigns scores according to the ranking result. The higher the ranking of the label, the higher the score. After determining the regional label for each direction, the relevance score output by the large language model is matched with the corresponding label through indexing, thereby obtaining the final fusion result. After obtaining the fusion result, it is matched with the regional search order to determine the next action direction. Based on the sensing range of the sensor, the corresponding sub-target point position is generated, and the search process is gradually pushed forward through local planning of the search path.

[0122] Table 1

[0123]

[0124] The method implementation process is as follows Figure 3As shown: first, the target name and prior map information are used to determine the search order by using a large language model, which is equivalent to pruning the search space to avoid unnecessary exploration and provide a basis for subsequent decision-making; then, during the search process, the unmanned aerial vehicle perception module observes different directions and obtains scene descriptions; then, the prior probability and large language model are combined to make autonomous decisions on the search direction, measure the matching degree of different directions, and obtain the corresponding scores, thereby obtaining the search direction, and combining the sensor perception range to determine the next sub-target position. Finally, through local planning of the search path, the unmanned aerial vehicle searches in the direction of the sub-target.

[0125] In another aspect, the embodiment further provides an electronic device, comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor executes the computing program to implement the method.

[0126] In another aspect, the embodiment further provides a computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the method.

[0127] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A search direction autonomous decision-making method integrating prior probability and large language model, characterized by: include: Based on the list of areas to be searched and the names of the targets to be searched, a global search order is generated using a large language model; Generate a list of object names in different directions based on the global search order and drone visual observation results; According to the object name list and the list of areas to be searched in different directions, the matching scores of the areas to be searched in different directions are calculated; The final search direction is determined based on the matching scores of the search areas in different directions and the global search order.

2. The method according to claim 1, characterized in that The process of generating a global search order includes: Based on the list of areas to be searched and the names of the targets to be searched, design role descriptions and example descriptions for the large language model; The list of areas to be searched and the names of targets to be searched are input into a large language model, and a global search order is generated according to the role description and the example description.

3. The method according to claim 2, characterized in that The role description includes a task scenario description and capability boundary definition, and the example description includes input specifications and output format.

4. The method according to claim 1, wherein The process of generating a list of object names in different directions includes: Based on the drone’s visual observations from different directions, the visual language model is used to identify objects in the scene and generate a list of object names. Sort the list of object names in different directions according to the global search order.

5. The method according to claim 1, wherein The process of calculating the matching scores of the areas to be searched in different directions includes: Based on the object name list in different directions and the pre-built region category-object name prior probability list, the matching scores of the search area in different directions are calculated; According to the matching scores, the search areas in different directions are sorted.

6. The method according to claim 5, characterized in that The "region category-item name" prior probability list is constructed using statistical methods from a public dataset containing multiple images and their recognition results.

7. The method according to claim 1, characterized in that Also includes: According to the final search direction and the sensor's sensing range, the sub-target point position is generated; According to the location of the sub-target point, the UAV is guided to move gradually towards the target direction through local path planning.

8. The method according to claim 1, characterized in that The final search direction is obtained based on the weighted fusion prior probability score and the large language model score.

9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein: When the processor executes the computing program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images