A method, apparatus and storage medium for automatic generation and evaluation of navigation language instructions.

Abstract

This invention relates to a method, apparatus, and storage medium for the automatic generation and evaluation of navigation language commands. The method involves acquiring an offline trajectory-command pair dataset, training a language command generation model using a first loss function based on multimodal contrastive learning loss to obtain an optimal language command generation model, and training a command scoring model using the same loss function to obtain an optimal command scoring model. Multiple navigation paths are randomly sampled and input into the optimal language command generation model to generate a trajectory-command pair dataset. This generated dataset is then input into the optimal command scoring model to obtain a similarity score for each trajectory-command pair and calculate an overall evaluation index for the dataset. Finally, trajectory-command pair datasets that meet first and second preset conditions are selected. Compared with existing technologies, this invention significantly improves the quality of command generation and avoids overfitting issues in language commands.

Description

Technical Field

[0001] This invention relates to the field of visual language navigation technology, and in particular to a method, apparatus and storage medium for automatic generation and evaluation of navigation language instructions. Background Technology

[0002] When service robots complete user tasks, they inevitably need to understand natural language instructions and explore unfamiliar environments. To improve service robots' ability to execute instructions in unmapped, unknown environments, visual-language navigation technology has received widespread attention in recent years. This technology emphasizes the robot's understanding and generalization ability of multimodal information, aiming to enable the robot to achieve reasonable navigation behavior solely through natural language descriptions and real-time visual observation.

[0003] Training visual language navigation capabilities heavily relies on expert-annotated data. Typically, the process involves pre-determining a series of navigation paths, then having annotators experience the navigation process in a simulation environment and provide corresponding linguistic descriptions. Due to the complexity of navigation scenarios, the abundance of visible objects, and the ambiguity of language, annotators often require a considerable amount of time to accurately describe a navigation path. Under the constraint of limited costs, current expert-annotated datasets for visual language navigation are relatively small, making it difficult to support the training of navigation models with a large number of parameters and high generalization capabilities.

[0004] While existing techniques have proposed expanding datasets by generating pseudo-labels, the models and training methods used in current solutions are relatively simple and often lack effective post-processing steps, leading to overfitting issues with the generated language commands. These generated commands often contain a large number of repetitive words or descriptions that do not conform to the navigation path, making it difficult to effectively improve the training performance of the navigation model and potentially even negatively impacting robot behavior. Therefore, improving the generation quality of visual navigation language commands has become a problem that needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art in terms of the difficulty in improving the quality of language command generation, and to provide a method, apparatus and storage medium for automatic generation and evaluation of navigation language commands. This invention can automatically generate large-scale, high-quality and highly generalized navigation language commands, and effectively evaluate and filter the generated commands, thereby effectively improving the visual language navigation capabilities of service robots.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a method for automatic generation and evaluation of navigation language commands based on multimodal contrastive learning is provided, comprising the following steps: S1, acquiring an offline trajectory-command pair dataset based on expert samples; S2, training a preset language command generation model for navigation trajectories using a preset first loss function based on the offline trajectory-command pair dataset, to obtain a trained optimal language command generation model, wherein the first loss function is jointly constructed based on multimodal contrastive learning loss and end-to-end language generation loss; S3, training a preset command scoring model using a preset multimodal contrastive learning loss function based on the offline trajectory-command pair dataset, to obtain a trained optimal command scoring model; 4. Randomly sample multiple navigation paths and input the navigation paths into the optimal language instruction generation model to obtain corresponding language instructions, thereby obtaining a generated trajectory-instruction pair dataset; S5. Input the generated trajectory-instruction pair dataset into the optimal instruction scoring model to obtain the similarity score of each trajectory-instruction pair in the generated trajectory-instruction pair dataset, and calculate the overall evaluation index of the dataset; S6. Based on the overall evaluation index of the dataset and the similarity score, select trajectory-instruction pair datasets that meet the first preset condition and the second preset condition. The first preset condition is used to determine the overall quality of the dataset, and the second preset condition is used to determine the quality of each trajectory-instruction pair.

[0008] As a preferred technical solution, S1 specifically includes: by running a virtual intelligent agent in a simulation environment, batch collecting navigation visual trajectory and language command pair data based on expert samples, and storing them in a standardized format to obtain the offline trajectory-command pair dataset.

[0009] As a preferred technical solution, S2 specifically includes: S21, constructing a language instruction generation model for navigation trajectory, wherein the language instruction generation model is an encoder-decoder structure; S22, constructing a first loss function:

[0010]

[0011] In the formula, Denotes the first loss function. This represents a multimodal contrastive learning loss function used to align the feature spaces of visual and linguistic modalities. S23: The loss function of the end-to-end language generation model is used to generate language commands that conform to the navigation trajectory and are natural and fluent. The parameters of the language command generation model are optimized by using gradient descent. The optimal model parameters are determined by minimizing the value of the first loss function, and the optimal language command generation model is obtained.

[0012] As a preferred technical solution, S3 specifically includes: S31, constructing an instruction scoring model, wherein the instruction scoring model is a dual parallel encoder structure; S32, constructing a multimodal contrastive learning loss function; S33, optimizing the parameters of the instruction scoring model using the gradient descent method, determining the optimal model parameters by minimizing the value of the multimodal contrastive learning loss function, and obtaining the optimal language instruction scoring model.

[0013] As a preferred technical solution, S4 specifically includes: S41, randomly selecting a starting point and an ending point in a preset simulator, running a virtual intelligent agent on the shortest path between the two points, collecting multiple navigation visual trajectories, and obtaining a set of navigation visual trajectories; S42, running the optimal language instruction generation model on the set of navigation visual trajectories to obtain the language instruction corresponding to each trajectory.

[0014] As a preferred technical solution, S6 specifically includes: S61, determining the first preset condition and the second preset condition, and calculating the overall evaluation index of the current sample dataset; S62, determining whether the overall evaluation index of the current dataset meets the first preset condition: if yes, then execute S63; if no, then adjust the hyperparameters of the language instruction generation model and return to S2; S63, calculating and determining whether the similarity score of the current single sample meets the second preset condition: if yes, then retain the sample; if no, then discard the sample; S64, determining whether all samples in the current dataset have been traversed: if yes, then the filtering is completed; if no, then return to S63.

[0015] As a preferred technical solution, the overall evaluation metrics for the dataset include the average information retrieval metrics Recall and mAP.

[0016] As a preferred technical solution, the first preset condition is that the calculated overall evaluation index of the dataset is greater than or equal to the first preset overall threshold σ. recall And greater than or equal to the second preset overall threshold σ map The second preset condition is that the calculated similarity score is greater than or equal to a preset similarity threshold, specifically:

[0017]

[0018] In the formula, D hq σ represents the final selected trajectory-instruction pair dataset. s The preset similarity threshold, The similarity score for trajectory-instruction pairs. For navigation visual trajectory, For each trajectory, the language instruction is defined, where the subscript i represents the sample number, and D... g This represents the generated trajectory-instruction pair dataset.

[0019] According to a second aspect of the present invention, an apparatus for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described therein.

[0020] According to a third aspect of the present invention, a storage medium is provided having a program stored thereon, characterized in that the program, when executed, implements the method described thereon.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. This invention introduces a contrastive learning loss function to train the instruction generation model, which can enhance the alignment between visual and linguistic feature spaces, thereby enabling instruction generation to more accurately capture details in the navigation trajectory, avoid deviations between semantics and trajectory, and significantly improve the quality of instruction generation;

[0023] 2. This invention constructs an end-to-end command scoring model suitable for visual navigation. By optimizing the model parameters through multimodal contrastive learning, it can effectively determine the matching degree between navigation trajectory and language commands, and can solve the data domain offset problem when existing video-text models are transferred to visual language navigation applications.

[0024] 3. This invention provides a comprehensive evaluation process for instruction sets based on an instruction scoring model, which covers the evaluation of the overall quality of the instruction set and the quality of individual samples. It can not only be used to evaluate the performance of instruction generation models, but also as a post-processing step to screen out high-quality samples and avoid the overfitting problem of language instructions.

[0025] 4. Through path acquisition, instruction generation, and screening and evaluation processes, this invention can generate a large-scale, high-quality navigation language instruction set with strong generalization capabilities. This instruction set can not only be used to expand the training data of the visual language navigation model, but also to further improve the performance of the instruction generation model. Attached Figure Description

[0026] Figure 1 This invention provides an overall flowchart of the method.

[0027] Figure 2 This is a schematic diagram of the navigation scenario and sample provided in the embodiments of the present invention;

[0028] Figure 3 This is a schematic diagram of the instruction generation model structure provided in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the training loss curve of the instruction scoring model provided in this embodiment of the invention;

[0030] Figure 5 This is a schematic diagram of the generation result provided in an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the similarity score matrix provided in an embodiment of the present invention;

[0032] Figure 7 This is a flowchart of the instruction filtering process provided in an embodiment of the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0034] Example

[0035] This embodiment provides a method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning. Through processes such as data collection, model training, instruction generation, and screening and evaluation, it can generate a large-scale, high-quality, and highly generalized navigation language instruction set. For example... Figure 1 As shown, the method includes: Step S1, obtaining an offline trajectory-instruction pair dataset based on expert samples; Step S2, based on the offline trajectory-instruction pair dataset, training a preset language instruction generation model for navigation trajectories using a preset first loss function to obtain a trained optimal language instruction generation model, wherein the first loss function is jointly constructed based on multimodal contrastive learning loss and end-to-end language generation loss; Step S3, based on the offline trajectory-instruction pair dataset, training a preset instruction scoring model using a preset multimodal contrastive learning loss function to obtain a trained optimal instruction scoring model; Step S4, randomly sampling multiple navigation paths, inputting the navigation paths into the optimal language instruction generation model to obtain corresponding language instructions, thus obtaining a generated trajectory-instruction pair dataset; Step S5, inputting the generated trajectory-instruction pair dataset into the optimal instruction scoring model to obtain the similarity score of each trajectory-instruction pair in the generated trajectory-instruction pair dataset, and calculating the overall evaluation index of the dataset; Step S6, based on the overall evaluation index and similarity score of the dataset, selecting trajectory-instruction pair datasets that meet the first preset condition and the second preset condition, wherein the first preset condition is used to determine the overall quality of the dataset, and the second preset condition is used to determine the quality of each trajectory-instruction pair. Specifically:

[0036] Step S1, Trajectory-Command Pair Data Acquisition: By running a virtual agent in a simulation environment, batch acquisition of navigation visual trajectory and language command pair data based on expert samples (i.e., offline trajectory-command pair dataset) is performed and stored in a standardized format.

[0037] For example, based on expert datasets And a simulator that collects paired datasets of navigation visual trajectories and language commands. Among them, expert dataset The simulator uses a 3D reconstructed scene recovered from real images, sourced from publicly available data sources. Optionally, the Matterport 3D reconstructed scene is selected as the navigation configuration scene, Habitat-sim is selected as the data acquisition simulator, and an expert dataset is chosen from the publicly available VLNCE data source. Example scenarios and samples are as follows Figure 2 As shown. The simulator operates as follows: given the scene, start point, end point, and descriptive text of the navigation path, the simulator sets the virtual agent at the start point in the scene and instructs it to move to the end point by following the shortest path, recording the image sequence and word sequence during the process, i.e.:

[0038] v i ,l i =Simulator(c i ,r i ,t i (1)

[0039] In the formula, N is the sample size, and c i ,r i ,t i v represents the setting scenario, navigation path, and description text of the i-th sample, respectively. i ,l i Let v represent the visual image sequence and the language word sequence of the i-th sample, respectively. All v i It will be standardized into a uniform three-channel RGB image.

[0040] In this embodiment, N = 10819, v i All images are normalized to 224×224 three-channel RGB images, compressed into JPEG format, and numbered according to the order in which they appear in the navigation path. i Converted to a string and encoded in UTF-8. Paired datasets. Organized into different folders according to the navigation path, each folder's rgb subfolder stores image files, and the inst subfolder stores instruction files.

[0041] Step S2, Generate Model Training: Using the collected offline trajectory-instruction pair dataset, combined with multimodal contrastive learning and end-to-end language generation loss function, train a language instruction generation model suitable for navigation trajectories, and record the optimal model parameters.

[0042] For example, this step specifically includes:

[0043] Step S21: Construct a language instruction generation model for navigation trajectories. θ is its set of parameters.

[0044] Among these, self-attention networks offer advantages over existing LSTM structures in multimodal processing tasks. (Language instruction generation model) An encoder-decoder structure is employed. The encoder is an image processing self-attention network used to receive image sequence input and generate visually representative features. and visual coding features The decoder is a language generation self-attention network used to autoregressively output linguistic symbols based on visually encoded features. The model structure is as follows: Figure 3 As shown, the model contains a large number of learnable parameters, denoted as the set θ.

[0045] In order to aggregate feature information from multiple frames of images into a single visual representative feature exist An additional feature mean pooling layer is added to the encoder, and the calculation process is as follows:

[0046]

[0047] In the formula, The encoding features of the k-th frame image are represented by T, where T is the total length of the image sequence.

[0048] Step S22, construct the first loss function for training the generative model. Learning loss function from multimodal contrastive and end-to-end language generation model loss function The multimodal contrastive learning loss function was calculated together. The loss function for aligning the feature spaces of visual and linguistic modalities in an end-to-end language generation model. Used to generate natural and fluent language commands that conform to the navigation trajectory.

[0049] Using only a language generation loss function makes it difficult for the encoder to be adequately trained. Insufficient modal association degrades the quality of generated instructions, thus necessitating the introduction of a contrastive learning loss function. Specifically, from the model... Additional language representation features are extracted in the language decoder. Combining visual representative features The cosine similarity s between the i-th and j-th samples in a training batch can be calculated. i,j and multimodal contrastive learning loss function

[0050]

[0051] In the formula, s() is a similarity measure, and the cosine of the angle between the vectors is used here; The L2 norm of the corresponding vector was calculated; InfoNCE() is a noise contrast estimation function, and the lower the function value, the better the multimodal alignment effect; B is the number of batch data.

[0052] Then, combine the instructions generated by the model. With truth instruction l i The language model loss function can be calculated:

[0053]

[0054] In the formula, CE() is a cross-entropy measure. The lower the function value, the more accurate the language model.

[0055] The total loss function (i.e., the first loss function) is the sum of the two partial loss functions:

[0056]

[0057] In practice, it is also necessary to calculate the average value of the loss function on a batch of samples to avoid the negative impact of excessively large values.

[0058] Step S23: Optimize the parameters of the language instruction generation model using gradient descent to obtain the optimal model.

[0059]

[0060] In the formula, The instruction generation model with parameter θ in dataset D o The average loss over time. By minimizing this first loss function. The optimal parameters can be obtained by finding the value of . The optimal language instruction generation model is obtained. Optionally, the AdamW algorithm is used to minimize the loss function. Using the AdamW algorithm to update the parameter set batch by batch not only adaptively sets the learning rate for each parameter but also has an improved regularization method, achieving better convergence and generalization. Once the loss function converges, the optimal model parameters are recorded.

[0061] In step S2, the main hyperparameters of the model and training are: 12 layers for the model encoder / decoder, 512 dimensions for the hidden state, 8 attention heads, a learning rate of 0.00001, a batch size of 2, and a maximum number of training epochs of 18.

[0062] Step S3, scoring model training: Using the collected offline dataset, a command scoring model for measuring the similarity between navigation trajectories and language commands is trained through a multimodal contrastive loss function, and the optimal model parameters are recorded.

[0063] For example, this step specifically includes:

[0064] Step S31: Construct a language instruction scoring model with a dual parallel encoder structure. ω is its set of parameters.

[0065] Optionally, a language instruction scoring model A dual-encoder structure is employed. The visual encoder is designed as a visual Transformer structure, receiving image sequence input and generating visually representative features. and visual encoding features The language encoding is designed using the BERT structure, which receives linguistic symbols as input and generates linguistic representation features. The model contains a large number of learnable parameters, denoted as the set ω.

[0066] To obtain visual representative features by relating them to temporal context exist An additional multi-head self-attention network layer was added to the visual encoder, and the calculation method is as follows:

[0067]

[0068] In the formula, W represents the learnable parameter matrix, and d k The feature dimension is Softmax(), the normalization function is H, the number of attention heads is H, and the feature matrices are all treated as column vectors concatenated column by column.

[0069] Step S32: Construct a loss function for training the scoring model. The scoring model evaluates the similarity between navigation trajectories and language commands. The loss calculation method can reuse the multimodal contrastive learning loss function from step S22, i.e. Specifically:

[0070]

[0071] Step S33: Optimize the parameters of the instruction scoring model using gradient descent to obtain the optimal model.

[0072]

[0073] In the formula, The instruction rating model with parameter ω is represented in dataset D. o The average loss over time. The optimal parameters can be obtained by minimizing the value of this loss function. The optimal language instruction generation model is obtained. Optionally, the AdamW algorithm is used to minimize the loss function. Figure 4 The curve showing the change in the loss function value during the training process is displayed.

[0074] In step S3, the main hyperparameters are: 12 layers for both the visual encoder and the language encoder, 768 hidden state dimensions, 12 attention heads, a learning rate of 0.00005, a batch size of 8, and a maximum number of training rounds of 20.

[0075] Step S4, navigation instruction generation: randomly sample large-scale navigation paths and use the recorded optimal instruction generation model to generate corresponding language instructions for these paths.

[0076] For example, this step specifically includes:

[0077] Step S41: Randomly select a start point and an end point in a preset simulator, run a virtual agent on the shortest path between the two points, and collect a set of M navigation visual trajectories. Where M >> M.

[0078] Specifically: For the i-th sample to be collected, firstly, its scene is randomly selected. Then, within the coordinate axis range of the scene, a starting point is randomly selected. and termination point Both are two-dimensional coordinates within the site. If an infeasible area is randomly selected, a new one is chosen. Dijkstra's algorithm is used to find the shortest path within the feasible region between the two points, which serves as the navigation path r for this sample. i g Then set up a virtual agent and traverse path r. i g Obtain the visual observation sequence along the path. (i.e., navigation visual trajectories). The agent is equipped with a monocular RGBD camera with an image resolution of 224, a field of view of 90 degrees, and a camera height of 1.25 meters. This process requires no human intervention, thus enabling the collection of a large-scale set of navigation visual trajectories, far exceeding the number of expert-annotated samples. Specifically, choose M = 159849.

[0079] Step S42, in the navigation visual trajectory set Then, run the optimal language instruction generation model to obtain the language instructions corresponding to each trajectory:

[0080]

[0081] In the formula, φ indicates that the language instruction generation model does not require input language modalities during forward inference. During forward inference, to ensure the richness of generated instructions, random sampling decoding can be used instead of greedy decoding. Random sampling decoding refers to the model generating a probability distribution over a set of language symbols for each generation position, and then randomly sampling a symbol from this distribution as the generation result for that position. Specifically, the top_p decoding method is used, retaining the set of symbols with a total probability of 0.7 and randomly sampling from it.

[0082] After the process is complete, This is a newly generated dataset. Figure 5 Some generated results are provided.

[0083] Step S5, Navigation Instruction Evaluation: The generated trajectory-instruction pairs are scored for similarity using the recorded optimal instruction scoring model, and the quality of the generated data is comprehensively evaluated through multiple information retrieval indicators.

[0084] For example, this step specifically includes:

[0085] For D g Running the instruction scoring model yields a similarity score for each data pair, i.e.:

[0086]

[0087] Here, the instruction scoring model is the optimal instruction scoring model obtained through training, and its evaluation score is... Measurable generated language instructions With path The degree of matching. The higher the score, the more accurate the instruction. Figure 6 The similarity score matrix of a batch of 32 samples is shown. To avoid gradient problems during the training process, the similarity scores are all multiplied by a fixed magnification factor of 1.74.

[0088] Furthermore, for the entire generation instruction set D g Based on similarity scores, overall evaluation metrics for the dataset can be calculated, such as Recall and mAP (mean average information retrieval rate), to evaluate the overall quality of the instruction set. The Recall score measures the ability of an instruction to match all positive instances, while the mAP score comprehensively assesses recall and precision. Higher scores indicate a more accurate instruction set.

[0089] The recall score refers to the ratio of the number of true values ​​among the top K highest-scoring samples to the total number of true values. The specific calculation process is as follows:

[0090]

[0091] In the formula, Representative and visual trajectory The number of matched instructions. Representatives based on scores Select the K instructions with the highest scores, which include visual trajectories. Number of matches.

[0092] The mAP score refers to the accuracy value recorded each time a true match is found, by iterating through the window from 1 to K. The average of all accuracy values ​​is the AP value, and the average of the AP values ​​over the entire dataset is the mAP. The specific calculation process is as follows:

[0093]

[0094] In the formula, POS is the set of window sizes where a matching truth value occurs. For the entire generated instruction set D... g The larger the average information retrieval metrics Recall and mAP, the more accurate the instruction set is.

[0095] In other embodiments, depending on the actual scenario, metrics such as HitRate (match hit rate) and MRR (mean reciprocal ranking) can also be used for evaluation.

[0096] Step S6, Navigation Instruction Application: Based on the evaluation metrics, high-quality generated instruction sets are selected, which can be used to train new visual language navigation models.

[0097] For example, this step specifically includes:

[0098] Step S61: Determine the first and second preset conditions, and calculate the overall evaluation index Recall(D) of the current sample dataset. g ) and mAP(D g );

[0099] Step S62: Determine whether the overall evaluation index of the current dataset meets the first preset condition.

[0100] If yes, proceed to step S63; if no, adjust the hyperparameters of the instruction-generated model and return to step S2.

[0101] Step S63: Calculate and determine the similarity score of the current single sample. Does it meet the second preset condition?

[0102] If yes, keep the sample; if no, discard the sample.

[0103] Step S64: Determine whether all samples in the current dataset have been traversed.

[0104] If yes, the filtering is complete; if no, return to step S63.

[0105] The first preset condition is that the calculated overall evaluation index of the dataset is greater than or equal to the first preset overall threshold σ. recall And greater than or equal to the second preset overall threshold σ map Specifically, Recall(D) is a metric based on the expert dataset. o ),mAP(D o As a reference, set the overall threshold σ. recall and σ map Specifically, σ can be selected. recall =0.50,σ map =0.40 is used as a reference value. If Recall(D g )<σ recall ∨mAP(D g )<σ map If the generated instruction set is of low quality, steps S2-S5 should be repeated until the overall performance of the instruction set reaches the threshold. During the repetition process, model hyperparameters (number of self-attention layers, hidden state feature dimension, activation function, etc.), training hyperparameters (learning rate, batch size, number of training steps, etc.), and generation hyperparameters (sampling method, sampling threshold, beam search, etc.) can be adjusted to optimize the generation effect.

[0106] The second preset condition is that the calculated similarity score is greater than or equal to a preset similarity threshold. Specifically, the similarity threshold σ is set. s Select high-quality dataset D hq , that is:

[0107]

[0108] Specifically, σ can be selected. s =0.60 is used as a reference value. Figure 7 This illustrates one implementation of the above screening process.

[0109] Then, based on the information recorded when collecting the random path in step S4, D can be... hq The reverse mapping is to the expert dataset D. e The same format. This generated dataset can be used for data augmentation training in visual language navigation, and also for optimizing instruction generation models, forming a semi-supervised training process.

[0110] In summary, the method provided in this embodiment can automatically annotate navigation language instructions, and has the advantages of high accuracy, low cost and ease of use.

[0111] Furthermore, this embodiment also provides an automatic generation and evaluation device for navigation language instructions based on multimodal contrastive learning, including a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements one or more steps of the aforementioned method, which will not be described in detail here.

[0112] Furthermore, this embodiment also provides a storage medium on which a program is stored. When the program is executed, it implements one or more steps of the aforementioned method, which will not be described in detail here.

[0113] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning, characterized in that, Includes the following steps: S1, Obtain an offline trajectory-instruction pair dataset based on expert samples; S2, based on the offline trajectory-instruction pair dataset, a preset language instruction generation model for navigation trajectory is trained using a preset first loss function to obtain the trained optimal language instruction generation model. The first loss function is constructed based on multimodal contrastive learning loss and end-to-end language generation loss. S3. Based on the offline trajectory-instruction pair dataset, a preset instruction scoring model is trained using a preset multimodal contrastive learning loss function to obtain the trained optimal instruction scoring model. S4. Randomly sample multiple navigation paths, input the navigation paths into the optimal language instruction generation model to obtain the corresponding language instructions, and then obtain the generated trajectory-instruction pair dataset; S5, input the generated trajectory-instruction pair dataset into the optimal instruction scoring model to obtain the similarity score of each trajectory-instruction pair in the generated trajectory-instruction pair dataset, and calculate the overall evaluation index of the dataset; the overall evaluation index of the dataset includes Recall, mAP, HitRate or MRR; S6. Based on the overall evaluation index of the dataset and the similarity score, select trajectory-instruction pair datasets that meet the first preset condition and the second preset condition. The first preset condition is used to determine the overall quality of the dataset, and the second preset condition is used to determine the quality of each trajectory-instruction pair.

2. The method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that, S1 specifically includes: by running a virtual intelligent agent in a simulation environment, batch collecting navigation visual trajectory and language command pair data based on expert samples, and storing them in a standardized format to obtain the offline trajectory-command pair dataset.

3. The method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that, S2 specifically includes: S21, Construct a language instruction generation model for navigation trajectory, wherein the language instruction generation model is an encoder-decoder structure; S22, Construct the first loss function: In the formula, Denotes the first loss function. This represents a multimodal contrastive learning loss function used to align the feature spaces of visual and linguistic modalities. This represents the loss function of the end-to-end language generation model, used to generate natural and fluent language commands that conform to the navigation trajectory. S23, the parameters of the language instruction generation model are optimized using the gradient descent method. The optimal model parameters are determined by minimizing the value of the first loss function, and the optimal language instruction generation model is obtained.

4. The method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that, S3 specifically includes: S31, Construct an instruction scoring model, wherein the instruction scoring model is a dual parallel encoder structure; S32, Construct a multimodal contrastive learning loss function; S33, the parameters of the instruction scoring model are optimized using the gradient descent method. The optimal model parameters are determined by minimizing the value of the multimodal contrastive learning loss function, thus obtaining the optimal language instruction scoring model.

5. The method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that, S4 specifically includes: S41, randomly select the starting point and the ending point in the preset simulator, run the virtual intelligent agent on the shortest path between the two points, collect multiple navigation visual trajectories, and obtain a set of navigation visual trajectories; S42, run the optimal language instruction generation model on the set of navigation visual trajectories to obtain the language instructions corresponding to each trajectory.

6. The method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that, S6 specifically includes: S61, determine the first preset condition and the second preset condition, and calculate the overall evaluation index of the dataset for the current sample; S62, determine whether the overall evaluation metric of the current dataset meets the first preset condition: If yes, then execute S63; if no, then adjust the hyperparameters of the language instruction generation model and return to S2. S63, calculate and determine whether the similarity score of the current single sample meets the second preset condition: If yes, then retain the sample; if no, then discard the sample. S64, determine whether all samples in the current dataset have been traversed: If yes, the filtering is complete; if no, return to S63.

7. The method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning according to claim 6, characterized in that, The overall evaluation metrics for the dataset include average information retrieval metrics Recall and mAP.

8. The method for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning according to claim 7, characterized in that, The first preset condition is that the calculated overall evaluation index of the dataset is greater than or equal to the first preset overall threshold. And greater than or equal to the second preset overall threshold ; The second preset condition is that the calculated similarity score is greater than or equal to a preset similarity threshold, specifically: In the formula, This represents the final selected trajectory-instruction pair dataset. The preset similarity threshold, The similarity score for trajectory-instruction pairs. For navigation visual trajectory, For each trajectory, the corresponding language instruction is given, where the subscript i represents the sample number. This represents the generated trajectory-instruction pair dataset.

9. A device for automatic generation and evaluation of navigation language instructions based on multimodal contrastive learning, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images