Navigation language instruction automatic generation and evaluation method and device and storage medium

Abstract

The invention relates to a navigation language instruction automatic generation and evaluation method and device and a storage medium. Obtaining an offline trajectory-instruction pair data set, training a language instruction generation model by using a first loss function based on multi-modal contrast learning loss to obtain an optimal language instruction generation model, and training an instruction scoring model by using the multi-modal contrast learning loss function to obtain an optimal instruction scoring model; randomly sampling a plurality of navigation paths, and inputting an optimal language instruction generation model to obtain a generated track-instruction pair data set; and inputting the generated data set into an optimal instruction scoring model, obtaining a similarity score of each track-instruction pair, calculating an overall evaluation index of the data set, and finally screening out the track-instruction pair data set meeting a first preset condition and a second preset condition. Compared with the prior art, the method has the advantages that the instruction generation quality is remarkably improved, and the over-fitting problem of language instructions is avoided.

Description

Technical Field

[0001] The present invention relates to the field of visual language navigation technology, and in particular to a method, device and storage medium for automatically generating and evaluating navigation language instructions. Background Art

[0002] When completing user tasks, service robots inevitably involve understanding natural language instructions and exploring unfamiliar environments. In order to improve the service robot's ability to execute instructions in unknown environments without maps, visual language navigation technology has received widespread attention in recent years. This technology emphasizes the robot's understanding and generalization ability of multimodal information, and aims to enable the robot to achieve reasonable navigation behavior only through natural language descriptions and real-time visual observations.

[0003] The training of visual language navigation ability is highly dependent on expert annotated data. Usually, the process of collecting expert data is to predetermine a series of navigation paths, and then let the annotators experience the navigation process in a simulated environment and provide corresponding language descriptions. Due to the complexity of the navigation scene, the large number of visible objects and the ambiguity of the language, it often takes a long time for the annotator to accurately describe a navigation path. Under the constraint of limited cost, the current expert annotated dataset for visual language navigation is small in scale and it is difficult to support the training of navigation models with large parameters and high generalization.

[0004] Although existing technologies have proposed to expand the data set by generating pseudo labels, the models and training methods used in existing solutions are relatively simple, and usually lack effective post-processing steps, resulting in overfitting problems in language instructions. These generated instructions often contain a large number of repetitive words or descriptions that do not conform to the navigation path, making it difficult to effectively improve the training effect of the navigation model and may even have a negative impact on the robot's behavior. Therefore, how to improve the generation quality of visual navigation language instructions has become a problem that needs to be solved in this field. Summary of the invention

[0005] The purpose of the present invention is to overcome the defect that the quality of language instruction generation is difficult to improve in the above-mentioned prior art and to provide a method, device and storage medium for automatic generation and evaluation of navigation language instructions, which can automatically generate large-scale, high-quality and strongly generalized navigation language instructions, and effectively evaluate and screen the generated instructions, so as to effectively improve the visual language navigation capability of the service robot.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided a method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning, comprising the following steps: S1, obtaining an offline trajectory-instruction pair data set based on expert samples; S2, based on the offline trajectory-instruction pair data set, using a preset first loss function to train a preset language instruction generation model for navigation trajectories to obtain a trained optimal language instruction generation model, wherein the first loss function is jointly constructed based on a multimodal contrastive learning loss and an end-to-end language generation loss; S3, based on the offline trajectory-instruction pair data set, using a preset multimodal contrastive learning loss function to train a preset instruction scoring model to obtain a trained optimal instruction scoring model; S 4. Randomly sample multiple navigation paths, input the navigation paths into the optimal language instruction generation model, obtain corresponding language instructions, and then obtain a generated trajectory-instruction pair data set; S5. Input the generated trajectory-instruction pair data set into the optimal instruction scoring model, obtain the similarity score of each trajectory-instruction pair in the generated trajectory-instruction pair data set, and calculate the overall evaluation index of the data set; S6. Based on the overall evaluation index of the data set and the similarity score, screen out the trajectory-instruction pair data set that meets the first preset condition and the second preset condition, the first preset condition is used to determine the overall quality of the data set, and the second preset condition is used to determine the quality of each trajectory-instruction pair.

[0008] As a preferred technical solution, S1 specifically includes: running a virtual intelligent agent in a simulation environment, batch collecting navigation visual trajectory and language instruction paired data based on expert samples, and storing them in a standardized format to obtain the offline trajectory-instruction pair data set.

[0009] As a preferred technical solution, the S2 specifically includes: S21, constructing a language instruction generation model for navigation trajectory, wherein the language instruction generation model is a codec structure; S22, constructing a first loss function:

[0010]

[0011] In the formula, represents the first loss function, represents the multimodal contrastive learning loss function, which is used to align the feature spaces of visual and language modalities, Represents an end-to-end language generation model loss function, which is used to generate language instructions that conform to the navigation trajectory and are natural and fluent; S23, using the gradient descent method to optimize the parameters of the language instruction generation model, determine the optimal model parameters by minimizing the value of the first loss function, and obtain the optimal language instruction generation model.

[0012] As a preferred technical solution, S3 specifically includes: S31, constructing an instruction scoring model, which is a dual parallel encoder structure; S32, constructing a multimodal contrastive learning loss function; S33, optimizing the parameters of the instruction scoring model using the gradient descent method, determining the optimal model parameters by minimizing the value of the multimodal contrastive learning loss function, and obtaining the optimal language instruction scoring model.

[0013] As a preferred technical solution, the S4 specifically includes: S41, randomly selecting a starting point and an end point in a preset simulator, running a virtual agent on the shortest path between the two points, collecting multiple navigation visual trajectories, and obtaining a navigation visual trajectory set; S42, running the optimal language instruction generation model on the navigation visual trajectory set to obtain language instructions corresponding to each trajectory.

[0014] As an optimal technical solution, S6 specifically includes: S61, determining the first preset condition and the second preset condition, and calculating the overall evaluation index of the data set of the current sample; S62, judging whether the overall evaluation index of the current data set meets the first preset condition: if yes, executing S63; if no, adjusting the hyperparameters of the language instruction generation model and returning to S2; S63, calculating and judging whether the similarity score of the current single sample meets the second preset condition: if yes, retaining the sample; if no, discarding the sample; S64, judging whether all samples in the current data set have been traversed: if yes, completing the screening; if no, returning to S63.

[0015] As a preferred technical solution, the overall evaluation indicators of the dataset include average information retrieval indicators Recall and mAP.

[0016] As a preferred technical solution, the first preset condition is that the calculated overall evaluation index of the data set is greater than or equal to the first preset overall threshold σ recall and is greater than or equal to the second preset overall threshold σ map ; The second preset condition is that the calculated similarity score is greater than or equal to the preset similarity threshold, specifically:

[0017]

[0018] Where D hq represents the final selected trajectory-instruction pair dataset, σ s is the preset similarity threshold, is the similarity score of the trajectory-command pair, To navigate the visual trajectory, is the language instruction corresponding to each trajectory, the subscript i represents the sample number, D g Represents the generated trajectory-instruction pair dataset.

[0019] According to a second aspect of the present invention, there is provided a device for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning, comprising a memory, a processor, and a program stored in the memory, wherein the method described is implemented when the processor executes the program.

[0020] According to a third aspect of the present invention, there is provided a storage medium having a program stored thereon, wherein the program implements the method described above when executed.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The present invention introduces a contrastive learning loss function to train the instruction generation model, which can enhance the alignment of visual and language feature spaces, so that instruction generation can more accurately capture the details in the navigation trajectory, avoid the deviation between semantics and trajectory, and significantly improve the quality of instruction generation;

[0023] 2. The present invention constructs an end-to-end instruction scoring model suitable for visual navigation, and optimizes the model parameters through multimodal contrast learning, so that it can effectively judge the matching degree between the navigation trajectory and the language instruction, and can solve the data domain offset problem when the existing video-text model is migrated to the visual language navigation application;

[0024] 3. The present invention provides a comprehensive instruction set evaluation process based on an instruction scoring model, which covers the evaluation of the overall quality of the instruction set and the quality of a single sample. It can not only be used to evaluate the performance of the instruction generation model, but also can be used as a post-processing step to screen out high-quality samples to avoid the overfitting problem of language instructions;

[0025] 4. Through the path collection, instruction generation and screening evaluation process, the present invention can generate a large-scale, high-quality navigation language instruction set with strong generalization ability. This instruction set can not only be used to expand the training data of the visual language navigation model, but also can be used to further improve the effect of the instruction generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention provides a schematic diagram of the overall process of the method;

[0027] Figure 2 A schematic diagram of a navigation scenario and a sample provided in an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of the structure of an instruction generation model provided in an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of a training loss curve of an instruction scoring model provided in an embodiment of the present invention;

[0030] Figure 5 A schematic diagram of a generation result provided in an embodiment of the present invention;

[0031] Figure 6 A schematic diagram of a similarity score matrix provided in an embodiment of the present invention;

[0032] Figure 7 This is a flowchart of instruction screening provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0034] Example

[0035] This embodiment provides a method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning. After data collection, model training, instruction generation, screening and evaluation, a large-scale, high-quality, and highly generalized navigation language instruction set can be generated. Figure 1 As shown, the method includes: step S1, obtaining an offline trajectory-instruction pair data set based on expert samples; step S2, based on the offline trajectory-instruction pair data set, using a preset first loss function to train a preset language instruction generation model for navigation trajectories to obtain a trained optimal language instruction generation model, the first loss function is jointly constructed based on multimodal contrast learning loss and end-to-end language generation loss; step S3, based on the offline trajectory-instruction pair data set, using a preset multimodal contrast learning loss function to train a preset instruction scoring model to obtain a trained optimal instruction scoring model; step S4, randomly sampling multiple navigation paths, inputting the navigation paths into the optimal language instruction generation model, obtaining corresponding language instructions, and obtaining a generated trajectory-instruction pair data set; step S5, inputting the generated trajectory-instruction pair data set into the optimal instruction scoring model, obtaining a similarity score of each trajectory-instruction pair in the generated trajectory-instruction pair data set, and calculating the overall evaluation index of the data set; step S6, based on the overall evaluation index of the data set and the similarity score, screening out trajectory-instruction pair data sets that meet the first preset condition and the second preset condition, the first preset condition is used to determine the overall quality of the data set, and the second preset condition is used to determine the quality of each trajectory-instruction pair. Specifically:

[0036] Step S1, trajectory-command pair data collection: by running a virtual agent in a simulation environment, batch collect navigation visual trajectory and language command paired data based on expert samples (i.e., offline trajectory-command pair dataset) and store them in a standardized format.

[0037] For example, based on the expert dataset And the simulator Simulator, collects paired data sets of navigation visual trajectories and language commands Among them, the expert dataset Taken from public data sources, the simulator uses a 3D reconstruction scene restored from real images. Optionally, select Matterport 3D reconstruction scene as the navigation configuration scene, select Habitat-sim as the data acquisition simulator, and select the expert dataset from the public data source VLNCE Example scenarios and samples such as Figure 2 As shown in the figure, the simulator runs in the following way: given the scene, starting point, end point and description text of the navigation path, the simulator sets the virtual agent at the starting point in the scene and lets it move to the end point by the shortest path, recording the image sequence and word sequence in the process, namely:

[0038] v i ,l i =Simulator(c i ,r i ,t i ) (1)

[0039] In the formula, N is the number of samples, c is i ,r i ,t i Respectively represent the setting scene, navigation path and description text of the i-th sample, v i ,l i Represent the visual image sequence and language word sequence of the i-th sample respectively. All v i Will be normalized into a unified three-channel RGB image.

[0040] In this embodiment, N = 10819, v i All images in the navigation path are normalized into three-channel RGB images of size 224×224, compressed into JPEG format, and numbered in the order in which the images appear in the navigation path. i is converted to a string and encoded in UTF-8. Organized into different folders according to the navigation path, the rgb subfolder under each folder stores the image files, and the inst subfolder stores the instruction files.

[0041] Step S2, generation model training: using the collected offline trajectory-command pair dataset, combined with multimodal contrastive learning and end-to-end language generation loss function, train a language command generation model suitable for navigation trajectories and record the optimal model parameters.

[0042] Exemplarily, this step specifically includes:

[0043] Step S21: construct a language instruction generation model for navigation trajectory θ is its parameter set.

[0044] Compared with the LSTM structure of the existing technical solution, the self-attention network has more advantages in multimodal processing tasks. Language instruction generation model The encoder-decoder structure is adopted. The encoder is an image processing self-attention network, which is used to receive image sequence input and generate visual representative features. And visual encoding features The decoder is a language generation self-attention network, which is used to output language symbols autoregressively under the condition of visual encoding features. The model structure is as follows Figure 3 As shown, the model contains a large number of learnable parameters, denoted as set θ.

[0045] In order to aggregate the feature information of multiple frames into a visual representative feature exist A feature mean pooling layer is added to the encoder of , and the calculation process is:

[0046]

[0047] In the formula, Represents the encoding features of the k-th frame image, and T is the total length of the image sequence.

[0048] Step S22, constructing a first loss function for generating model training Loss function learned by multimodal contrast and the end-to-end language generation model loss function Calculated together, where the multimodal contrastive learning loss function Loss function for end-to-end language generation model used to align feature spaces of visual and linguistic modalities Used to generate language instructions that are consistent with the navigation trajectory and natural and fluent.

[0049] If only the language generation loss function is used, it is difficult for the encoder to be fully trained. Insufficient modality association will reduce the quality of generated instructions, so it is necessary to introduce a contrastive learning loss function. Specifically, from the model The language decoder additionally elicits language representation features Combining visual representation features The cosine similarity s between the i-th sample and the j-th sample in a training batch can be calculated i,j And the multimodal contrastive learning loss function

[0050]

[0051] In the formula, the s() function is a similarity measure, and the cosine value of the vector angle is used here; The binary norm of the corresponding vector is calculated; InfoNCE() is a noise contrast estimation function. The lower the function value, the better the multimodal alignment effect; B is the number of batch data.

[0052] Then, combined with the instructions generated by the model With the truth instruction l i , the language model loss function can be calculated:

[0053]

[0054] In the formula, the CE() function is the cross entropy metric. The lower the function value, the more accurate the language model.

[0055] The total loss function (i.e. the first loss function) is the sum of the two loss functions:

[0056]

[0057] In actual implementation, it is also necessary to average the loss function values ​​on a batch of samples to avoid the negative impact caused by excessively large values.

[0058] Step S23, using the gradient descent method to optimize the parameters of the language instruction generation model to obtain the optimal model:

[0059]

[0060] In the formula, The instruction generation model with the representative parameter θ in the dataset D o The average loss on . By minimizing the first loss function The optimal parameter can be obtained by The optimal language instruction generation model is obtained. Optionally, the AdamW algorithm is used to minimize the loss function. Using the AdamW algorithm to update the parameter set batch by batch can not only adaptively set the learning rate for each parameter, but also has an improved regularization method, which can achieve better convergence and generalization. When the loss function value converges, the optimal model parameters are recorded.

[0061] In step S2, the main hyperparameters of the model and training are: the number of model encoder-decoder layers is 12, the hidden state dimension is 512, the number of attention heads is 8, the learning rate is 0.00001, the batch size is 2, and the maximum number of training rounds is 18.

[0062] Step S3, scoring model training: using the collected offline data set, through the multimodal contrast loss function, train the instruction scoring model used to measure the similarity between the navigation trajectory and the language instruction, and record the optimal model parameters.

[0063] Exemplarily, this step specifically includes:

[0064] Step S31: constructing a language instruction scoring model with a dual parallel encoder structure ω is its parameter set.

[0065] Optionally, a language instruction scoring model Two parallel encoder structures are used. The visual encoder is designed as a visual Transformer structure, which receives image sequence input and generates visual representative features. and visual encoding features The language encoding is designed as a BERT structure, which receives language symbol input and generates language representative features. The model contains a large number of learnable parameters, denoted as set ω.

[0066] To obtain visual representation features in conjunction with temporal context exist An additional layer of multi-head self-attention network is added to the visual encoder of , and the calculation method is:

[0067]

[0068] Where W represents the learnable parameter matrix, d k is the feature dimension, Softmax() is the normalization function, H is the number of attention heads, and the feature matrices are all regarded as column vectors and concatenated column by column.

[0069] Step S32, construct a loss function for scoring model training. The function of the scoring model is to evaluate the similarity between the navigation trajectory and the language instruction. The loss calculation method can reuse the multimodal contrast learning loss function in step S22, that is, Specifically:

[0070]

[0071] Step S33, using the gradient descent method to optimize the parameters of the instruction scoring model to obtain the optimal model:

[0072]

[0073] In the formula, The instruction scoring model with the representative parameter ω in the dataset D o The average loss on . By minimizing the value of this loss function, the optimal parameter can be obtained The optimal language instruction generation model is obtained. Optionally, the AdamW algorithm is used to minimize the loss function. Figure 4 The changing curve of the loss function value during the training process is shown.

[0074] In step S3, the main hyperparameters are: the number of layers of the visual encoder and the language encoder is 12, the hidden state dimension is 768, the number of attention heads is 12, the learning rate is 0.00005, the batch size is 8, and the maximum number of training rounds is 20.

[0075] Step S4, navigation instruction generation: randomly sample large-scale navigation paths, and use the recorded optimal instruction generation model to generate corresponding language instructions for these paths.

[0076] Exemplarily, this step specifically includes:

[0077] Step S41, randomly select a starting point and an end point in a preset simulator, run the virtual agent on the shortest path between the two points, and collect a set of M navigation visual trajectories Among them, M>>M.

[0078] Specifically: for the i-th sample to be collected, first randomly select the scene to which it belongs Then, within the coordinate axis range of the scene, randomly select the starting point and end point Both are two-dimensional coordinates in the field. If they are randomly in an infeasible area, they will be reselected. The Dijkstra algorithm is used to solve the shortest path in the feasible area between two points as the navigation path r of the sample. i g Then set up a virtual agent to traverse the path r i g , obtain the visual observation sequence on the path (i.e., navigation visual trajectory). The agent is equipped with a monocular RGBD camera with an image resolution of 224, a field of view of 90 degrees, and a camera height of 1.25 meters. This process does not require human intervention, so it is possible to collect a large number of navigation visual trajectories that far exceed the samples annotated by experts, and obtain a collection of navigation visual trajectories. Specifically choose M=159849.

[0079] Step S42, in the navigation visual trajectory collection Run the optimal language instruction generation model on to get the language instructions corresponding to each trajectory:

[0080]

[0081] In the formula, φ indicates that the language instruction generation model does not need to input the language modality during forward reasoning. During the forward reasoning process, in order to ensure the richness of the generated instructions, random sampling decoding can be used instead of greedy decoding. Random sampling decoding means that for each generation position, the model generates a probability distribution on a language symbol set, and then randomly samples a symbol from the distribution as the generation result of the position. Specifically, the top_p decoding method is used to retain the symbol set with a total probability of 0.7 and randomly sample from it.

[0082] After the run is completed, This is the new dataset that is automatically generated. Figure 5 Some generated results are given.

[0083] Step S5, navigation instruction evaluation: use the recorded optimal instruction scoring model to perform similarity scoring on the generated trajectory-instruction pairs, and comprehensively evaluate the quality of the generated data through multiple information retrieval indicators.

[0084] Exemplarily, this step specifically includes:

[0085] Right D g Running the instruction scoring model obtains the similarity score for each data pair, namely:

[0086]

[0087] Here, the instruction scoring model is the optimal instruction scoring model obtained through training, and its evaluation score is Measurable language instruction With path The larger the score, the more accurate the instruction. Figure 6 The similarity score matrix of a batch of 32 samples is shown. To avoid gradient problems in the training process, the similarity scores are multiplied by a fixed magnification factor of 1.74.

[0088] In addition, for the entire generation instruction set D g , the overall evaluation index of the dataset can be calculated based on the similarity score, such as the average information retrieval index Recall and mAP, which are used to evaluate the overall quality of the instruction set. The Recall score is used to measure the ability of the instruction to match all positive examples, and the mAP score is used to comprehensively judge the recall and precision capabilities. The larger the two scores, the more accurate the instruction set.

[0089] The recall score refers to the ratio of the number of true values ​​among the first K samples with the highest scores to the number of all true values. The specific calculation process is:

[0090]

[0091] In the formula, Representation and visual trajectories The number of instructions that matched, Representatives based on ratings Select the K instructions with the largest scores, including the visual trajectory The number of matches.

[0092] The mAP score refers to the window size from 1 to K. Each time a matching true value appears, the current precision value is recorded. The average of all precision values ​​is the AP value, and the average of the AP value over the entire data set is the mAP. The specific calculation process is:

[0093]

[0094] Where POS is the window size set where the matching true value appears. For the entire generation instruction set D g ,The larger the average information retrieval indicators Recall and mAP, the more accurate the instruction set.

[0095] In some other embodiments, according to the actual scenario, indicators such as HitRate (matching hit rate) and MRR (mean reciprocal ranking) can also be used for evaluation.

[0096] Step S6, navigation instruction application: a high-quality generated instruction set is selected according to the evaluation index, which can be used to train a new visual language navigation model.

[0097] Exemplarily, this step specifically includes:

[0098] Step S61, determine the first preset condition and the second preset condition, and calculate the overall evaluation index Recall (D g ) and mAP(D g );

[0099] Step S62, determining whether the overall evaluation index of the current data set meets the first preset condition:

[0100] If yes, execute step S63; if no, adjust the hyperparameters of the instruction generation model and return to step S2;

[0101] Step S63: Calculate and determine the similarity score of the current single sample Whether the second precondition is met:

[0102] If yes, keep the sample; if no, discard the sample;

[0103] Step S64, determine whether all samples in the current data set have been traversed:

[0104] If yes, the screening is completed; if no, return to step S63.

[0105] The first preset condition is that the calculated overall evaluation index of the data set is greater than or equal to the first preset overall threshold σ recall and is greater than or equal to the second preset overall threshold σ map Specifically, the indicator Recall (D o ),mAP(D o ) as a reference, setting the overall threshold σ recall and σ map Specifically, we can choose σ recall =0.50,σ map =0.40 as a reference value. g )<σ recall ∨mAP(D g )<σ map , then the quality of the generated instruction set is too low, and steps S2-S5 should be repeated until the overall indicator of the instruction set reaches the threshold condition. During the repetition process, the model hyperparameters (number of self-attention layers, hidden state feature dimensions, activation function, etc.), training hyperparameters (learning rate, batch size, number of training steps, etc.), and generation hyperparameters (sampling method, sampling threshold, beam search, etc.) can be adjusted to achieve the purpose of optimizing the generation effect.

[0106] The second preset condition is that the calculated similarity score is greater than or equal to a preset similarity threshold. Specifically, the similarity threshold σ is set s , filter out high-quality data sets D hq , that is:

[0107]

[0108] Specifically, we can choose σ s =0.60 is taken as a reference value. Figure 7 An implementation of the above screening process is shown.

[0109] Then, according to the information recorded when collecting random paths in step S4, D hq The reverse mapping is and expert dataset D e The generated dataset can be used for data enhancement training of visual language navigation on the one hand, and for optimization of instruction generation models on the other hand, forming a semi-supervised training process.

[0110] In summary, the method provided in this embodiment can realize automatic labeling of navigation language instructions, and has the advantages of high accuracy, low cost and easy use.

[0111] Furthermore, this embodiment also provides a navigation language instruction automatic generation and evaluation device based on multimodal comparative learning, including a memory, a processor, and a program stored in the memory. When the processor executes the program, one or more steps of the aforementioned method are implemented, which will not be repeated here.

[0112] Furthermore, this embodiment also provides a storage medium on which a program is stored. When the program is executed, one or more steps of the aforementioned method are implemented, which will not be described in detail here.

[0113] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning, characterized in that: The following steps are involved: S1, obtain offline trajectory-instruction pair dataset based on expert samples; S2, based on the offline trajectory-instruction pair data set, using a preset first loss function to train a preset language instruction generation model for the navigation trajectory to obtain a trained optimal language instruction generation model, wherein the first loss function is jointly constructed based on a multimodal contrastive learning loss and an end-to-end language generation loss; S3, based on the offline trajectory-instruction pair data set, using a preset multimodal contrastive learning loss function to train a preset instruction scoring model to obtain a trained optimal instruction scoring model; S4, randomly sampling a plurality of navigation paths, inputting the navigation paths into the optimal language instruction generation model, obtaining corresponding language instructions, and further obtaining a generated trajectory-instruction pair data set; S5, inputting the generated trajectory-instruction pair data set into the optimal instruction scoring model, obtaining the similarity score of each trajectory-instruction pair in the generated trajectory-instruction pair data set, and calculating the overall evaluation index of the data set; S6, based on the overall evaluation index of the data set and the similarity score, screen out the trajectory-instruction pair data set that meets the first preset condition and the second preset condition, wherein the first preset condition is used to determine the overall quality of the data set, and the second preset condition is used to determine the quality of each trajectory-instruction pair.

2. The method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that: The S1 specifically includes: running a virtual agent in a simulation environment, batch collecting navigation visual trajectory and language instruction paired data based on expert samples, and storing them in a standardized format to obtain the offline trajectory-instruction pair data set.

3. The method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that: The S2 specifically includes: S21, constructing a language instruction generation model for navigation trajectory, wherein the language instruction generation model is a codec structure; S22, construct the first loss function: In the formula, represents the first loss function, represents the multimodal contrastive learning loss function, which is used to align the feature spaces of visual and language modalities, represents the loss function of the end-to-end language generation model, which is used to generate natural and fluent language instructions that conform to the navigation trajectory; S23, optimizing the parameters of the language instruction generation model by using a gradient descent method, determining the optimal model parameters by minimizing the value of the first loss function, and obtaining the optimal language instruction generation model.

4. The method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that: The S3 specifically includes: S31, constructing an instruction scoring model, wherein the instruction scoring model is a dual parallel encoder structure; S32, construct multimodal contrastive learning loss function; S33, optimizing the parameters of the instruction scoring model by using the gradient descent method, determining the optimal model parameters by minimizing the value of the multimodal contrastive learning loss function, and obtaining the optimal language instruction scoring model.

5. The method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that: The S4 specifically includes: S41, randomly selecting a starting point and an end point in a preset simulator, running a virtual agent on the shortest path between the two points, collecting multiple navigation visual trajectories, and obtaining a navigation visual trajectory set; S42, running the optimal language instruction generation model on the navigation visual trajectory set to obtain the language instruction corresponding to each trajectory.

6. The method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning according to claim 1, characterized in that: The S6 specifically includes: S61, determining a first preset condition and a second preset condition, and calculating an overall evaluation index of a data set of a current sample; S62, judging whether the overall evaluation index of the current data set meets the first preset condition: If yes, execute S63; if no, adjust the hyperparameters of the language instruction generation model and return to S2; S63, calculating and determining whether the similarity score of the current single sample meets the second preset condition: If yes, keep the sample; if no, discard the sample; S64, determining whether all samples in the current data set have been traversed: If yes, the screening is completed; if no, the process returns to S63.

7. The method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning according to claim 6, characterized in that: The overall evaluation indicators of the dataset include average information retrieval indicators Recall and mAP.

8. The method for automatically generating and evaluating navigation language instructions based on multimodal contrastive learning according to claim 7, characterized in that: The first preset condition is that the calculated overall evaluation index of the data set is greater than or equal to the first preset overall threshold σ recall and is greater than or equal to the second preset overall threshold σ map ; The second preset condition is that the calculated similarity score is greater than or equal to a preset similarity threshold, specifically: Where D hq represents the final selected trajectory-instruction pair dataset, σ s is the preset similarity threshold, is the similarity score of the trajectory-command pair, To navigate the visual trajectory, is the language instruction corresponding to each trajectory, the subscript i represents the sample number, D g Represents the generated trajectory-instruction pair dataset.

9. A navigation language instruction automatic generation and evaluation device based on multimodal contrastive learning, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images