Auto-driving edge scene understanding retrieval enhancement method and device and medium

Abstract

The invention relates to an automatic driving edge scene understanding retrieval enhancement method and device and a medium. The method comprises the following steps: carrying out fine tuning training on a cross-modal vector embedding model by adopting a difficult-to-negative and semi-difficult-to-negative sample sampling comparison learning method; the trained cross-modal vector embedding model is adopted to generate embedding of an image-text pair, the text coding model is adopted to generate embedding of text description, and the image-text pair and the text description are respectively stored in a multi-modal database and a text database which share the same index; inputting a real image of an automatic driving edge scene, generating embedding of an image-text pair, retrieving a closest scene image from a multi-modal database according to embedding information, and splicing the closest scene image with the automatic driving edge scene and the real image to obtain an automatic driving edge scene image; and inputting the spliced image, the text description corresponding to the image closest to the scene and a set prompt word into a visual language model to obtain edge scene understanding without illusion. Compared with the prior art, the illusion problem existing in edge scene understanding can be effectively relieved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving scene understanding, and in particular to an enhanced method, device, and medium for understanding and retrieving autonomous driving edge scenes. Background Technology

[0002] In the field of autonomous driving, Visual Language Models (VLMs) often suffer from the "illusion problem" when dealing with edge scenes. This manifests as a significant discrepancy between the model's generated results and the actual scene, leading to incorrect understandings and judgments. For example, when recognizing a partially occluded traffic sign, the model might incorrectly "fill in" a non-existent complete sign based on past experience and training data, causing the autonomous driving system to make erroneous decisions. In complex nighttime construction scenarios, the model might misjudge the shadows of construction equipment as obstacles or misinterpret the gestures of traffic controllers, issuing instructions that contradict reality. This is mainly because traditional VLMs rely on joint representations of images and text. The model extracts features from training data and fuses them for reasoning and generation. However, due to the limited and fixed nature of the dataset, the model cannot dynamically acquire new knowledge, often resulting in poor performance, especially when handling complex tasks.

[0003] A search revealed that Chinese invention patent application CN117851883A discloses a scene text detection and recognition method based on a cross-modal large language model. This method utilizes an image encoder to extract image modal feature embeddings, fuses these embeddings with corresponding ground truth labeled text feature embeddings to construct a multimodal feature sequence of image and natural language, and then inputs it into a large language model for cross-modal encoding and decoding, followed by network fine-tuning under an autoregressive mechanism. However, this approach primarily considers the detection and recognition accuracy in conventional scenes and does not optimize for the recognition illusion problem in edge scenes.

[0004] Therefore, there is an urgent need to design an enhanced scheme for understanding and retrieving edge scenes in autonomous driving, so as to effectively alleviate the illusion problem in edge scene understanding. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing an enhanced method, device, and medium for understanding and retrieving edge scenes in autonomous driving, which can effectively alleviate the illusion problem in visual language models in edge scene understanding.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, an enhanced method for understanding and retrieving edge scenes in autonomous driving is provided, comprising:

[0008] A cross-modal vector embedding model is fine-tuned using a learning method based on hard-negative and hard-negative sample sampling.

[0009] The image and text pair embeddings are generated using a fine-tuned cross-modal vector embedding model, and the text description embeddings are generated using a text encoding model. The two embeddings are stored in a multimodal database and a text database that share the same index, respectively.

[0010] The real-world image of the autonomous driving edge scene is input into the fine-tuned cross-modal vector embedding model to generate image and text pair embeddings. Based on the embedding information, the closest scene image is retrieved from the multimodal database, and the corresponding text description information is obtained based on the index.

[0011] The real-world image of the autonomous driving edge scene is stitched together with the retrieved closest scene image. The stitched image, the text description information corresponding to the retrieved closest scene image, and the set prompts are then input into the visual language model to obtain the edge scene understanding that eliminates illusions.

[0012] Preferably, the step of fine-tuning the cross-modal vector embedding model using a comparative learning method based on hard-negative and semi-hard-negative sample sampling specifically includes:

[0013] Embed the image-text pairs used for training into a cross-modal vector embedding model;

[0014] Based on the adjustable training strategy and the performance of the model's contrastive loss during training, the sample mining method is dynamically selected to mine hard-negative or semi-hard-negative samples.

[0015] Based on the contrast loss obtained during the mining process, the gradient descent algorithm is used to optimize and train the cross-modal vector embedding model; wherein, the contrast loss includes signal-noise contrast estimation loss and triplet loss.

[0016] Preferably, for hard-to-bear samples, the contrast loss considers both the signal-noise contrast estimation loss and the triplet loss; for semi-hard-to-bear samples, the contrast loss considers only the signal-noise contrast estimation loss.

[0017] Preferably, for hard-to-bear samples, the contrastive loss considers both the signal-noise contrastive estimation loss and the triplet loss, and the calculation expression is as follows:

[0018] L(S)=L con +λL triplet

[0019] In the formula: L con For signal-noise contrast estimation loss, L triplet λ represents the triplet loss; λ is the influence coefficient of the triplet.

[0020] Preferably, the signal-to-noise contrast estimation loss L con The calculation expression is:

[0021]

[0022] In the formula: L i2t For image-text matching loss; L t2i B is the text-image matching loss; Z is the batch size; i,i For logit matrix The parameter in the equation; τ is the temperature coefficient;

[0023] Let d represent the normalized image embedding and normalized text embedding of the i-th and j-th samples in the batch, respectively, where d is the embedding dimension. The calculation expression is:

[0024]

[0025] In the formula: Represents pooled image features; t i Output the original text; W img W txt b img and b txt These are the learning parameters.

[0026] Preferably, the triplet loss L triplet The calculation expression is:

[0027]

[0028] In the formula: B is the batch size; α is the boundary value, which takes the value (0, 1); Z i,i For logit matrix Parameters in; Let represent the normalized image embedding and normalized text embedding of the i-th and j-th samples in the batch, respectively, where d is the embedding dimension and τ is the temperature coefficient.

[0029] Preferably, the step of generating the image-text pair embedding using a fine-tuned cross-modal vector embedding model is as follows: for each image-text pair, the image and the corresponding text are fed into the fine-tuned cross-modal vector embedding model for processing to generate a set of hidden states, and the final image-text pair embedding is obtained by averaging the hidden states of the last layer in the sequence dimension.

[0030] Preferably, retrieving the closest scene image from the multimodal database based on the embedded information specifically involves:

[0031] The similarity between the embedding of the real-world image of the autonomous driving edge scene and each cross-modal embedding is calculated and matched. The similarity expression is as follows:

[0032] similarity=(1-α)·similarity img +α·similarity txt

[0033] In the formula: α is the boundary value, taking the value (0, 1); similarity img Image similarity; txt The text similarity is represented by cosine similarity.

[0034] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.

[0035] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) This invention utilizes the image and text pairs generated by a finely tuned cross-modal vector embedding model and the text descriptions generated by a text encoding model to embed massive edge scene images and their corresponding ground truth descriptions into a multimodal database. Subsequently, based on the embedding information of real-world edge scene images for autonomous driving, the closest scene image and corresponding text description information are retrieved from the multimodal database and input into the visual language model, effectively alleviating the illusion problem in edge scene understanding of the visual language model. Moreover, this invention does not require training a large model, but enhances the input of the large model by attaching an external knowledge base. When encountering a similar scene a second time, the corresponding scene description can be retrieved as prior information, thereby guiding the retrieval enhancement model to provide an answer that reduces the illusion. With this mechanism, the autonomous intelligence during operation can be continuously improved without integrating new data with the original training data to repeatedly train the model.

[0038] (2) This invention employs a comparative learning method based on hard-negative and semi-hard-negative sample sampling to fine-tune the cross-modal vector embedding model. During the vector embedding process, for hard-negative samples, the comparative loss considers both signal-noise comparative estimation loss and triplet loss to enhance the model's ability to distinguish subtle differences. For semi-hard-negative samples, the comparative loss only considers signal-noise comparative estimation loss to maintain training stability, thereby achieving clearer image-text alignment in the embedding space. By distinguishing between hard-negative and semi-hard-negative samples during the training phase, the cross-modal embedding model can more accurately distinguish between semantically similar but mismatched images and texts during representation learning, thereby improving the model's semantic discrimination ability when handling visual edge scenes. This mechanism effectively enhances the expressiveness and discriminativeness of the embedding space, avoiding the generation of inaccurate "illusions" when the model generates scene descriptions, and ultimately improving the accuracy of edge scene understanding.

[0039] (3) When calculating the contrast loss of hard negative samples, the present invention introduces an additional coefficient λ to measure the influence of triplet. The triplet loss penalizes the generation of high similarity non-matching pairs, encouraging the model to maintain a stricter separation between matching and non-matching samples, thereby improving the discriminative ability of the embedding. The parameter λ can be used to adjust the influence of this loss in the final loss and prevent the model from overfitting.

[0040] (4) In this invention, a massive number of edge scene images and their corresponding ground truth descriptions are embedded into a multimodal database by pre-tuning a cross-modal vector embedding model. In subsequent practical applications, it is only necessary to calculate the similarity between the embedding of the real image of the autonomous driving edge scene and each cross-modal embedding in the multimodal database to match the closest scene image, thereby improving the efficiency and accuracy of matching.

[0041] (5) When calculating the signal-noise contrast estimation loss, this invention comprehensively considers the normalization characteristics of image embedding and text embedding, the contrast information between batch samples, and the adjustment effect of temperature coefficient on similarity distribution, thereby enhancing the model's ability to perceive the similarity difference between matched and unmatched pairs, and effectively improving the discriminability and robustness of cross-modal embedding space. Attached Figure Description

[0042] Figure 1 It is a structural diagram of a vector database construction and retrieval enhancement generation framework for mitigating visual language model illusions;

[0043] Figure 2 This is a diagram of an embedded workflow;

[0044] Figure 3 This is a schematic diagram of the retrieval workflow;

[0045] Figure 4 This is a schematic diagram showing the test results of selecting an irregularly shaped traffic light edge scene. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] Example

[0048] like Figure 1 As shown, this embodiment provides an enhanced method for understanding and retrieving edge scenes in autonomous driving based on a visual language model. The method includes the following steps:

[0049] Step 1: Training based on comparative learning of hard-to-negative / semi-hard-to-negative sample sampling.

[0050] To achieve better alignment between different modalities during the embedding process, a contrastive learning approach was employed for the BLIP embedding model. The training process implemented a multimodal contrastive learning algorithm involving image and text embeddings. The algorithm aims to learn a joint representation space where semantically similar image-text pairs are grouped close together, while semantically dissimilar pairs are grouped far apart. This is achieved through a contrastive loss function that maximizes the similarity between matching pairs while minimizing the similarity between non-matching pairs.

[0051] Similarity is defined as cosine similarity:

[0052]

[0053] Among them, ||v i ||and||t j || denotes the Euclidean norm of a vector.

[0054] The training algorithm consists of four steps: 1) Data embedding, using the BLIP model to embed image-text pairs for training; 2) Negative sample mining, mining hard / semi-hard negative samples according to an adjustable training strategy; 3) Loss calculation; 4) Optimization calculation, performing gradient descent optimization on the model.

[0055] The core of this algorithm is the contrastive loss and the triplet loss. The loss function is calculated as follows:

[0056] L(S)=L con +λL triplet

[0057] In the formula: L con For the contrast loss based on infoNCE, Ltriplet The triplet loss is denoted by λ, which is a coefficient measuring the influence of the triplet and ranges from 0 to 1. The addition of the triplet loss can promote a more rigorous separation between matching and unmatched embeddings, ultimately resulting in a more effective embedding. In this embodiment, λ is set to 0.1.

[0058] Contrast loss based on infoNCE L con The calculation expression is:

[0059]

[0060] Image-text matching L i2t Matching text images L t2i The calculation expression is:

[0061]

[0062] In the formula: Z is the logit matrix; B is the batch size; Let represent the normalized image and text embeddings of the i-th and j-th samples in the batch, respectively, where d is a common embedding dimension; τ is a temperature coefficient used to adjust the smoothness of the logits distribution in the contrastive loss function, controlling the model's sensitivity to the difference in similarity between positive and negative samples. A smaller temperature coefficient τ can amplify the relative distance between positive and negative samples, which helps to improve the model's ability to distinguish between matching and non-matching samples.

[0063] Normalized image and text embedding The calculation expression is:

[0064]

[0065] In the formula: Represents pooled image features; t i Output the original text; W img W txt b img and b txt These are the learning parameters.

[0066] Triple loss L triplet The calculation expression is:

[0067]

[0068] Where: the triplet loss L triplet The loss calculation for hard-to-bear samples is used, while for semi-hard-to-bear samples, only the contrastive loss L is used. con As part of loss calculation.

[0069] The training process and algorithm aim to fine-tune the BLIP model using a cross-modal contrastive learning approach, optimizing the model by training new weights. The goal is to align the image embeddings generated by the BLIP model with the text embeddings generated by the bge-base-en-v1.5 model and embed them into a shared embedding space. This alignment is achieved by optimizing a contrastive loss function that encourages high similarity between matching image-text pairs while reducing similarity between mismatched pairs.

[0070] Specifically, this training process fine-tunes the BLIP model to enhance its ability to generate image embeddings aligned with corresponding text embeddings, thereby improving the model's ability to capture semantic relationships between images and text descriptions. After training, the fine-tuned BLIP model can be more effectively applied to image-text matching tasks, such as image retrieval, text-based image search, and other cross-modal applications.

[0071] Step 2: Embed the workflow. The workflow for embedding a workflow is as follows: Figure 2 As shown, this method employs two types of models: a multimodal encoding model (BLIP) for generating image-text representations and a text encoding model (bge-base-en-v1.5) for encoding text descriptions.

[0072] The core idea of ​​this process is to generate cross-modal embeddings that collectively represent information about images and text within a shared embedding space. For each image-text pair, the image is first processed by the BLIP processor and then input into the model along with the corresponding text. The model generates a set of hidden states and obtains the final embedding vector by averaging the hidden states of the last layer along the sequence dimension. The image and text are encoded separately, and their respective vectors are concatenated to form a unified vector space embedding representing the image content and its corresponding text description. The text description is independently encoded using the bge-base-en-v1.5 model, which is specifically designed for text encoding and retrieval tasks.

[0073] The generated image-text pair embeddings (cross-modal embeddings) and text description embeddings (unimodal text embeddings) are stored in two separate databases that share the same indexes for easy retrieval and comparison of embeddings based on content. This ensures that the embeddings can be reloaded and used in subsequent RAG tasks.

[0074] Step 3: Retrieve Workflow. The workflow for retrieving the workflow is as follows: Figure 3 As shown, new images are processed, and their embeddings are extracted using a trained BLIP processor. The core of the retrieval process is comparing the embeddings of the new image with those in a cross-modal database.

[0075] In this embodiment, cosine similarity is used to measure the similarity between two vectors. Matching is performed by calculating the similarity between the new image embedding and each cross-modal embedding (including image and text embeddings). The similarity is weighted using a parameter α, which determines the importance of text-image similarity.

[0076] The retrieved similarity is a weighted sum of the image and text similarity scores, calculated as follows:

[0077] similarity=(1-α)·similarity img +α·similarity txt

[0078] The embedding with the highest similarity is selected as the most relevant match, and the corresponding image and text indices are recorded. After obtaining the retrieved image and text indices, the corresponding images and text can be retrieved again. This pipeline demonstrates how to leverage Visual Language Models (VLMs) for cross-modal retrieval tasks, combining image and text understanding within a unified embedding space.

[0079] Step 4: Generate workflow. For workflow generation, please refer to... Figure 1 The corresponding part of this approach, using retrieved images and text as prior information, can effectively guide any Visual Language Model (VLM) to generate descriptions of new images. When a new edge scene image triggers a query, the most similar image and its corresponding text description are retrieved from the database according to the two-step process described above. The retrieved image is concatenated with the new image and provided as input to the VLM, while the retrieved text description serves as part of a prompt, combined with another pre-designed prompt to guide the VLM's generation task. This method leverages the similarity information between images and text, effectively enhancing the VLM's performance in generating descriptions of new images.

[0080] Step 5: Experimental Analysis. The experimental analysis results are shown in Table 1 below. Experiments were conducted on eleven baseline models. A dataset of 1057 images from the CODA dataset was selected as the test set. Scene descriptions automatically annotated by GPT-4o were used as ground truth, and cosine similarity, as well as F1 score, precision, and recall from the ROUGE-L metric, were used as evaluation metrics.

[0081] Table 1

[0082]

[0083]

[0084] Intuitively, the closer the generated text is to the reference text, the lower the degree of hallucination. As mentioned earlier, cosine similarity is the most direct indicator for evaluating the similarity between generated and reference texts. Therefore, the cosine similarity between all generated answers and reference texts in the test set is calculated. The text encoding model used is bge-base-en-v1.5. Finally, the arithmetic mean of all cosine similarities is used as an indicator of the degree of hallucination relief.

[0085] Furthermore, the ROUGE-L metric is primarily used to evaluate the generated descriptions holistically. More specific metrics within ROUGE-L include precision (P), recall (R), and F1 score (F1). In the context of ROUGE-L, precision is defined as the ratio of the length of the longest common subsequence (LCS) between the generated and reference texts to the length of the generated text. It measures the proportion of content in the generated text that matches the reference text.

[0086] Overall, models employing Retrieval Augmentation (RAG) consistently exhibited higher cosine similarity scores compared to models without this technique, indicating improved semantic alignment with the reference text. The cosine similarity scores typically improved by 0.02 to 0.04, with InternVL2-40B achieving the highest score (0.7146) with RAG, compared to 0.7024 without it. Similarly, the F1 score, balancing precision and recall, also showed significant improvement with RAG enabled. For example, LLaVA-v1.6-7B achieved an F1 score of 0.2608 with RAG, far exceeding the 0.1776 without it.

[0087] Precision also significantly improved when using RAG, meaning a reduction in hallucination-related content (often characterized by inaccuracy or irrelevant information). For example, the precision of Volcano-LLaVA-13B jumped from 0.1811 without RAG to 0.2175 with RAG. This improvement reflects better lexical accuracy, as the model generates more precise and relevant responses. Similarly, recall of the Visual Language Model (VLM) also significantly improved when using RAG. This indicates that the generated text better covers the content of the reference text, ensuring that key traffic participants are not overlooked in extreme situations, thus enhancing safety.

[0088] Step 6: Real-world edge scene testing. Select an irregularly shaped traffic light edge scene at an intersection in a certain city as the test scenario, such as... Figure 4As shown, many human drivers would mistake this for a stop signal. Because this scenario wasn't initially stored in the database, even with retrieval-enhanced RAG generation, LLaVA's description was misleading. Manual correction was then performed, adding the scenario and the corrected description to the database. Upon encountering the same or similar scenario a second time, the corresponding scenario description could be retrieved as prior information, guiding LLaVA to provide a less misleading answer with high alignment to the description given by GPT-4o. This mechanism enables continuous improvement of the system's autonomous intelligence during operation without needing to integrate new data with the original training data for repeated model training.

[0089] The processing unit performs the various methods and processes described above. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the methods by any other suitable means (e.g., by means of firmware).

[0090] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0091] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An enhanced method for understanding and retrieving edge scenes in autonomous driving, characterized in that, include: A cross-modal vector embedding model is fine-tuned using a learning method based on hard-negative and hard-negative sample sampling. The image and text pair embeddings are generated using a fine-tuned cross-modal vector embedding model, and the text description embeddings are generated using a text encoding model. The two embeddings are stored in a multimodal database and a text database that share the same index, respectively. The real-world image of the autonomous driving edge scene is input into the fine-tuned cross-modal vector embedding model to generate image and text pair embeddings. Based on the embedding information, the closest scene image is retrieved from the multimodal database, and the corresponding text description information is obtained based on the index. The real-world image of the autonomous driving edge scene is stitched together with the retrieved closest scene image. The stitched image, the text description information corresponding to the retrieved closest scene image, and the set prompts are then input into the visual language model to obtain the edge scene understanding that eliminates illusions.

2. The method for enhancing understanding and retrieval of autonomous driving edge scenes according to claim 1, characterized in that, The method of fine-tuning the cross-modal vector embedding model using a comparative learning approach based on hard-negative and semi-hard-negative sample sampling specifically includes: Embed the image-text pairs used for training into a cross-modal vector embedding model; Based on the adjustable training strategy and the performance of the model's contrastive loss during training, the sample mining method is dynamically selected to mine hard-negative or semi-hard-negative samples. Based on the contrast loss obtained during the mining process, the gradient descent algorithm is used to optimize and train the cross-modal vector embedding model; wherein, the contrast loss includes signal-noise contrast estimation loss and triplet loss.

3. The method for enhancing understanding and retrieval of autonomous driving edge scenes according to claim 2, characterized in that, For hard-to-bear samples, the contrast loss considers both the signal-noise contrast estimation loss and the triplet loss; for semi-hard-to-bear samples, the contrast loss considers only the signal-noise contrast estimation loss.

4. The method for enhancing understanding and retrieval of autonomous driving edge scenes according to claim 3, characterized in that, For hard-to-bear samples, the contrastive loss considers both the signal-noise contrastive estimation loss and the triplet loss, and is calculated as follows: L(S)=L con +λL triplet In the formula: L con For signal-noise contrast estimation loss, L triplet λ represents the triplet loss; λ is the influence coefficient of the triplet.

5. The method for enhancing understanding and retrieval of autonomous driving edge scenes according to claim 3, characterized in that, The signal-noise comparison estimation loss L con The calculation expression is: In the formula: L i2t For image-text matching loss; L t2i B is the text-image matching loss; Z is the batch size; i,i For logit matrix The parameter in the equation; τ is the temperature coefficient; Let d represent the normalized image embedding and normalized text embedding of the i-th and j-th samples in the batch, respectively, where d is the embedding dimension. The calculation expression is: In the formula: Represents pooled image features; t i Output the original text; W img W txt b img and b txt These are the learning parameters.

6. The method for enhancing understanding and retrieval of autonomous driving edge scenes according to claim 3, characterized in that, The triplet loss L triplet The calculation expression is: In the formula: B is the batch size; α is the boundary value, which takes the value (0, 1); Z i,i For logit matrix Parameters in; Let represent the normalized image embedding and normalized text embedding of the i-th and j-th samples in the batch, respectively, where d is the embedding dimension and τ is the temperature coefficient.

7. The method for enhancing understanding and retrieval of autonomous driving edge scenes according to claim 1, characterized in that, The method of generating image-text pair embeddings using a fine-tuned cross-modal vector embedding model is as follows: for each image-text pair, the image and the corresponding text are fed into the fine-tuned cross-modal vector embedding model for processing to generate a set of hidden states. The final image-text pair embedding is obtained by averaging the hidden states of the last layer in the sequence dimension.

8. The method for enhancing understanding and retrieval of autonomous driving edge scenes according to claim 1, characterized in that, The step of retrieving the closest scene image from the multimodal database based on the embedded information specifically involves: The similarity between the embedding of the real-world image of the autonomous driving edge scene and each cross-modal embedding is calculated and matched. The similarity expression is as follows: similarity=(1-α)·similarity img +α·similarity txt In the formula: α is the boundary value, taking the value (0, 1); similarity img Image similarity; txt Let be the text similarity, where the similarity is represented by cosine similarity.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images