Autonomous driving target detection method and device for edge cases using multimodal large language model

Abstract

The present application provides an autonomous driving target detection method and device for edge cases using a multimodal large language model, which relates to the field of target detection technology, constructs an edge case image-text pair dataset, constructs a first question prompt text based on a text description of a first edge case image, inputs the first edge case image and the first question prompt text into the multimodal large language model, obtains an image and an answer text marking the target position and name, fine-tunes the multimodal large language model based on the first edge case image and the text description corresponding to the edge case image and the output result of the multimodal large language model, obtains a trained multimodal large language model, inputs the image to be detected into the trained multimodal large language model, and obtains a target detection result. Fine-tuning the multimodal large language model by using the constructed edge case image-text pair dataset significantly improves its detection accuracy in target detection.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to an autonomous driving target detection method and device for edge cases using a multimodal large language model. Background Art

[0002] Self-driving cars need to have a keen sense of their surroundings to facilitate route planning and behavioral decisions. Currently proposed deep learning-based object detection models, such as the YOLO (You Only Look Once) series of models, can effectively detect common objects on the road by processing video clips from the camera to identify surrounding objects.

[0003] Although object detection models have achieved excellent performance in detecting training datasets that identify objects, vehicles will inevitably encounter complex and rare scenes on real roads, such as deep potholes caused by road damage or traffic guidance signs temporarily erected due to special events. These rare objects or scenes are called edge cases, which, although uncommon, can have a significant impact on traffic safety when they occur.

[0004] Due to the scarcity of training data for edge cases, if the target detection model learns edge cases zero or few times, the effectiveness and robustness of the target detection model will face major challenges, resulting in a decrease in detection accuracy. Summary of the invention

[0005] The present application provides an autonomous driving target detection method and device for edge cases using a multimodal large language model to solve the above-mentioned problems.

[0006] In a first aspect of an embodiment of the present application, a method for detecting an autonomous driving target for edge cases using a multimodal large language model is provided, the method comprising:

[0007] Constructing an edge case image-text pair dataset, wherein each edge case image-text pair in the edge case image-text pair dataset includes: an edge case image and a text description corresponding to the edge case image, wherein the text description includes a name of a target in the edge case image and a description of a location of the target, and the category to which the target belongs includes at least: traffic facilities, obstacles, and unusual objects;

[0008] Constructing a first question prompt text based on a text description of a first edge case image in the data set by the edge case image text;

[0009] Inputting the first edge case image and the first question prompt text into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, and an answer text corresponding to the first question prompt text, wherein the answer text is used to describe the name and location of each target in the first edge case image;

[0010] Based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and name of the target, and the answer text corresponding to the first question prompt text, fine-tune the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model to obtain a trained multimodal large language model, wherein the modal alignment layer is used to map the features of the first edge case image to the text embedding space that can be processed by the pre-trained large language model, and the bounding box decoder is used to obtain the location information of the target in the first edge case image from the answer text;

[0011] The image to be detected is input into the trained multimodal large language model to obtain the target detection result of the image to be detected.

[0012] In an optional embodiment of the present application, the first edge case image and the first question prompt text are input into a multimodal large language model to obtain an answer text corresponding to the first question prompt text, including:

[0013] Inputting the first edge case image into the image encoder of the multimodal large language model to obtain an image encoding result, and inputting the image encoding result into the modality alignment layer of the multimodal large language model to obtain an image encoding result aligned to the text modality;

[0014] Inputting the first question prompt text and the image encoding result aligned to the text modality into the text encoder of the multimodal large language model to obtain a multimodal text encoding result;

[0015] Inputting the multimodal text encoding result into a pre-trained large language model of the multimodal large language model to obtain a text representation of a target detection result, wherein the text representation of the target detection result is a token sequence including the name and location information of the target;

[0016] The target detection result text representation is input into the text decoder of the multimodal large language model to obtain the answer text corresponding to the first question prompt text.

[0017] In an optional embodiment of the present application, the first edge case image and the first question prompt text are input into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, including:

[0018] The answer text corresponding to the first question prompt text is input into the bounding box decoder of the multimodal large language model to obtain the first edge case image marked with the location box and name of the target.

[0019] In an optional embodiment of the present application, constructing an edge case image-text pair dataset includes:

[0020] Obtain an edge case image dataset, wherein each object in the edge case image dataset carries pre-labeled bounding box coordinates and a category, wherein the category is one of a plurality of predefined categories of edge cases to which the corresponding object belongs, wherein the predefined categories include at least: traffic facilities, obstacles, and uncommon objects;

[0021] A text description is added to each object in the edge case image dataset to obtain the edge case image-text pair dataset.

[0022] In an optional embodiment of the present application, the multimodal large language model is obtained by fine-tuning the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model through text loss and position loss, the text loss is calculated by the answer text corresponding to the first question prompt text and the text description of the first edge case image, and the position loss is calculated by the position box of the target in the first edge case image marked with the position box and name of the target output by the multimodal large language model and the position of the target in the first edge case image;

[0023] The total loss function used by the multimodal large language model during training The calculation formula is as follows:

[0024] ,

[0025] In the formula, For text loss, is the position loss, is a weighting parameter used to balance the weights of text loss and position loss.

[0026] In an optional embodiment of the present application, the first question prompt text includes a question text, a format description text, and a command text; constructing the first question prompt text based on the text description of the first edge case image in the data set by the edge case image text includes:

[0027] constructing the question text based on the name of the target in the dataset of the first edge case image and the edge case image text;

[0028] Constructing a format description text for describing the location information of the target in the first edge case image, wherein the question text and the format description text are used to trigger the multimodal large language model to output an answer text corresponding to the first question prompt text;

[0029] A command text for locating a target and a corresponding location box in the first edge case image is constructed, wherein the command text is used to trigger the multimodal large language model to output the first edge case image marked with the location box and the name of the target.

[0030] In a second aspect of an embodiment of the present application, a device for detecting an autonomous driving target for edge cases using a multimodal large language model is provided, the device comprising:

[0031] A data set construction module is used to construct an edge case image-text pair data set, wherein each edge case image-text pair in the edge case image-text pair data set includes: an edge case image and a text description corresponding to the edge case image, wherein the text description includes the name of a target in the edge case image and a description of the location of the target, and the category to which the target belongs includes at least: traffic facilities, obstacles, and uncommon objects;

[0032] a prompt text construction module, configured to construct a first question prompt text based on a text description of a first edge case image in a data set by the edge case image text;

[0033] An input module, configured to input the first edge case image and the first question prompt text into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, and an answer text corresponding to the first question prompt text, wherein the answer text is used to describe the name and location of each target in the first edge case image;

[0034] a fine-tuning module, configured to fine-tune the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and the name of the target, and the answer text corresponding to the first question prompt text, so as to obtain a trained multimodal large language model, wherein the modal alignment layer is configured to map the features of the first edge case image to the text embedding space that can be processed by the pre-trained large language model, and the bounding box decoder is configured to obtain the location information of the target in the first edge case image from the answer text;

[0035] The detection module is used to input the image to be detected into the trained multimodal large language model to obtain the target detection result of the image to be detected.

[0036] In an optional embodiment of the present application, the input module includes:

[0037] An image encoding submodule, configured to input the first edge case image into an image encoder of the multimodal large language model to obtain an image encoding result, and input the image encoding result into a modality alignment layer of the multimodal large language model to obtain an image encoding result aligned to a text modality;

[0038] A text encoding submodule, used for inputting the first question prompt text and the image encoding result aligned to the text modality into the text encoder of the multimodal large language model to obtain a multimodal text encoding result;

[0039] A text representation acquisition submodule, used for inputting the multimodal text encoding result into a pre-trained large language model of the multimodal large language model to obtain a text representation of a target detection result, wherein the text representation of the target detection result is a token sequence including the name and location information of the target;

[0040] The text decoding submodule is used to input the text representation of the target detection result into the text decoder of the multimodal large language model to obtain the answer text corresponding to the first question prompt text.

[0041] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the autonomous driving target detection method for edge cases using a multimodal large language model as described in any one of the first aspects above.

[0042] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is proposed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method for autonomous driving target detection for edge cases using a multimodal large language model as described in any one of the first aspects above is implemented.

[0043] The present application includes the following advantages: In a first aspect of an embodiment of the present application, a method and apparatus for autonomous driving target detection for edge cases using a multimodal large language model are proposed, and an edge case image-text pair dataset is constructed. Each edge case image-text pair in the edge case image-text pair dataset includes: an edge case image and a text description corresponding to the edge case image, and the text description includes the name of the target in the edge case image and a description of the location of the target, and the categories to which the target belongs include at least: traffic facilities, obstacles, and uncommon objects; based on the text description of the first edge case image in the edge case image-text pair dataset, a first question prompt text is constructed; the first edge case image and the first question prompt text are input into the multimodal large language model to obtain a first edge case image marked with a location box and a name of the target, and the first question prompt text The corresponding answer text is used to describe the name and position of each target in the first edge case image; based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and name of the target, and the answer text corresponding to the first question prompt text, the modal alignment layer, the pre-trained large language model and the bounding box decoder in the multimodal large language model are fine-tuned to obtain a trained multimodal large language model, the modal alignment layer is used to map the features of the first edge case image to the text embedding space that the pre-trained large language model can process, and the bounding box decoder is used to obtain the position information of the target in the first edge case image from the answer text; the image to be detected is input into the trained multimodal large language model to obtain the target detection result of the image to be detected. By making full use of the prior knowledge and reasoning ability of the large language model, and fine-tuning the multimodal large language model through the constructed edge case image text data set, the detection accuracy of the edge case is significantly improved in the process of target detection, and the safety and reliability of the autonomous driving system are further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 It is a schematic diagram of the steps of a method for detecting an autonomous driving target for edge cases using a multimodal large language model provided in an embodiment of the present application;

[0046] Figure 2It is a flowchart of an autonomous driving target detection method for edge cases using a multimodal large language model proposed in an embodiment of the present application;

[0047] Figure 3 It is an architecture diagram of an autonomous driving target detection device for edge cases using a multimodal large language model provided in an embodiment of the present application;

[0048] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] Self-driving cars need to have a keen sense of their surroundings to facilitate route planning and behavioral decisions. Currently proposed deep learning-based object detection models, such as the YOLO (You Only Look Once) series of models, can effectively detect common objects on the road by processing video clips from the camera to identify surrounding objects.

[0051] Although object detection models have achieved excellent performance in detecting training datasets that identify objects, vehicles will inevitably encounter complex and rare scenes on real roads, such as deep potholes caused by road damage or traffic guidance signs temporarily erected due to special events. These rare objects or scenes are called edge cases, which, although uncommon, can have a significant impact on traffic safety when they occur.

[0052] Due to the scarcity of training data for edge cases, if the target detection model learns edge cases zero or few times, the effectiveness and robustness of the target detection model will face major challenges, resulting in a decrease in detection accuracy.

[0053] The target detection model is trained by a training data set of a determined target (the training data set of a determined target can be considered as a closed vocabulary data set). The target detection model can detect all objects that appear in the training data set. For example, in the field of target detection in autonomous driving, common targets include vehicles, traffic lights, etc. The target detection model trained with a closed vocabulary data set of a determined target (such as vehicles, traffic lights) can only detect targets (vehicles, traffic lights). An edge case is a rare object or scene, which can be considered as an open vocabulary target outside the training data set of the determined target. For example, in addition to vehicles and traffic lights, the data set to be detected also contains construction signs, road rollers, rescue vehicles, art exhibits, wild animals, natural objects formed by special weather, etc. Considering that a large language model (LLM) can accumulate rich prior knowledge through pre-training, this application detects open vocabulary targets in edge cases by using a large language model.

[0054] Based on this, in a first aspect of an embodiment of the present application, a method for detecting an autonomous driving target for edge cases using a multimodal large language model is provided. Figure 1 , Figure 1 1 is a schematic diagram of a step flow of a method for detecting an autonomous driving target for edge cases using a multimodal large language model provided in an embodiment of the present application, the method comprising the following steps:

[0055] Step 101: construct an edge case image-text pair dataset, wherein each edge case image-text pair in the edge case image-text pair dataset includes: an edge case image and a text description corresponding to the edge case image, wherein the text description includes the name of a target in the edge case image and a description of the location of the target, and the category to which the target belongs includes at least: traffic facilities, obstacles, and unusual objects;

[0056] Step 102: construct a first question prompt text based on the text description of the first edge case image in the data set by the edge case image text;

[0057] Step 103: inputting the first edge case image and the first question prompt text into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, and an answer text corresponding to the first question prompt text, wherein the answer text is used to describe the name and location of each target in the first edge case image;

[0058] Step 104: based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and name of the target, and the answer text corresponding to the first question prompt text, fine-tune the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model to obtain a trained multimodal large language model, wherein the modal alignment layer is used to map the features of the first edge case image to the text embedding space that can be processed by the pre-trained large language model, and the bounding box decoder is used to obtain the location information of the target in the first edge case image from the answer text;

[0059] Step 105: input the image to be detected into the trained multimodal large language model to obtain the target detection result of the image to be detected.

[0060] In order to clearly illustrate the proposed method of autonomous driving target detection for edge cases using a multimodal large language model, Figure 2 For example, Figure 2 It is a flow chart of an autonomous driving target detection method for edge cases using a multimodal large language model proposed in an embodiment of the present application.

[0061] When step 101 is specifically implemented, in order to enable the large language model to process the input multimodal data (including data of multiple different modes of information such as images and texts), a multimodal dataset for training is first constructed. The multimodal dataset is used to train the multimodal large language model proposed in the embodiment of the present application for edge cases and scenarios. The multimodal dataset includes at least text and edge case images corresponding to the text description. Specifically, a data set of edge case image-text pairs is constructed, which contains multiple edge case image-text pairs. Each edge case image-text pair includes an edge case image and a text description corresponding to the edge case image. The text description includes at least the name of the target in the edge case image and a description of the location of the target. The location description of the target refers to the bounding box of the target. For example, the text description may be: In Figure A, there is a traffic signal and an obstacle on the road. The location boxes corresponding to the traffic signal and the obstacle are [xmin1, ymin1, xmax1, ymax1] and [xmin2, ymin2, xmax2, ymax2], respectively, wherein any vertex of the location box is taken as the origin of the coordinate axis, xmin and ymin represent the minimum values ​​of the location box in the x-axis direction and the y-axis direction, respectively, and xmax and ymax represent the maximum values ​​of the location box in the x-axis direction and the y-axis direction, respectively. Since the edge case image-text pair dataset constructed by the embodiments of the present application is aimed at detecting edge cases in edge case images, the categories to which the targets contained in each edge case image belong include at least edge cases such as traffic facilities, obstacles and uncommon objects, and may also include pedestrians, bicycles, vehicles, animals, etc.

[0062] Optionally, the above text description may also include the number of the edge case image, the size information of the edge case image, the question text regarding whether the edge case is included in the edge case image, the answer reasoning text of the question text, the answer text for the question text and the reasoning text, and the indication text of the location box output for each edge case when multiple edge cases are included in the edge case image. Taking Figure A as an example, the number of the edge case image is Figure A, the size of the edge case image is: height 720, width 1281, and the question text regarding whether the edge case is included in the edge case image is: whether Figure A contains an edge case. The answer reasoning text for the question text is: In Figure A, there is a traffic signal and an obstacle on the road. Therefore, the answer is yes. The answer text for the question text and the reasoning text is: yes; the location box of each edge case is: [xmin1, ymin1, xmax1, ymax1], [xmin2, ymin2, xmax2, ymax2], and the output sequence is: [0], [1].

[0063] In order to obtain an edge case image-text pair dataset, in an optional embodiment of the present application, the process of constructing the edge case image-text pair dataset is as follows: First, an edge case image dataset is obtained, in which each target contained in the edge case image dataset carries pre-marked bounding box coordinates and a category, which is one of the predefined categories of multiple edge cases to which the corresponding target belongs, and the predefined categories at least include traffic facilities (such as traffic cones, roadblocks, traffic signs, etc.), obstacles (including various obstacles on the road, such as fallen trees and houses, abandoned vehicles, flooded roads, etc.) and uncommon objects (including objects that do not belong to the above categories, such as handbags, suitcases, etc., which may be discarded or forgotten) and other categories to which edge cases belong. The predefined categories also include: pedestrians (including walking or running pedestrians), cyclists (including people riding bicycles, electric vehicles or motorcycles), vehicles (including various types of cars, trucks, motorcycles), animals (such as cats, dogs, etc., which may appear on the road, depending on different environments). In the embodiment of the present application, the edge case image dataset is preferably the CODA (Corner Case Dataset for Autonomous Driving) dataset, which contains 9768 2D images of more than 1 million scenes from three autonomous driving benchmark datasets KITTI, nuScences and ONCE. Rare objects in the CODA dataset can greatly affect the average recall rate of the pre-trained deep learning target detection model when directly applied, and can improve the accuracy of detecting rare objects.

[0064] Then, we add text descriptions to each object in the edge case image dataset to obtain the edge case image text pair dataset. The construction of the edge case image text pair dataset aims to achieve semantic alignment between text and images so that the multimodal large language model can understand the text description and accurately locate the relevant objects in the image.

[0065] When specifically implementing step 102, in order to enable the multimodal large language model to better follow instructions and perform corresponding tasks after fine-tuning the edge case image text pair dataset, the embodiment of the present application proposes a question prompt text dataset, which contains question prompt texts corresponding to all edge case images in the edge case image text pair dataset, and the question prompt texts at least include natural language instructions, optional contextual input information, and expected output information. By fine-tuning the multimodal large language model based on the edge case image text pair dataset on the question prompt text dataset, the fine-tuned multimodal large language model can better follow instructions and perform corresponding tasks. Among them, the construction of the question prompt text dataset is based on the text description of all edge case images contained in the edge case image text pair dataset. Taking the first edge case image in the edge case image text pair dataset (the first edge case image can be any edge case image in the edge case image text pair dataset) as an example, a corresponding first question prompt text is generated for the text description of the first edge case image. That is, if the edge case image-text pair dataset contains n edge case images, n question prompt texts are generated based on the text descriptions of the n edge case images, and the question prompt text dataset contains n question prompt texts, where n is an integer greater than 1. In the subsequent description, the process of fine-tuning the multimodal large language model using the first question prompt text and the first edge case image is the same as the process of fine-tuning the multimodal large language model using the remaining edge case images and the corresponding question prompt texts in the edge case image-text pair dataset. The following description is explained using the process of fine-tuning the multimodal large language model using the first question prompt text and the first edge case image.

[0066] In an optional embodiment of the present application, the first question prompt text includes a question text, a format description text and a command text. The first question prompt text is constructed by using the text description of the first edge case image in the data set by the edge case image text, which specifically includes the following process: First, based on the names of the targets in the data set of the first edge case image and the edge case image text, the question text is constructed. For the first edge case image, for example, the names of the targets contained in the first edge case image include traffic cones and road posts. The question text can be: the location of the target in <first edge case image>; then, a format description text is constructed to describe the location information of the target in the first edge case image. The format text is: the target is in [xmin, ymin, xmax, ymax]. The question text and the format description text are used together to trigger the multimodal large language model to output the answer text corresponding to the first question prompt text; for example, the answer text can be: a traffic cone is in [1020, 561, 1081, 690], and a road post is in [1136, 456, 17, 63]. Finally, a command text is constructed for locating the target in the first edge case image and the corresponding location box. For example, the command text can be: <name of the target> at the location of <first edge case image>, and the two are kept consistent with the image. Therefore, the command text can trigger the multimodal large language model to output the first edge case image marked with the location box and name of the target. For example, the first edge case image marked with traffic cones [1020, 561, 1081, 690] and road posts [1136, 456, 17, 63] can be obtained.

[0067] When step 103 is specifically implemented, after obtaining the first question prompt text, the first question prompt text and the first edge case image are input into the multimodal large language model, and the multimodal large language model outputs the first edge case image marked with a location box and name of the target, and the multimodal large language model outputs an answer text corresponding to the first question prompt text, and the answer text is used to describe the name and location of each target in the first edge case image.

[0068] In an optional embodiment of the present application, the first edge case image and the first question prompt text are input into a multimodal large language model to obtain an answer text corresponding to the first question prompt text, which specifically includes the following processes: first, the first edge case image is input into an image encoder of the multimodal large language model to obtain an image encoding result, and the image encoding result is input into a modal alignment layer of the multimodal large language model to obtain an image encoding result aligned to the text modality; then, the first question prompt text and the image encoding result aligned to the text modality are input into a text encoder of the multimodal large language model to obtain a multimodal text encoding result; subsequently, the multimodal text encoding result is input into a pre-trained large language model of the multimodal large language model to obtain a text representation of a target detection result, wherein the text representation of the target detection result is a token sequence containing the name and location information of the target. In the embodiment of the present application, the pre-trained large language model is pre-trained on a large-scale data set, and the large language model has accumulated rich prior knowledge through the pre-training; finally, the text representation of the target detection result is input into a text decoder of the multimodal large language model to obtain an answer text corresponding to the first question prompt text.

[0069] In an optional embodiment of the present application, a first edge case image and a first question prompt text are input into a multimodal large language model to obtain a first edge case image marked with a location box and a name of the target, which specifically includes the following process: an answer text corresponding to the first question prompt text is input into a bounding box decoder of the multimodal large language model to obtain a first edge case image marked with a location box and a name of the target, the first edge case image marked with a location box and a name of the target and the description and location box coordinates of the target in the answer text are the same, and accurate target detection and positioning of the multimodal large language model is achieved based on this.

[0070] When step 104 is specifically implemented, based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and name of the target, and the answer text corresponding to the first question prompt text, the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model are fine-tuned to obtain a trained multimodal large language model, wherein during the fine-tuning process, the image encoder, the text encoder, and the text decoder in the multimodal large language model all freeze the parameters. The multimodal large language model can detect the target in the image according to the given question prompt text and output the bounding box corresponding to the target. During the fine-tuning process, the generated answer text and the text description of the edge case image are compared, and the bounding box information is extracted using a regular expression and compared with the actual position of the target in the edge case image, thereby enhancing the visual stress and performance of the multimodal large language model on specific tasks, and can complete the edge case detection of open vocabulary without a large amount of retraining process or architecture modification. After fine-tuning the multimodal large language model using the constructed edge case image-text pair dataset, the ability of the multimodal large language model to accurately identify and respond to rare and complex scenarios is enhanced, improving the safety and reliability of the autonomous driving system in these scenarios.

[0071] In an optional embodiment of the present application, the modal alignment layer is used to map the features of the first edge case image to a text embedding space that can be processed by a pre-trained large language model to achieve alignment of multimodal data, and the bounding box decoder is used to obtain the position information of the target in the first edge case image from the answer text; the modal alignment layer and the bounding box decoder can adopt an MLP (Multilayer Perceptron) structure, and the head of the pre-trained large language model can be selected as a transformer-based structure, such as Vicuna-7B, to generate natural language consistent with the input image data, and promote the output ability of the multimodal large language model to respond to image data and generate context-appropriate text.

[0072] In an optional embodiment of the present application, the multimodal large language model is obtained by fine-tuning the modal alignment layer, the pre-trained large language model and the bounding box decoder in the multimodal large language model through text loss and position loss, the text loss is calculated by the answer text corresponding to the first question prompt text and the text description of the first edge case image, and the position loss is calculated by the position box of the target in the first edge case image marked with the position box and name of the target output by the multimodal large language model and the position of the target in the first edge case image; the total loss function used by the multimodal large language model during training is The calculation formula is as follows:

[0073] ,

[0074] In the formula, For text loss, is the position loss, is a weighting parameter used to balance the weights of text loss and position loss.

[0075] It is assumed that the description of the true position box of the i-th target in the first edge case image can be expressed as , the location box of the target in the first edge case image output by the multimodal large language model can be expressed as ,in, and Represent the horizontal coordinates of the center point of the real position frame and the output position frame of the i-th target, respectively. and Respectively represent the ordinates of the center points of the real position frame and the output position frame of the i-th target, and Respectively represent the width of the true position box and the output position box of the i-th target, and Represents the height of the true position box and the output position box of the i-th target, respectively, and the position loss The calculation formula can be expressed as:

[0076] ,

[0077] Sum the bounding boxes of the N objects contained in the first edge case image to obtain a smooth The loss function is calculated as follows:

[0078] ,

[0079] In the formula, , represents the difference between the true location box in the text description of the same target and the location box output by the multimodal large prediction model.

[0080] When implementing step 105, after fine-tuning the multimodal large language model using the edge case image text pair dataset constructed by this application, the image to be detected is input into the trained multimodal large language model, and the target contained in the image to be detected is detected to obtain the target detection result of the image to be detected. When encountering various complex and rare situations, by making full use of the prior knowledge and reasoning ability of the large language model, and fine-tuning the multimodal large language model through the constructed edge case image text pair dataset, the detection accuracy in target detection is significantly improved, and the safety and reliability of the autonomous driving system are further enhanced.

[0081] In a first aspect of an embodiment of the present application, a method for autonomous driving target detection for edge cases using a multimodal large language model is proposed, and an edge case image-text pair dataset is constructed. Each edge case image-text pair in the edge case image-text pair dataset includes: an edge case image and a text description corresponding to the edge case image, and the text description includes the name of the target in the edge case image and a description of the location of the target. The categories to which the target belongs include at least: traffic facilities, obstacles, and uncommon objects; based on the text description of the first edge case image in the edge case image-text pair dataset, a first question prompt text is constructed; the first edge case image and the first question prompt text are input into the multimodal large language model to obtain a first edge case image marked with a location box and a name of the target, and an answer text corresponding to the first question prompt text , the answer text is used to describe the name and position of each target in the first edge case image; based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the target location box and name, and the answer text corresponding to the first question prompt text, the modal alignment layer, the pre-trained large language model and the bounding box decoder in the multimodal large language model are fine-tuned to obtain a trained multimodal large language model, the modal alignment layer is used to map the features of the first edge case image to the text embedding space that the pre-trained large language model can process, and the bounding box decoder is used to obtain the position information of the target in the first edge case image from the answer text; the image to be detected is input into the trained multimodal large language model to obtain the target detection result of the image to be detected. By making full use of the prior knowledge and reasoning ability of the large language model, and fine-tuning the multimodal large language model through the constructed edge case image text pair data set, the detection accuracy of the large language model is significantly improved in the process of target detection in edge cases, and the safety and reliability of the autonomous driving system are further enhanced.

[0082] In a second aspect of the embodiment of the present application, an automatic driving target detection device for edge cases using a multimodal large language model is provided, see Figure 3 , Figure 3 : is an architecture diagram of an autonomous driving target detection device for edge cases using a multimodal large language model provided in an embodiment of the present application, the device comprising:

[0083] A data set construction module 301 is used to construct an edge case image-text pair data set, wherein each edge case image-text pair in the edge case image-text pair data set includes: an edge case image and a text description corresponding to the edge case image, wherein the text description includes the name of a target in the edge case image and a description of the location of the target, and the category to which the target belongs includes at least: traffic facilities, obstacles, and unusual objects;

[0084] A prompt text construction module 302 is used to construct a first question prompt text based on the text description of the first edge case image in the data set by the edge case image text;

[0085] An input module 303 is used to input the first edge case image and the first question prompt text into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, and an answer text corresponding to the first question prompt text, wherein the answer text is used to describe the name and location of each target in the first edge case image;

[0086] A fine-tuning module 304 is configured to fine-tune the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and the name of the target, and the answer text corresponding to the first question prompt text, so as to obtain a trained multimodal large language model, wherein the modal alignment layer is configured to map the features of the first edge case image to the text embedding space that can be processed by the pre-trained large language model, and the bounding box decoder is configured to obtain the location information of the target in the first edge case image from the answer text;

[0087] The detection module 305 is used to input the image to be detected into the trained multimodal large language model to obtain the target detection result of the image to be detected.

[0088] Wherein, the input module includes:

[0089] An image encoding submodule, configured to input the first edge case image into an image encoder of the multimodal large language model to obtain an image encoding result, and input the image encoding result into a modality alignment layer of the multimodal large language model to obtain an image encoding result aligned to a text modality;

[0090] A text encoding submodule, used for inputting the first question prompt text and the image encoding result aligned to the text modality into the text encoder of the multimodal large language model to obtain a multimodal text encoding result;

[0091] A text representation acquisition submodule, used for inputting the multimodal text encoding result into a pre-trained large language model of the multimodal large language model to obtain a text representation of a target detection result, wherein the text representation of the target detection result is a token sequence including the name and location information of the target;

[0092] The text decoding submodule is used to input the text representation of the target detection result into the text decoder of the multimodal large language model to obtain the answer text corresponding to the first question prompt text.

[0093] Wherein, the input module also includes:

[0094] The bounding box decoding submodule is used to input the answer text corresponding to the first question prompt text into the bounding box decoder of the multimodal large language model to obtain the first edge case image marked with the location box and name of the target.

[0095] Wherein, the data set construction module includes:

[0096] An acquisition submodule is used to obtain an edge case image dataset, each target in the edge case image dataset carries a pre-marked bounding box coordinate and a category, and the category is one of the predefined categories of multiple edge cases to which the corresponding target belongs, and the predefined categories include at least: traffic facilities, obstacles, and uncommon objects;

[0097] A submodule is constructed, which is used to add a text description to each target in the edge case image dataset to obtain the edge case image-text pair dataset.

[0098] The multimodal large language model in the fine-tuning module is obtained by fine-tuning the modal alignment layer, the pre-trained large language model and the bounding box decoder in the multimodal large language model through text loss and position loss, the text loss is calculated by the answer text corresponding to the first question prompt text and the text description of the first edge case image, and the position loss is calculated by the position box of the target in the first edge case image marked with the position box and name of the target output by the multimodal large language model and the position of the target in the first edge case image;

[0099] The total loss function used by the multimodal large language model during training The calculation formula is as follows:

[0100] ,

[0101] In the formula, For text loss, is the position loss, is a weighting parameter used to balance the weights of text loss and position loss.

[0102] The first question prompt text includes question text, format description text and command text; the prompt text construction module includes:

[0103] a question text construction submodule, configured to construct the question text based on the name of the target in the first edge case image and the edge case image text pair data set;

[0104] a format description text construction submodule, used to construct a format description text for describing the location information of the target in the first edge case image, wherein the question text and the format description text are used to trigger the multimodal large language model to output an answer text corresponding to the first question prompt text;

[0105] A command text construction submodule is used to construct a command text for locating a target and a corresponding location box in the first edge case image, wherein the command text is used to trigger the multimodal large language model to output the first edge case image marked with a location box and a name of the target.

[0106] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 4 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 4 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement an autonomous driving target detection method for edge cases using a multimodal large language model disclosed in an embodiment of the present application.

[0107] Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, an autonomous driving target detection method for edge cases using a multimodal large language model disclosed in the embodiment of the present application is implemented.

[0108] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0109] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0113] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0114] The above is a detailed introduction to the method and device for autonomous driving target detection for edge cases using a multimodal large language model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for autonomous driving target detection for edge cases using a multimodal large language model, characterized in that: The method comprises: Constructing an edge case image-text pair dataset, wherein each edge case image-text pair in the edge case image-text pair dataset includes: an edge case image and a text description corresponding to the edge case image, wherein the text description includes a name of a target in the edge case image and a description of a location of the target, and the category to which the target belongs includes at least: traffic facilities, obstacles, and unusual objects; Constructing a first question prompt text based on a text description of a first edge case image in the data set by the edge case image text; Inputting the first edge case image and the first question prompt text into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, and an answer text corresponding to the first question prompt text, wherein the answer text is used to describe the name and location of each target in the first edge case image; Based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and name of the target, and the answer text corresponding to the first question prompt text, fine-tune the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model to obtain a trained multimodal large language model, wherein the modal alignment layer is used to map the features of the first edge case image to the text embedding space that can be processed by the pre-trained large language model, and the bounding box decoder is used to obtain the location information of the target in the first edge case image from the answer text; Inputting the image to be detected into the trained multimodal large language model to obtain the target detection result of the image to be detected; The first question prompt text includes a question text, a format description text, and a command text; constructing the first question prompt text based on the text description of the first edge case image in the data set by the edge case image text includes: constructing the question text based on the name of the target in the dataset of the first edge case image and the edge case image text; Constructing a format description text for describing the location information of the target in the first edge case image, wherein the question text and the format description text are used to trigger the multimodal large language model to output an answer text corresponding to the first question prompt text; A command text for locating a target and a corresponding location box in the first edge case image is constructed, wherein the command text is used to trigger the multimodal large language model to output the first edge case image marked with the location box and the name of the target.

2. The method for autonomous driving target detection for edge cases using a multimodal large language model according to claim 1, characterized in that: Inputting the first edge case image and the first question prompt text into a multimodal large language model to obtain an answer text corresponding to the first question prompt text includes: Inputting the first edge case image into the image encoder of the multimodal large language model to obtain an image encoding result, and inputting the image encoding result into the modality alignment layer of the multimodal large language model to obtain an image encoding result aligned to the text modality; Inputting the first question prompt text and the image encoding result aligned to the text modality into the text encoder of the multimodal large language model to obtain a multimodal text encoding result; Inputting the multimodal text encoding result into a pre-trained large language model of the multimodal large language model to obtain a text representation of a target detection result, wherein the text representation of the target detection result is a token sequence including the name and location information of the target; The target detection result text representation is input into the text decoder of the multimodal large language model to obtain the answer text corresponding to the first question prompt text.

3. The method for autonomous driving target detection for edge cases using a multimodal large language model according to claim 2, characterized in that: Inputting the first edge case image and the first question prompt text into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, including: The answer text corresponding to the first question prompt text is input into the bounding box decoder of the multimodal large language model to obtain the first edge case image marked with the location box and name of the target.

4. The method for autonomous driving target detection for edge cases using a multimodal large language model according to claim 1, characterized in that: Construct edge case image-text pair dataset, including: Obtain an edge case image dataset, wherein each object in the edge case image dataset carries pre-labeled bounding box coordinates and a category, wherein the category is one of a plurality of predefined categories of edge cases to which the corresponding object belongs, wherein the predefined categories include at least: traffic facilities, obstacles, and uncommon objects; A text description is added to each object in the edge case image dataset to obtain the edge case image-text pair dataset.

5. The method for autonomous driving target detection for edge cases using a multimodal large language model according to claim 1, characterized in that: The multimodal large language model is obtained by fine-tuning the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model through text loss and position loss, wherein the text loss is calculated by the answer text corresponding to the first question prompt text and the text description of the first edge case image, and the position loss is calculated by the position box of the target in the first edge case image marked with the position box and name of the target output by the multimodal large language model and the position of the target in the first edge case image; The total loss function used by the multimodal large language model during training The calculation formula is as follows: ， In the formula, For text loss, is the position loss, is a weighting parameter used to balance the weights of text loss and position loss.

6. An autonomous driving target detection device for edge cases using a multimodal large language model, characterized in that: The device comprises: A data set construction module is used to construct an edge case image-text pair data set, wherein each edge case image-text pair in the edge case image-text pair data set includes: an edge case image and a text description corresponding to the edge case image, wherein the text description includes the name of a target in the edge case image and a description of the location of the target, and the category to which the target belongs includes at least: traffic facilities, obstacles, and uncommon objects; a prompt text construction module, configured to construct a first question prompt text based on a text description of a first edge case image in a data set by the edge case image text; An input module, configured to input the first edge case image and the first question prompt text into a multimodal large language model to obtain a first edge case image marked with a location box and a name of a target, and an answer text corresponding to the first question prompt text, wherein the answer text is used to describe the name and location of each target in the first edge case image; a fine-tuning module, configured to fine-tune the modal alignment layer, the pre-trained large language model, and the bounding box decoder in the multimodal large language model based on the first edge case image and the text description corresponding to the edge case image, the first edge case image marked with the location box and the name of the target, and the answer text corresponding to the first question prompt text, so as to obtain a trained multimodal large language model, wherein the modal alignment layer is configured to map the features of the first edge case image to the text embedding space that can be processed by the pre-trained large language model, and the bounding box decoder is configured to obtain the location information of the target in the first edge case image from the answer text; A detection module, used for inputting the image to be detected into the trained multimodal large language model to obtain the target detection result of the image to be detected; The first question prompt text includes question text, format description text and command text; the prompt text construction module includes: a question text construction submodule, configured to construct the question text based on the name of the target in the first edge case image and the edge case image text pair data set; a format description text construction submodule, used to construct a format description text for describing the location information of the target in the first edge case image, wherein the question text and the format description text are used to trigger the multimodal large language model to output an answer text corresponding to the first question prompt text; A command text construction submodule is used to construct a command text for locating a target and a corresponding location box in the first edge case image, wherein the command text is used to trigger the multimodal large language model to output the first edge case image marked with a location box and a name of the target.

7. The automatic driving target detection device for edge cases using a multimodal large language model according to claim 6, characterized in that: The input module comprises: An image encoding submodule, configured to input the first edge case image into an image encoder of the multimodal large language model to obtain an image encoding result, and input the image encoding result into a modality alignment layer of the multimodal large language model to obtain an image encoding result aligned to a text modality; A text encoding submodule, used for inputting the first question prompt text and the image encoding result aligned to the text modality into the text encoder of the multimodal large language model to obtain a multimodal text encoding result; A text representation acquisition submodule, used for inputting the multimodal text encoding result into a pre-trained large language model of the multimodal large language model to obtain a text representation of a target detection result, wherein the text representation of the target detection result is a token sequence including the name and location information of the target; The text decoding submodule is used to input the text representation of the target detection result into the text decoder of the multimodal large language model to obtain the answer text corresponding to the first question prompt text.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the autonomous driving target detection method for edge cases using a multimodal large language model as described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, which, when executed by a processor, implements the autonomous driving target detection method for edge cases using a multimodal large language model as described in any one of claims 1-5.

Images