A multimodal reasoning segmentation method and related device for intelligent driving
By employing a multimodal reasoning segmentation method that combines audio, image, and text inputs, and utilizing LLaVA and Grounded SAM models to process multi-target detection and segmentation, this approach addresses the inaccuracy of traditional models in complex scenarios, achieving high-precision detection and segmentation for intelligent driving systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2025-07-23
- Publication Date
- 2026-06-30
AI Technical Summary
In existing intelligent driving technologies, traditional small deep learning models struggle to accurately detect and segment multiple targets in complex scenarios. They also lack cross-modal fusion capabilities and cannot effectively process multimodal input information, resulting in insufficient accuracy and robustness in detection and segmentation, which makes it difficult to meet the safety and real-time requirements of intelligent driving systems.
A multimodal reasoning segmentation method is adopted, which combines audio, image and text input. The trained LLaVA model and Grounded SAM model are used for processing. The LLaVA model generates reasoning language response and the Grounded SAM model is used for target segmentation, thus realizing multi-target detection and segmentation.
It improves the accuracy of detection and segmentation in complex scenarios, supports the joint processing of multimodal information, enhances the practicality and flexibility of the system in complex environments, and meets the safety and real-time requirements of intelligent driving systems.
Smart Images
Figure CN120876861B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of multimodal artificial intelligence, semantic reasoning and object detection and segmentation technology, and specifically relates to a multimodal reasoning and segmentation method and related device for intelligent driving. Background Technology
[0002] With the continuous development of intelligent driving technology, accurately understanding the vehicle's surrounding environment and making safe and reliable driving decisions has become a key research focus. Object detection and image segmentation, as crucial tasks, are widely used in scenarios such as pedestrian recognition, vehicle recognition, traffic light recognition, and lane line recognition to assist autonomous driving systems in path planning and risk avoidance. Specifically, pedestrian detection effectively avoids collisions, vehicle detection ensures safe driving distances, traffic light recognition helps the system comply with traffic rules, and lane line detection improves the rationality and safety of driving paths.
[0003] However, with the increasing complexity and diversity of real-world applications, existing technologies place higher demands on image segmentation and object detection in terms of accuracy, robustness, and multi-object task processing capabilities. Traditional Small Deep Learning Models (SDLM models), due to their limited number of parameters and insufficient feature extraction capabilities, struggle to accurately detect and segment multiple types of objects in complex scenes, easily leading to missed detections, false detections, or segmentation errors. Furthermore, SDLM models heavily rely on high-quality labeled data and are less capable when facing unknown scenarios or data, typically requiring frequent fine-tuning to adapt to different tasks, thus significantly increasing model development and deployment costs.
[0004] Meanwhile, most existing SDLM models only support single-modal inputs such as images or text, lacking cross-modal fusion and collaborative processing capabilities. Especially in complex tasks that combine semantic understanding and visual reasoning, such as image segmentation based on natural language prompts, the performance of traditional models is limited, making it difficult to meet the needs of multimodal understanding and reasoning.
[0005] In contrast, large models (LM models) demonstrate significant advantages in visual understanding and language reasoning tasks. Compared to SDLM models, LM models have a larger parameter scale, stronger feature extraction capabilities, and deeper network structures, enabling them to accurately distinguish different targets from complex backgrounds, significantly improving detection and segmentation accuracy, and effectively reducing false negative and false positive rates. Furthermore, LM models possess excellent zero-shot generalization capabilities, adapting to new scenes and tasks without fine-tuning, reducing reliance on labeled data. More importantly, LM models support the joint processing of multimodal information such as text, images, and audio, possessing powerful multimodal reasoning capabilities, and have broad application prospects in cue-driven intelligent perception tasks. Existing research has already attempted to apply large models to visual segmentation and detection tasks. For example, the Segment Anything Model (SAM) supports zero-shot segmentation; the Large Language and Vision Assistant Model (LLaVA) combines a visual encoder and a large language model, possessing visual-language understanding capabilities; and the Large Language Instructed Segmentation Assistant Model (LISA) supports semantic segmentation based on input text prompts. However, the LISA and Grounded Segment Anything Models (Grounded SAM) only support text prompts; the LISA model still has insufficient segmentation accuracy in complex multi-object scenarios, and is prone to missing target classifications. The Grounded SAM model combines SAM (Segment Anything Model, a general segmentation model) with Grounding DINO (an open set detector) to achieve keyword-based multi-object segmentation, but it cannot fully handle long text prompts, has weak understanding of sentence-level instructions, and struggles to achieve high-precision semantic segmentation.
[0006] Furthermore, in intelligent driving applications, the system not only needs to possess multi-target detection and segmentation capabilities, but also needs to support multimodal input formats such as text and audio to adapt to diverse real-time interaction needs. In practical applications, completing target detection and segmentation tasks through voice prompts not only improves interaction efficiency but also enhances the system's practicality and flexibility in complex environments.
[0007] In summary, there is a need for an intelligent segmentation and detection method that can integrate multimodal input information such as audio, images, and text, and possess high-precision reasoning capabilities, in order to meet the comprehensive requirements of intelligent driving systems for safety, accuracy, and real-time performance. Summary of the Invention
[0008] To address the problems existing in the prior art, the present invention aims to provide a multimodal reasoning segmentation method and related apparatus for intelligent driving. The present invention can integrate multimodal input information such as audio, images and text, and has high-precision reasoning capabilities, which can meet the comprehensive requirements of intelligent driving systems for safety, accuracy and real-time performance.
[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0010] A multimodal reasoning segmentation method for intelligent driving includes the following steps:
[0011] Acquire data, including text prompts and initial image data collected by a cloud server;
[0012] The pre-trained LLaVA model is used to process the text prompts and initial image data to obtain inference language responses;
[0013] Extract keywords from the text prompts;
[0014] The trained Grounded SAM model is used to process the keywords and initial image data to obtain segmented images, thereby achieving the segmentation of the target object.
[0015] Preferably, the text prompt is a first text prompt that is directly entered and / or a second text prompt that converts audio instructions into text.
[0016] Preferably, the steps for converting audio instructions into second text prompts are as follows:
[0017] The audio instructions are encoded to extract audio features.
[0018] The audio features are processed using the ALLM model to generate the second text prompt.
[0019] Preferably, the specific steps for processing the text prompts and initial image data using a pre-trained LLaVA model to obtain the inference language response are as follows:
[0020] The initial image data is visually encoded to extract its visual features;
[0021] The LLM module in the LLaVA model is used to process the text prompts and visual features to obtain the inference language response.
[0022] Preferably, the LLM module in the LLaVA model adopts the LLaMA-7B model;
[0023] When training the LLaVA model, a low-rank adaptation method was used, based on DeepSpeed, and the same dataset as the LISA model was used to train LLaVA.
[0024] Preferably, the inference loss L when training the LLaVA model is the autoregressive cross-entropy loss, and the inference loss L is as follows:
[0025]
[0026] in, Represents a reasoning language response, and Let X represent the transfer functions of the visual encoder and the large language model in the LLaVA model, respectively. T and X I These represent the text prompt and the initial image data, respectively; Y L Represents a real-world language inference label sequence; t is the position index; N is the sequence length; y t-1 The true label is at position t-1; Let be the predicted label for the t-th position; Let θ be the conditional probability for the model parameters θ and φ.
[0027] Preferably, the specific steps for processing the keywords and initial image data using a pre-trained Grounded SAM model to obtain the segmented image are as follows:
[0028] The Grounding DINO module in the Grounded SAM model detects the location of the target object involved in the keyword in the initial image data based on the keyword, and then generates the localized image data;
[0029] The SAM module in the Grounded SAM model segments the target object based on the initial image data and the localized image data, resulting in a segmented image.
[0030] The present invention also provides a multimodal reasoning segmentation system for intelligent driving, for implementing the above-mentioned multimodal reasoning segmentation method for intelligent driving, comprising:
[0031] Data acquisition unit: used to acquire data, including text prompts and initial image data collected by the cloud server;
[0032] First data processing unit: used to process the text prompts and initial image data using the LLaVA model to obtain inference language responses;
[0033] Keyword extraction unit: used to extract keywords from the text prompts;
[0034] The second data processing unit is used to process the keywords and initial image data using the pre-trained Grounded SAM model to obtain segmented images and achieve the segmentation of target objects.
[0035] The present invention also provides an electronic device, comprising:
[0036] One or more processors;
[0037] A storage device on which one or more programs are stored;
[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal reasoning segmentation method for intelligent driving described above.
[0039] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the multimodal reasoning and segmentation method for intelligent driving described above.
[0040] The present invention has the following beneficial effects:
[0041] By processing each module differently, this invention achieves clearer and more accurate visual results compared to existing technologies (such as LISA and Grounded SAM) in single-target detection and segmentation tasks. For multi-target complex scene detection and segmentation tasks, it clearly displays the masks and boundary information of multiple targets in the visual presentation. Furthermore, it demonstrates excellent semantic understanding and visual collaborative reasoning capabilities when processing complex text prompts containing multiple semantic entities and environmental information. Attached Figure Description
[0042] Figure 1 This is the system architecture of the multimodal reasoning segmentation method (denoted as AVLLM-MRSD method) for intelligent driving proposed in this invention.
[0043] Figure 2 This is a visual comparison of the AVLLM-MRSD proposed in this invention with LISA and Grounded SAM in achieving single-object reasoning segmentation and detection.
[0044] Figure 3 This is a visual comparison of the results of the present invention AVLLM-MRSD with LISA and Grounded SAM in achieving multi-target reasoning segmentation and detection.
[0045] Figure 4 This is a visual comparison of the results of AVLLM-MRSD and LISA in terms of reasoning segmentation and detection under complex prompts. Detailed Implementation
[0046] The present invention will now be clearly and completely described with reference to the accompanying drawings and embodiments. The described embodiments are merely a part of the embodiments of the present invention, and not all of them.
[0047] This invention proposes a multimodal reasoning segmentation method for intelligent driving. This method is an audio-visual-language large model-based multimodal reasoning segmentation and detection (AVLLM-MRSD) approach. See also... Figure 1 This method includes two core processing flows:
[0048] 1. Voice or text prompt input process: Chinese or English voice prompts are processed into recognizable text prompts by the audio encoder and large language model in the Multimodal Audio Large Language Model (ALLM). The text prompts are then input together with the image into the Multimodal Vision Large Language Model (VLLM module) to complete subsequent inference, segmentation and detection tasks.
[0049] 2. Supports simultaneous input of Chinese or English text prompts and images to the multimodal vision-large language model (VLLM). Through encoding and decoding processes, it outputs inference language responses and segmentation detection images to achieve multi-target and single-target detection and segmentation tasks.
[0050] In addition, the method supports multi-round interactive reasoning and detection, allowing users to flexibly choose Chinese or English text prompts or voice prompts in each round, which significantly improves the system's interactive flexibility and ease of use in intelligent driving applications.
[0051] For details, see Figure 1 The multimodal reasoning and segmentation method for intelligent driving of the present invention specifically includes the following steps:
[0052] Step 1. Obtain text hints, including:
[0053] Step 1.1 If the information provided by the user is directly text, it can be used directly as a text prompt without further processing.
[0054] Step 1.2 If the information provided by the user is an audio command, it must first be converted into a text prompt by the Multimodal Audio Large Language Model (ALLM) module. The conversion process is as follows: First, the audio command needs to be encoded into an audio signal by an audio encoder to extract audio features. The audio features are then input into the Large Language Model (LLM module)①. Next, the Large Language Model (LLM module)① generates the corresponding text prompt based on the received audio features.
[0055] Step 2: Pass the text prompts generated in Step 1 to the Multimodal Vision Large Language Model (VLLM module), and process the text prompts in conjunction with the corresponding image data. The final output includes:
[0056] Step 2.1 (Processed by the LLaVA module): The text prompt is passed to the Large Language Model (LLM module) ②. At the same time, the image data acquired in real time from the cloud server (the data source of the cloud server includes real-time image (such as photos and videos) collected by the traffic management department's equipment (such as cameras) for observing traffic conditions) is passed to the Vision Encoder. The Vision Encoder encodes the image, extracts the visual features of the image data, and also passes the extracted visual features to the Large Language Model (LLM module) ②. The Large Language Model (LLM module) ② generates an inference language response based on the received text prompt and visual features to meet the user's needs.
[0057] The reasoning language response generation process is as follows:
[0058] If we represent the input text prompts and image data as X respectively T and X I The transfer functions of the visual encoder and the large language model (LLM②) in the LLaVA module are expressed as follows: and Where θ and φ represent the model parameters of the visual encoder and the Large Language Model (LLM②), respectively, when the text prompt X T and image X I After being input into the LLaVA module, the final inference language response It can be represented as:
[0059] The LLaVA module training process includes the following: To ensure effective generation of inference language responses in the LLaVA module, the following training and optimization strategy is adopted: The LLaVA module used in the VLLM module for generating inference language responses is trained and fine-tuned using the Low-Rank Adaptation (LoRA) method. Specifically, since LLaMA-7B (known) only supports text input, a pre-trained model LLaVA-7B-Lightening-v1-1 with image-text joint understanding capability is first obtained by loading LLaVA-Lightening-7B-delta-v1-1 (incremental weights) onto LLaMA-7B. This model possesses the LoRA method. This invention uses the LoRA method, based on DeepSpeed, and the same dataset as LISA to fine-tune LLaVA. After fine-tuning, the trained LoRA weights are merged with the weights of the pre-trained model LLaVA-7B-Lightening-v1-1 to obtain the final full-weight model.
[0060] The training objective is as follows: The goal of training the LLaVA module is to minimize the inference loss L, which is the autoregressive cross-entropy loss, and can be defined as follows:
[0061]
[0062] Among them, Y L Represents a real-world language inference label sequence; N is the sequence length; y t-1 The true label is at position t-1; p is the predicted label for the t-th position; (θ,φ) The conditional probabilities are given by the model parameters θ (visual encoder) and φ (large language model).
[0063] LoRA actually fine-tunes the linear transformation weight matrices q_proj (Query Projection) and v_proj (Value Projection) within the Attention submodules of each TransformerBlock in the LLM (Vicuna) Transformer module. Specifically, these two modules map the representation vector of each token processed by the tokenizer (i.e., the model's contextual semantic representation of the token in the current layer) to the Query and Value vectors in the attention mechanism, respectively, making them key components of the self-attention mechanism. By injecting trainable low-rank parameters into these mapping matrices, LoRA can achieve efficient parameter fine-tuning with low computational cost while maintaining the original model's weights frozen, thereby significantly improving the model's adaptability and performance in downstream specific tasks.
[0064] Note: Transformer is a deep neural network architecture based on self-attention mechanism, widely used for processing sequential data, and is particularly suitable for natural language understanding and generation tasks.
[0065] Step 2.2 (processing the pre-trained Grounded SAM model): The text prompt is passed to the KeywordExtractor to extract keywords. Then, the image data acquired in real time from the cloud server is passed to the GroundingDINO module. The GroundingDINO module detects the location of the target object involved in the keyword in the real-time image data based on the extracted keywords and generates the localized image data. The localized image data is then passed to the SAM module. The SAM module performs precise segmentation of the target object based on the initial real-time image data and the localized image data generated by the GroundingDINO module, and finally outputs the segmented image, achieving fine division of the target object.
[0066] like Figure 1 As shown, this invention constructs a system architecture for an Audio-Vision-Language Large Model-based Multimodal Reasoning Segmentation and Detection (AVLLM-MRSD) method. This architecture mainly includes two core modules:
[0067] 1) The Multimodal Audio Large Language Model (ALLM) module processes audio commands and converts them into text prompts. This module uses the Qwen-Audio model and includes the following sub-components:
[0068] Audio Encoder: Employs the Whisper-large-v2 model for extracting audio features;
[0069] Large Language Model (LLM)①: The Qwen-7B model is used for semantic understanding and language conversion;
[0070] 2) The Multimodal Vision Large Language Model (VLLM) module can process text prompts generated by the ALLM module and image data obtained in real time from the cloud server to generate inference language responses and segmented images.
[0071] The model consists of two parts:
[0072] 1. The LLaVA module (also known as the Large Language and Vision Assistant module) includes the following sub-modules:
[0073] Vision Encoder: Uses CLIP ViT-L / 14 to encode images for extracting visual features from image data;
[0074] Large Language Model (LLM)②: Used to process text prompts using Vicuna;
[0075] 2. Pre-trained Grounded SAM module: includes the following sub-components:
[0076] Keyword Extractor: Employs the basic Natural Language Toolkit (NLTK) to extract keywords from text prompts.
[0077] Grounding DINO: Performs detection on image data acquired in real time from a cloud server, locates objects mentioned in the received keywords, and generates located image data.
[0078] SAM (Segment Anything Model): Performs segmentation tasks by combining the localized image data with the original image data acquired in real time from a cloud server to perform fine segmentation of the detected and located objects.
[0079] The processing and training procedures for each module are as follows: In this invention, the pre-trained Grounded SAM module in Multimodal ALLM and MultimodalVLLM both use pre-trained and frozen models. Only the LLaVA module in MultimodalVLLM is fine-tuned using the Low-Rank Adaptation (LoRA) method. This strategy is beneficial for fully utilizing the prior knowledge of the pre-trained multimodal model and leveraging the zero-shot capability of large models.
[0080] Example
[0081] The simulation experiment process and results of this embodiment are as follows:
[0082] 1. Experimental setup:
[0083] Summary of LoRA fine-tuning settings:
[0084] 1) Hardware configuration:
[0085] Eight NVIDIA RTX 4090 (24GB) cards were used for training;
[0086] One NVIDIA RTX 4090 (24GB) was used for inference;
[0087] The entire fine-tuning process took less than a day.
[0088] 2) LoRA parameter settings:
[0089] Setting the rank of a low-rank matrix to 8 strikes a balance between training speed and model performance.
[0090] 3) Optimizer settings:
[0091] Use the AdamW optimizer;
[0092] Learning rate: 0.0003;
[0093] Weight decay: 0;
[0094] 4) Training parameters:
[0095] Total training rounds: 10;
[0096] Number of training steps per round: 500;
[0097] Batch size per GPU: 2;
[0098] Gradient accumulation steps: 10;
[0099] 2. Experimental Results
[0100] 1) Visual results of single-object reasoning segmentation and detection achieved by AVLLM-MRSD, LISA, and Grounded SAM are compared. Table 1 shows the relative IOU and GIOU of the five image types for AVLLM-MRSD, LISA, and Grounded SAM.
[0101] Table 1
[0102] Image Ours(IoU) LISA(IoU) Grounded SAM(IoU) Car 1.00 0.55 0.28 Pedestrian 1.00 0.80 0.72 Lane 1.00 0.59 0.38 Barrier 1.00 0.65 0.11 Traffic light 1.00 0.95 0.25 gIoU 1.00 0.71 0.35
[0103] like Figure 2 As shown in Table 1, AVLLM-MRSD outperforms LISA and Grounded SAM in terms of inference response capability and segmentation detection quality, especially in handling small targets at a distance. LISA is better than Grounded SAM in understanding sentence-level prompts, while Grounded SAM, which only supports word or phrase-level prompts, produces incorrect and confusing results.
[0104] 2) Visual results of multi-object reasoning segmentation and detection achieved by AVLLM-MRSD and compared with LISA and Grounded SAM:
[0105] like Figure 3 As shown, in multi-target reasoning segmentation and detection tasks, LISA cannot clearly segment multiple targets, while Grounded SAM performs poorly in small target detection. In contrast, AVLLM-MRSD can accurately detect and segment all targets, including small targets, and its advantage lies in its stronger ability to understand complex cues.
[0106] 3) Comparison of visual results between AVLLM-MRSD and LISA, used to achieve inference segmentation and detection of complex cues:
[0107] like Figure 4 As shown, for complex prompts, AVLLM-MRSD provides a more detailed and accurate scene description; in contrast, LISA generates relatively vague segmentation results with lower accuracy. Grounded SAM was not evaluated in this round of experiments because it cannot handle sentence-level complex prompts.
[0108] Furthermore, embodiments of the present invention also provide a system for implementing the above-described multimodal reasoning segmentation method for intelligent driving, the system comprising:
[0109] Data acquisition unit: used to acquire data, including text prompts and initial image data collected by the cloud server;
[0110] First data processing unit: used to process the text prompts and initial image data using the trained LLaVA model to obtain inference language responses;
[0111] Keyword extraction unit: used to extract keywords from the text prompts;
[0112] The second data processing unit is used to process the keywords and initial image data using the pre-trained Grounded SAM model to obtain segmented images and achieve the segmentation of target objects.
[0113] The embodiments of the present invention also provide corresponding electronic devices and computer-readable storage media for implementing the solutions provided in the embodiments of the present invention.
[0114] The electronic device includes a storage device and one or more processors. The storage device stores instructions or code, and the processors execute the instructions or code to enable the device to perform the multimodal reasoning segmentation method for intelligent driving as described in any embodiment of this application.
[0115] The storage medium stores a computer program, which, when executed by a processor, implements the multimodal reasoning and segmentation method for intelligent driving described in any embodiment of this application.
[0116] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.
[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A multimodal reasoning segmentation method for intelligent driving, characterized in that, Includes the following steps: The acquired data includes text prompts and initial image data collected by a cloud server; the text prompts are either a first text prompt input directly or a second text prompt that converts audio instructions into text; the steps for converting audio instructions into the second text prompt are as follows: the audio instructions are encoded to extract audio features; the audio features are processed using an ALLM model to generate the second text prompt. The text prompts and initial image data are processed using a pre-trained LLaVA model to obtain an inference language response. The specific steps are as follows: visual encoding is performed on the initial image data to extract visual features; the LLM module in the LLaVA model is used to process the text prompts and visual features to obtain an inference language response. Extract keywords from the text prompts; The trained Grounded SAM model is used to process the keywords and initial image data to obtain segmented images, thereby achieving the segmentation of target objects. The specific steps are as follows: The GroundingDINO module in the Grounded SAM model detects the position of the target object involved in the keywords in the initial image data based on the keywords, and then generates the localized image data. The SAM module in the Grounded SAM model segments the target object based on the initial image data and the localized image data, resulting in a segmented image.
2. The multimodal reasoning segmentation method for intelligent driving according to claim 1, characterized in that, The LLM module in the LLaVA model adopts the LLaMA-7B model; When training the LLaVA model, a low-rank adaptation method was used, based on DeepSpeed, and the same dataset as the LISA model was used to train LLaVA.
3. The multimodal reasoning segmentation method for intelligent driving according to claim 1, characterized in that, The inference loss L during training of the LLaVA model is the autoregressive cross-entropy loss, which is as follows: in, Represents a reasoning language response, , and Let represent the transfer functions of the visual encoder and the large language model in the LLaVA model, respectively. and These represent the text prompt and the initial image data, respectively. This represents a real-world language inference label sequence; t is the position index; N is the sequence length; The true label is at position t-1; Let be the predicted label for the t-th position; For model parameters and The conditional probability under the given conditions.
4. A multimodal reasoning and segmentation system for intelligent driving, characterized in that, The method for implementing the multimodal reasoning segmentation method for intelligent driving according to any one of claims 1-3 includes: Data acquisition unit: used to acquire data, including text prompts and initial image data collected by the cloud server; First data processing unit: used to process the text prompts and initial image data using the trained LLaVA model to obtain inference language responses; Keyword extraction unit: used to extract keywords from the text prompts; The second data processing unit is used to process the keywords and initial image data using the pre-trained Grounded SAM model to obtain segmented images and achieve the segmentation of target objects.
5. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal reasoning segmentation method for intelligent driving as described in any one of claims 1-3.
6. A storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the multimodal reasoning segmentation method for intelligent driving as described in any one of claims 1-3.