An intelligent cockpit local large model inference acceleration method, system and cockpit accelerator

Abstract

This invention discloses a method, system, and cockpit accelerator for accelerating inference of a local large-scale model in an intelligent cockpit. Its features include: encoding multimodal data based on the main processor and inference accelerator of the cockpit accelerator to determine a first embedded feature vector and a second embedded feature vector; fusing the first and second embedded feature vectors based on the main processor to determine a feature fusion vector; and submitting the feature fusion vector to the local cockpit model of the intelligent cockpit for understanding and generation based on the main processor to determine the model generation result. The dual-processor architecture of this invention can effectively meet the computing power requirements of large-scale models on the vehicle side, effectively reducing equipment costs while ensuring multimodal data processing efficiency, and is adaptable to the vehicle installation environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle cockpits, and more particularly to a method, system, and cockpit accelerator for accelerating inference of a local large model of an intelligent cockpit. Background Technology

[0002] With the development of intelligent connected vehicles, the intelligent cockpit has become the core entry point for human-vehicle interaction. Passengers in intelligent cockpits have increasingly higher demands for AI interaction within the cockpit, moving beyond conventional voice dialogue to multimodal interactive experiences such as gesture recognition, driver monitoring, in-vehicle entertainment, and navigation. Existing intelligent cockpits deploy large language models (LLMs) or multimodal large models on the vehicle side, supporting users' multimodal processing needs through the vehicle's built-in local terminal. In current technologies, deploying large language models in intelligent cockpits typically uses a high-performance application processor paired with a dedicated neural network inference accelerator. While this can handle single-modal tasks well in traditional intelligent cockpits, it suffers from insufficient computing power in multimodal inference scenarios involving large models, failing to meet the high computing power requirements of multimodal data. Furthermore, meeting high computing power increases the equipment cost for multimodal data processing in the vehicle's intelligent cockpit, and the power consumption and heat dissipation bottlenecks of vehicle-side chips prevent high-performance chips from operating continuously and efficiently. Therefore, due to limitations in cost and installation environment, the current single-chip architecture of in-vehicle smart cockpits has poor operating efficiency and inference performance in multimodal inference scenarios, failing to meet user needs. Summary of the Invention

[0003] This invention provides a method, system, and cockpit accelerator for accelerating inference of a local large model in an intelligent cockpit, in order to solve the technical problem of poor efficiency in processing multimodal data in multimodal inference scenarios of vehicle-mounted intelligent cockpits in the prior art.

[0004] According to one aspect of the present invention, a method for accelerating inference of a local large model of an intelligent cockpit is provided, applied to a cockpit accelerator, comprising: The main processor and inference accelerator of the cockpit accelerator encode multimodal data to determine a first embedded feature vector and a second embedded feature vector. Based on the main processor, the first embedded feature vector and the second embedded feature vector are fused to determine the feature fusion vector; The main processor submits the feature fusion vector to the local cockpit model of the smart cockpit for understanding and generation, and determines the model generation result.

[0005] According to another aspect of the present invention, an inference acceleration system for a local large model of an intelligent cockpit is provided, deployed in a cockpit accelerator, comprising: The data processing module is used to encode multimodal data based on the main processor and inference accelerator of the cockpit accelerator, and determine the first embedded feature vector and the second embedded feature vector. The feature vector fusion module is used to perform vector fusion of the first embedded feature vector and the second embedded feature vector based on the main processor to determine the feature fusion vector; The edge-side large model module is used to understand and generate the local cockpit model of the smart cockpit based on the feature fusion vector submitted by the main processor, and to determine the model generation result.

[0006] According to another aspect of the present invention, a cockpit accelerator is provided, characterized in that the cockpit accelerator comprises: a main processor and an inference accelerator; The main processor and the inference accelerator are used to execute the inference acceleration method for the local large model of the intelligent cockpit as described in any embodiment of the present invention.

[0007] The technical solution of this invention, through the design of a dual-inference accelerator architecture, adopts a dual-computing-power chip design of a main processor and an inference accelerator, which can effectively meet the computing power requirements of large models on the vehicle side. Furthermore, by employing cross-modal alignment and fusion of multimodal features, it effectively improves processing efficiency while retaining the effective features of multimodal data. This not only improves the processing efficiency of multimodal data and effectively reduces the equipment cost for deploying local multimodal large language models in vehicles, but also adapts to the automotive-grade installation environment. It solves the technical problem of poor multimodal data processing efficiency in multimodal inference scenarios of vehicle-side intelligent cockpits in existing technologies, achieving the goal of meeting the computing power requirements of large models on the vehicle side and effectively reducing equipment costs while ensuring multimodal data processing efficiency.

[0008] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of an inference acceleration method for a local large model of an intelligent cockpit is provided as an embodiment of the present invention; Figure 2A flowchart of another method for accelerating inference of a local large model of a smart cockpit provided in an embodiment of the present invention; Figure 3 A schematic diagram of the inference acceleration system for a local large model of an intelligent cockpit provided in an embodiment of the present invention; Figure 4 A schematic diagram of a cockpit accelerator implementing an embodiment of the present invention is disclosed. Detailed Implementation

[0011] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0012] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0013] Figure 1 This invention provides a flowchart of an inference acceleration method for a local large-scale model of a smart cockpit. This embodiment is applicable to situations involving local data processing and feature extraction of multimodal data from a vehicle's smart cockpit. This method is applied to a cockpit accelerator and can be executed by an inference acceleration system for a local large-scale model of the smart cockpit. Figure 1 As shown, the method includes: S110. The main processor and inference accelerator of the cockpit accelerator encode the multimodal data to determine the first embedded feature vector and the second embedded feature vector.

[0014] Optionally, the cockpit accelerator can be a separate hardware module installed within the vehicle. For example, the cockpit accelerator can be located within the intelligent cockpit domain controller, belonging to the vehicle's cockpit domain; the specific installation location of the cockpit accelerator can be below the vehicle's center console, behind the glove box, or inside the dashboard.

[0015] Optionally, the cockpit accelerator consists of a main processor and an inference accelerator. The main processor can support general smart cockpit applications and has the ability to communicate with the smart cockpit, acquiring multimodal data from vehicle data acquisition devices and / or the smart cockpit. This multimodal data can be data in different modalities, including text, images, audio, and / or video.

[0016] Optionally, multimodal data can be collected by vehicle acquisition equipment, which may be equipped with cameras, microphone arrays, and network antennas in the smart cockpit; the smart cockpit cameras can collect video and image data inside and outside the vehicle cockpit; the smart cockpit microphone arrays can collect audio data; and the smart cockpit network antennas can receive audio, image, and video data. Optionally, multimodal data can be audio, video, and text data input by the user using the smart cockpit. It should be noted that the smart cockpit's in-vehicle interface provides users with selection controls for inputting audio, video, and text data. Users can select and input audio, video, image, and / or text data by operating the selection controls.

[0017] Optionally, the first embedding feature vector can be a set of vectors representing image features, audio features, and / or video features. It should be noted that the first embedding feature vector can be a set of vectors representing non-textual features, and includes at least one of image embedding vectors, audio embedding vectors, and video embedding vectors. The first embedding feature vector is obtained by encoding image data, audio data, and video data from multimodal data.

[0018] Optionally, the second embedded feature vector can be a feature vector representing text features; it should be noted that, in this invention, the second embedded feature vector is a plain text type feature vector, which is obtained by encoding text data in multimodal data.

[0019] Specifically, the main processor and inference accelerator of the cockpit accelerator encode the multimodal data to determine the first embedded feature vector and the second embedded feature vector.

[0020] S120. Based on the main processor, the first embedded feature vector and the second embedded feature vector are fused to determine the feature fusion vector.

[0021] Optionally, when the main processor and inference accelerator encode multimodal data, the encoding output is set to an embedding vector of the same dimension, resulting in a first embedding feature vector and a second embedding feature vector of the same dimension.

[0022] Optionally, the feature fusion vector can be a cross-modal feature vector from audio data, video data, image data, and / or text data. It should be noted that the feature fusion vector is composed of image embedding vectors, audio embedding vectors, video embedding vectors, and / or text embedding vectors, and can be used to represent the cross-modal semantics of multimodal data to generate cross-modal contexts applicable to large language models.

[0023] Specifically, the main processor performs vector fusion on the first embedded feature vector and the second embedded feature vector to determine the feature fusion vector.

[0024] Optionally, in another optional embodiment of the present invention, the step of performing vector fusion of the first embedded feature vector and the second embedded feature vector based on the main processor to determine the feature fusion vector includes: The first embedded feature vector and the second embedded feature vector are cross-modal aligned to determine a multimodal vector set; the multimodal vector set is fused along the vector dimension based on a preset vector fusion method to determine the determined feature fusion vector.

[0025] The preset vector fusion method can be a pre-set method for fusing various multimodal vectors.

[0026] Optionally, in this invention, the preset vector fusion method can be co-space projection fusion, where a projection network maps the first and second embedded feature vectors to a common multimodal semantic space, respectively, to obtain an aligned multimodal vector set. The multimodal vector set is then further compressed directly in the common multimodal semantic space using a projection layer to obtain a feature fusion vector. The multimodal vector set can be composed of aligned multimodal vectors. Optionally, the preset vector fusion method can also be simple concatenation, which directly merges the embedded feature vectors of each modality in the multimodal vector set to obtain the feature fusion vector.

[0027] Optionally, the preset vector fusion method can also be cross-modal attention. Cross-modal attention is used to weightedly fuse the embedded feature vectors of each modality in the multimodal vector set to obtain a feature fusion vector.

[0028] Specifically, the first and second embedded feature vectors are cross-modal aligned to determine a multimodal vector set; the multimodal vector set is then fused along the vector dimension based on a preset vector fusion method to determine the feature fusion vector.

[0029] S130. Based on the main processor, the feature fusion vector is submitted to the local cockpit model of the smart cockpit for understanding and generation, and the model generation result is determined.

[0030] Optionally, the local cockpit model can be a multimodal large language model configured on the local device side within the smart cockpit. It should be noted that the local cockpit model is set up within the vehicle's smart cockpit. During the interaction process between the local large model and the user, it can receive multimodal data based on user needs. After processing the multimodal data through a cockpit accelerator, a feature fusion vector is obtained. The local cockpit model then generates a cross-modal context based on the feature fusion vector, producing a model generation result corresponding to the user's needs.

[0031] Optionally, the model-generated result can be the response output of the local cockpit model. It should be noted that the model-generated result can be any form of response. The model-generated result can be at least one of images, videos, audio, and text. For example, a user inputs multimodal data—a driving video recorded by an in-vehicle camera—to the local large model, requesting the local large model to output the vehicle color and vehicle information from the driving video; the cockpit accelerator generates a feature fusion vector based on the driving video and generates a cross-modal context; the local large model, based on the cross-modal context, generates a vehicle color of white and vehicle information as "Wenjie Brand New Energy Vehicle".

[0032] Optionally, the model generation results can be used to describe videos and images in multimodal data; answer questions about the modal content in multimodal data; and perform inference based on multimodal data to obtain inference results.

[0033] Specifically, the main processor submits the feature fusion vector to the local cockpit model of the smart cockpit for understanding and generation, and determines the model generation result.

[0034] The technical solution of this invention, through the design of a dual-inference accelerator architecture, adopts a dual-computing-power chip design of a main processor and an inference accelerator, which can effectively meet the computing power requirements of large models on the vehicle side. Furthermore, by employing cross-modal alignment and fusion of multimodal features, it effectively improves processing efficiency while retaining the effective features of multimodal data. This not only improves the processing efficiency of multimodal data and effectively reduces the equipment cost for deploying local multimodal large language models in vehicles, but also adapts to the automotive-grade installation environment. It solves the technical problem of poor multimodal data processing efficiency in multimodal inference scenarios of vehicle-side intelligent cockpits in existing technologies, achieving the goal of meeting the computing power requirements of large models on the vehicle side and effectively reducing equipment costs while ensuring multimodal data processing efficiency.

[0035] Figure 2 This is a flowchart illustrating another method for accelerating inference of a local large model in a smart cockpit, provided by an embodiment of the present invention. The relationship between this embodiment and the previous embodiments is to specifically describe the detailed process of the cockpit accelerator encoding multimodal data. For example... Figure 2 As shown, the method includes: S210. Based on the main processor, the multimodal data is decomposed and extracted to determine the audio / video data and text data.

[0036] Optionally, the main processor includes an encoding / decoding processing unit, a first central processing unit, and a first neural network unit; the encoding / decoding processing unit may be a real-time audio and video processing unit composed of a codec (encoder-decoder) and a DSP (Digital Signal Processor) in the main processor, the first central processing unit may be a CPU (Central Processing Unit) in the main processor, and the first neural network unit may be an NPU (Natural Processing Unit) in the main processor.

[0037] Optionally, the main processor calls the encoding / decoding processing unit and the first neural network unit through the first central processing unit to decompose and extract multimodal data, and determine the audio / video data and text data in the multimodal data.

[0038] Specifically, the main processor decomposes and extracts multimodal data to determine audio / video data and text data.

[0039] Optionally, in another optional embodiment of the present invention, the step of decomposing and extracting the multimodal data based on the main processor to determine the audio / video data and the text data includes: The first central processing unit decomposes the multimodal data to determine audio / video data and original text; the first neural network unit performs text processing on the audio / video data to determine audio / video text; and the original text and the audio / video text are used to determine the text data.

[0040] Optionally, the audio and video data can be a multimedia dataset composed of audio data, video data, and image data from multimodal data; the original text can be text data from multimodal data.

[0041] Optionally, the encoding / decoding processing unit based on the main processor receives and preprocesses audio and video data. The first central processing unit identifies and classifies each data point in the multimodal data, determining the modality type corresponding to each data point, namely audio, video, image, and text. Based on the timestamps of the audio, video, and image data, it performs time synchronization and data slicing to obtain a time-aligned multimedia dataset, i.e., audio and video data. It also extracts text data from the multimodal data as the original text. If the image data itself does not have a timestamp, the timestamp of the image data entering the smart cockpit is used as the default timestamp for the image data.

[0042] Optionally, the audio and video text can be text that exists within the audio and video data; it should be noted that audio data contains speech text, video data contains video content text, and image data contains image text; the first central processing unit calls the first neural network unit to extract text from the audio data, video data, and image data to obtain the audio and video text.

[0043] Optionally, the first neural network unit employs a pre-trained local edge-side text recognition neural network to extract text from audio, video, and image data, obtaining audio-visual text. The local edge-side text recognition neural network can consist of a speech recognition neural network and a video frame text recognition neural network. The speech recognition neural network is used to extract text from the audio data, and the video frame text recognition neural network is used to extract text from the video and image data. For example, the speech recognition neural network can be an end-to-end ASR model; the video frame text recognition neural network can be an end-to-end OCR model.

[0044] Optionally, audio and video text can be set with timestamp and data identifiers to associate the audio and video text with the corresponding audio data, video data, or image data.

[0045] Specifically, the multimodal data is decomposed based on the first central processing unit to determine the audio and video data and the original text in the multimodal data; the audio and video data is processed by the first neural network unit to determine the audio and video text; and the audio and video text and the original text are combined into text data.

[0046] S220. Based on the main processor, the text data is sent to the inference accelerator through a high-speed hardware transmission line.

[0047] Optionally, the high-speed hardware transmission line can be a high-speed hardware link between the main processor and the inference accelerator within the cockpit accelerator. It should be noted that the cockpit accelerator resides on an onboard motherboard, and can be understood as a dual-computing-power chip architecture. The main processor and inference accelerator of the cockpit accelerator are two complete processor systems on the smart cockpit motherboard; the high-speed hardware transmission line can be a hardware line connecting the main processor and the inference accelerator on the onboard motherboard. For example, the high-speed hardware transmission line can be a PCI Express line.

[0048] Optionally, after obtaining the audio / video data and text data, the main processor transmits the text data to the inference accelerator via a high-speed hardware transmission line. After the inference accelerator has received all the text data, the main processor interrupts the data transmission process.

[0049] Specifically, the main processor sends text data to the inference accelerator via a high-speed hardware transmission line.

[0050] S230. Encode the audio and video data based on the main processor to determine the first embedded feature vector; encode the text data based on the inference accelerator to determine the second embedded feature vector.

[0051] Optionally, the inference accelerator consists of a second central processing unit and a second neural network unit. The second central processing unit can be the CPU in the inference accelerator, and the second neural network unit can be the NPU in the inference accelerator.

[0052] Specifically, the main processor encodes the audio and video data to determine the first embedded feature vector; the inference accelerator encodes the text data to determine the second embedded feature vector.

[0053] Optionally, in another optional embodiment of the present invention, the step of encoding the audio and video data based on the main processor to determine the first embedded feature vector includes: The first central processing unit allocates cockpit system memory to the first neural network unit; after the cockpit system memory allocation is completed, the first neural network unit encodes the audio and video data based on the cockpit system memory to determine the first embedded feature vector.

[0054] Optionally, the cockpit system memory can be the high-speed operating memory of the vehicle's intelligent cockpit software system. It should be noted that during the audio and video data encoding process, the first neural network unit needs to use the cockpit system memory for encoding. The first neural network unit initiates a memory access request to the first central processing unit to await a response from the in-vehicle intelligent cockpit.

[0055] Optionally, in this invention, the vehicle's intelligent cockpit software system can run on the main processor, with cockpit system memory located outside the main processor. A first central processing unit and a memory controller control the cockpit system memory to support the operation of the intelligent cockpit software system. When the intelligent cockpit software system is running on the main processor, when the first central processing unit calls the first neural network unit to encode audio and video data, it allocates long-term valid cockpit system memory to the first neural network unit. Once the cockpit system memory allocation is complete, the first neural network unit initiates direct memory access to encode the audio and video data, obtaining a first embedded feature vector.

[0056] Optionally, in another scenario of the present invention, the vehicle's intelligent cockpit software system runs on a dedicated processor, and the cockpit system memory is set as dynamically shared memory. Under normal circumstances, the intelligent cockpit software system runs on the dynamically shared memory first. When the first neural network unit needs to request cockpit system memory, the first central processing unit sends a memory scheduling request. If the memory scheduling request is responded to, cockpit system memory is allocated to the first neural network unit. After the cockpit system memory allocation is completed, the first neural network unit initiates direct memory access to encode the audio and video data to obtain the first embedded feature vector.

[0057] Optionally, during the time interval when the first neural network unit uses the cockpit system memory, a real-time memory scheduling mechanism is used for the intelligent cockpit software system and the first neural network unit to meet the real-time operation and interaction of the intelligent cockpit and ensure the audio and video data encoding of the first neural network unit.

[0058] Optionally, when encoding audio and video data, the first neural network unit uses an image encoder for image data, an audio encoder for audio data, and a video encoder for video data. For example, the image encoder can be Vision Transformer, ResNet, or CLIP; the audio encoder can be Spectrogram + CNN or AudioTransformer; and the video encoder can be 3DCNN, TimeSformer, or ViViT.

[0059] Specifically, the first central processing unit allocates cockpit system memory to the first neural network unit; after the cockpit system memory allocation is completed, the first neural network unit encodes audio and video data based on the cockpit system memory to determine the first embedded feature vector.

[0060] Optionally, in another optional embodiment of the present invention, the step of encoding the text data based on the inference accelerator to determine the second embedded feature vector includes: When the second central processing unit receives the text data, it calls the inference onboard memory and the second neural network unit to encode the text data and determine the second embedded feature vector; the second central processing unit then sends the second embedded feature vector to the main processor through the hardware high-speed transmission line.

[0061] The inference onboard memory can be dedicated memory configured for the inference accelerator within the cockpit accelerator. It should be noted that the inference onboard memory is located on the motherboard of the cockpit accelerator and is used for the encoding process of the second neural network unit.

[0062] Optionally, after the second central processing unit finishes receiving the text data sent by the first central processing unit based on the high-speed hardware transmission line, the second central processing unit and the memory controller call the inference board memory to start the second neural network unit to encode the text data. The second neural network unit encodes the text data based on the inference board memory to determine the second embedded feature vector.

[0063] Optionally, the second central processing unit obtains the second embedded feature vector and sends the second embedded feature vector to the main processor through a high-speed hardware transmission line.

[0064] Specifically, when the second central processing unit receives text data, it calls the inference board memory and the second neural network unit to encode the text data and determine the second embedded feature vector; the second central processing unit then sends the second embedded feature vector to the main processor through a high-speed hardware transmission line.

[0065] S240. Based on the main processor, the first embedded feature vector and the second embedded feature vector are fused to determine the feature fusion vector.

[0066] S250. Based on the main processor, the feature fusion vector is submitted to the local cockpit model of the smart cockpit for understanding and generation, and the model generation result is determined.

[0067] Optionally, in another optional embodiment of the present invention, the step of submitting the feature fusion vector to the local cockpit model of the smart cockpit by the main processor for understanding and generation, and determining the model generation result, includes: The main processor converts the feature fusion vector into a semantic word sequence and submits the semantic word sequence to the local cockpit model of the smart cockpit. When the local cockpit model receives the semantic word sequence, it understands and generates based on the user requirements submitted by the smart cockpit and the semantic word sequence, and determines the model generation result corresponding to the user requirements.

[0068] The semantic word sequence can be the tokens corresponding to the feature fusion vector. It should be noted that the input content of the local cockpit model consists of a cross-modal context composed of semantic word sequences. The feature fusion vector is converted into tokens and submitted to the local cockpit model of the intelligent cockpit. The tokens corresponding to the feature fusion vector typically number in the thousands, with each token being 64 bits. Fast exchange of semantic word sequences can be achieved via PCIe / USB.

[0069] Among them, user requirements can be the output type that users request for local large models to process multimodal data.

[0070] Optionally, the feature fusion vector is mapped to at least one discrete token, and these discrete tokens are combined into a discrete or semi-discrete token stream understood by the large language model, serving as the input prefix for the local cockpit model. The main processor submits the generated semantic lexical sequence to the local cockpit model. After receiving the semantic lexical sequence, the local cockpit model executes the following internal process: converting user requirements into a prompt token sequence, fusing or concatenating it with the semantic lexical sequence to obtain the overall sequence, and deeply understanding the user requirements and semantic lexical sequence in the overall sequence, outputting the model-generated result according to the task type.

[0071] Optionally, the local cockpit model may have different output modules: a language decoder for text generation, a fully connected classifier for classification tasks, and a specific visual head for detection or segmentation, such as DETR (DEtectionTRansformer).

[0072] Specifically, the main processor converts the feature fusion vector into a semantic word sequence, and submits the semantic word sequence to the local cockpit model of the smart cockpit. When the local cockpit model receives the semantic word sequence, it understands and generates based on the user requirements and semantic word sequence submitted by the smart cockpit, and determines the model generation result corresponding to the user requirements.

[0073] The technical solution of this invention, through the design of a dual-inference accelerator architecture, adopts a dual-computing-power chip design of a main processor and an inference accelerator, which can effectively meet the computing power requirements of large models on the vehicle side. Furthermore, by employing cross-modal alignment and fusion of multimodal features, it effectively improves processing efficiency while retaining the effective features of multimodal data. This not only improves the processing efficiency of multimodal data and effectively reduces the equipment cost for deploying local multimodal large language models in vehicles, but also adapts to the automotive-grade installation environment. It solves the technical problem of poor multimodal data processing efficiency in multimodal inference scenarios of vehicle-side intelligent cockpits in existing technologies, achieving the goal of meeting the computing power requirements of large models on the vehicle side and effectively reducing equipment costs while ensuring multimodal data processing efficiency.

[0074] Figure 3 This is a schematic diagram of the inference acceleration system for a local large model of an intelligent cockpit, provided as an embodiment of the present invention. Figure 3 As shown, the device includes: a data processing module 310, a feature vector fusion module 320, and an edge-side large model module 330; wherein, Data processing module 310 is used to encode multimodal data based on the main processor and inference accelerator of the cockpit accelerator, and determine a first embedded feature vector and a second embedded feature vector. The feature vector fusion module 320 is used to perform vector fusion of the first embedded feature vector and the second embedded feature vector based on the main processor to determine the feature fusion vector. The edge-side large model module 330 is used to understand and generate the local cockpit model of the smart cockpit based on the feature fusion vector submitted by the main processor, and to determine the model generation result.

[0075] The technical solution of this invention, through the design of a dual-inference accelerator architecture, adopts a dual-computing-power chip design of a main processor and an inference accelerator, which can effectively meet the computing power requirements of large models on the vehicle side. Furthermore, by employing cross-modal alignment and fusion of multimodal features, it effectively improves processing efficiency while retaining the effective features of multimodal data. This not only improves the processing efficiency of multimodal data and effectively reduces the equipment cost for deploying local multimodal large language models in vehicles, but also adapts to the automotive-grade installation environment. It solves the technical problem of poor multimodal data processing efficiency in multimodal inference scenarios of vehicle-side intelligent cockpits in existing technologies, achieving the goal of meeting the computing power requirements of large models on the vehicle side and effectively reducing equipment costs while ensuring multimodal data processing efficiency.

[0076] Optionally, the data processing module 310 is specifically used for: The main processor decomposes and extracts the multimodal data to determine audio / video data and text data. The main processor transmits the text data to the inference accelerator via a high-speed hardware transmission line. The first embedded feature vector is determined by encoding the audio and video data based on the main processor. The text data is encoded using the inference accelerator to determine the second embedded feature vector.

[0077] Optionally, the main processor includes a first central processing unit and a first neural network unit; Optionally, the data processing module 310 is further configured to: The multimodal data is decomposed based on the first central processing unit to determine the audio and video data and the original text; Based on the first neural network unit, text processing is performed on the audio and video data to determine the audio and video text; The text data is determined based on the original text and the audio / video text.

[0078] Optionally, the data processing module 310 is further configured to: The first central processing unit allocates cockpit system memory to the first neural network unit; Once the cockpit system memory allocation is complete, the first neural network unit encodes the audio and video data based on the cockpit system memory to determine the first embedded feature vector.

[0079] Optionally, the inference accelerator includes a second central processing unit and a second neural network unit; Optionally, the data processing module 310 is further configured to: When the second central processing unit receives the text data, the second central processing unit calls the inference onboard memory and the second neural network unit to encode the text data and determine the second embedded feature vector; The second central processing unit sends the second embedded feature vector to the main processor through the high-speed hardware transmission line.

[0080] The first embedded feature vector and the second embedded feature vector are cross-modal aligned to determine a multimodal vector set; The multimodal vector set is fused along the vector dimension based on a preset vector fusion method to determine the defined feature fusion vector.

[0081] Optionally, the end-side large model module 330 is specifically used for: Based on the main processor, the feature fusion vector is converted into a semantic word sequence, and the semantic word sequence is submitted to the local cockpit model of the smart cockpit; When the local cockpit model receives the semantic lexical sequence, it understands and generates based on the user requirements submitted by the smart cockpit and the semantic lexical sequence, and determines the model generation result corresponding to the user requirements.

[0082] The inference acceleration device for the local large model of the intelligent cockpit provided in the embodiments of the present invention can execute the inference acceleration method for the local large model of the intelligent cockpit provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0083] Figure 4 A schematic diagram of a cockpit accelerator implementing an embodiment of the present invention is disclosed. For example... Figure 4 As shown, the cockpit accelerator includes: a main processor and an inference accelerator; The main processor and the inference accelerator are connected by a high-speed hardware transmission line for data communication. The main processor consists of a codec, a digital signal processor, a first central processing unit, a modem, and a first neural network unit; the inference accelerator consists of a second central processing unit, a second neural network unit, and inference onboard memory.

[0084] Optionally, the main processor's encoding and decoding processing unit consists of a codec and a DSP (Digital Signal Processor). The first central processing unit is the CPU in the main processor, and the modem is a 5G Modem (5G Modulator-Demodulator). The first neural network unit is a 60TOPSNPU. The main processor is also equipped with a GPU. The first neural network unit is used to process audio and video data in multimodal data. DDR (Double Data Rate), i.e., cockpit system memory, is also configured outside the main processor.

[0085] Optionally, the second central processing unit is the CPU of the inference accelerator, and the second neural network unit is a 16TOPSNPU. The second neural network unit is used to process text data in multimodal data. The inference accelerator is also equipped with Embedded DDR (Embedded Double Data Rate Synchronous Dynamic Random Access Memory), which is the inference onboard memory.

[0086] A high-speed hardware transmission line is set up between the main processor and the inference accelerator to exchange data.

[0087] The cockpit accelerator provided in this embodiment of the invention can execute the inference acceleration method of the local large model of the intelligent cockpit provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the inference acceleration method steps for a local large model of an intelligent cockpit as provided in any embodiment of the present invention. The method includes: The main processor and inference accelerator of the cockpit accelerator encode multimodal data to determine a first embedded feature vector and a second embedded feature vector. Based on the main processor, the first embedded feature vector and the second embedded feature vector are fused to determine the feature fusion vector; The main processor submits the feature fusion vector to the local cockpit model of the smart cockpit for understanding and generation, and determines the model generation result.

[0089] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0090] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0091] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0092] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of mesh, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a grid of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0094] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0095] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for accelerating inference using a local large-scale model of an intelligent cockpit, characterized in that, Applications in cockpit accelerators include: The main processor and inference accelerator of the cockpit accelerator encode multimodal data to determine a first embedded feature vector and a second embedded feature vector. Based on the main processor, the first embedded feature vector and the second embedded feature vector are fused to determine the feature fusion vector; The main processor submits the feature fusion vector to the local cockpit model of the smart cockpit for understanding and generation, and determines the model generation result.

2. The method according to claim 1, characterized in that, The main processor and inference accelerator based on the cockpit accelerator encode multimodal data to determine a first embedded feature vector and a second embedded feature vector, including: The main processor decomposes and extracts the multimodal data to determine audio / video data and text data. The main processor transmits the text data to the inference accelerator via a high-speed hardware transmission line. The first embedded feature vector is determined by encoding the audio and video data based on the main processor. The text data is encoded using the inference accelerator to determine the second embedded feature vector.

3. The method according to claim 2, characterized in that, The main processor includes a first central processing unit and a first neural network unit; the step of decomposing and extracting the multimodal data based on the main processor to determine the audio / video data and the text data includes: The multimodal data is decomposed based on the first central processing unit to determine the audio and video data and the original text; Based on the first neural network unit, text processing is performed on the audio and video data to determine the audio and video text; The text data is determined based on the original text and the audio / video text.

4. The method according to claim 3, characterized in that, The step of encoding the audio and video data based on the main processor to determine the first embedded feature vector includes: The first central processing unit allocates cockpit system memory to the first neural network unit; Once the cockpit system memory allocation is complete, the first neural network unit encodes the audio and video data based on the cockpit system memory to determine the first embedded feature vector.

5. The method according to claim 3, characterized in that, The inference accelerator includes a second central processing unit and a second neural network unit; the process of encoding the text data based on the inference accelerator to determine the second embedded feature vector includes: When the second central processing unit receives the text data, the second central processing unit calls the inference onboard memory and the second neural network unit to encode the text data and determine the second embedded feature vector; The second central processing unit sends the second embedded feature vector to the main processor through the high-speed hardware transmission line.

6. The method according to claim 1, characterized in that, The step of fusing the first embedded feature vector and the second embedded feature vector based on the main processor to determine the feature fusion vector includes: The first embedded feature vector and the second embedded feature vector are cross-modal aligned to determine a multimodal vector set; The multimodal vector set is fused along the vector dimension based on a preset vector fusion method to determine the defined feature fusion vector.

7. The method according to claim 1, characterized in that, The process of submitting the feature fusion vector to the local cockpit model of the smart cockpit based on the main processor for understanding and generation, and determining the model generation result, includes: Based on the main processor, the feature fusion vector is converted into a semantic word sequence, and the semantic word sequence is submitted to the local cockpit model of the smart cockpit; When the local cockpit model receives the semantic lexical sequence, it understands and generates based on the user requirements submitted by the smart cockpit and the semantic lexical sequence, and determines the model generation result corresponding to the user requirements.

8. An inference acceleration system for a local large-scale model of an intelligent cockpit, characterized in that, Deployed in the cockpit accelerator, including: The data processing module is used to encode multimodal data based on the main processor and inference accelerator of the cockpit accelerator, and determine the first embedded feature vector and the second embedded feature vector. The feature vector fusion module is used to perform vector fusion of the first embedded feature vector and the second embedded feature vector based on the main processor to determine the feature fusion vector; The edge-side large model module is used to understand and generate the local cockpit model of the smart cockpit based on the feature fusion vector submitted by the main processor, and to determine the model generation result.

9. A cockpit accelerator, characterized in that, The cockpit accelerator includes: a main processor and an inference accelerator; The main processor and the inference accelerator are used to execute the inference acceleration method for the local large model of the intelligent cockpit as described in any one of claims 1-7.

10. The cockpit accelerator according to claim 9, characterized in that, include: The main processor and the inference accelerator are connected by a high-speed hardware transmission line for data communication. The main processor consists of a codec processing unit, a first central processing unit, a modem, and a first neural network unit. The inference accelerator consists of a second central processing unit, a second neural network unit, and inference onboard memory.

Images