Vehicle-mounted voice interaction data processing method and system

Abstract

The application provides a vehicle-mounted voice interaction data processing method and system, comprising: obtaining a voice instruction and its corresponding instruction text, obtaining a corresponding voice query vector and a text query vector; determining a dynamic similarity threshold according to a voice quality parameter; determining a first candidate vector set according to a first similarity of the voice query vector and each reference voice vector, and calculating a second similarity; if the first candidate vector set is not empty, determining the reference voice vector with the maximum first similarity as a target matching vector; otherwise, determining the reference text vector with the maximum second similarity as the target matching vector; and generating a vehicle control instruction according to vehicle function operation information associated with the target matching vector. In the method, the similarity threshold is dynamically adjusted according to the voice quality, and the voice vector and the text vector are matched cooperatively, which can adapt to changes in the acoustic environment in the vehicle, thereby reducing false matching and missed matching, and further improving the accuracy of the vehicle control instruction.

Description

Technical Field

[0001] This application relates to the field of vehicle networking technology, and in particular to a method and system for processing in-vehicle voice interaction data. Background Technology

[0002] With the development of intelligent connected vehicle technology, in-vehicle voice interaction has been widely used in control scenarios for vehicle functions such as air conditioning adjustment, navigation control, and audio-visual playback. During vehicle operation, interference from engine noise, wind noise, and passenger conversations can alter the quality of collected voice commands, placing higher demands on the accuracy of matching voice commands with vehicle functions.

[0003] Existing technologies typically encode the features of acquired voice commands and use voice query vectors to retrieve corresponding vehicle functions from a vector database, or first transcribe the voice commands into text and then perform semantic retrieval based on the text query vectors. However, when using voice query vectors for retrieval, a fixed similarity threshold is difficult to adapt to changes in voice clarity and noise levels, easily leading to missed or false matches. When using text query vectors for retrieval, textual biases generated during voice transcription can be passed on to subsequent retrieval processes, affecting the matching results for vehicle functions. Furthermore, existing technologies lack a mechanism for collaboratively controlling voice vector retrieval and text vector retrieval based on voice quality, making it difficult to fully utilize the semantic complementarity between the two types of vectors.

[0004] In summary, existing in-vehicle voice interaction data processing methods struggle to coordinate the use of voice and text information based on changes in voice quality, resulting in low accuracy in matching voice commands with vehicle functions in complex in-vehicle acoustic environments. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method and system for processing in-vehicle voice interaction data. By dynamically adjusting the similarity threshold according to the voice quality and coordinating the matching of voice vectors and text vectors, it can adapt to changes in the in-vehicle acoustic environment, thereby reducing mismatches and missed matches, and thus improving the accuracy of vehicle control commands.

[0006] In a first aspect, the present invention provides a method for processing in-vehicle voice interaction data, applied to an in-vehicle terminal equipped with an in-vehicle multimodal vector database. The in-vehicle multimodal vector database stores multiple sets of reference data, each set of reference data including semantically related reference voice vectors and reference text vectors, as well as in-vehicle function operation information associated with the reference voice vectors and reference text vectors; the method includes: Acquire the voice commands collected by the in-vehicle voice interaction module and the corresponding command text.

[0007] The voice command and the command text are respectively encoded to obtain the voice query vector and the text query vector.

[0008] The dynamic similarity threshold is determined based on the speech quality parameters of the speech command.

[0009] Calculate the first similarity between the speech query vector and each reference speech vector. Form a first candidate vector set by referring speech vectors whose first similarity is greater than or equal to the dynamic similarity threshold. Calculate the second similarity between the text query vector and each reference text vector.

[0010] If the first candidate vector set is not empty, the reference speech vector with the highest similarity is determined as the target matching vector; otherwise, the reference text vector with the highest similarity is determined as the target matching vector.

[0011] Vehicle control commands are generated based on the vehicle function operation information associated with the target matching vector.

[0012] In an optional implementation, feature encoding is performed on the voice command and the command text respectively to obtain a voice query vector and a text query vector, including: The acoustic and semantic features of the speech commands are extracted to obtain the initial speech vector.

[0013] Extract the semantic features of the instruction text to obtain the initial text vector.

[0014] The initial speech vector and initial text vector are mapped to the common semantic space where the reference speech vector and reference text vector are located by a preset semantic alignment algorithm, respectively, to obtain the speech query vector and text query vector.

[0015] In an optional implementation, the speech quality parameters include speech intelligibility and noise interference level.

[0016] The dynamic similarity threshold is determined based on the speech quality parameters of the speech command, including: The quality of voice commands is evaluated to obtain the speech clarity and noise interference level of the voice commands.

[0017] The speech clarity and noise interference levels are matched with corresponding preset parameter ranges to obtain the speech clarity level and noise interference level.

[0018] Combine the speech clarity level and noise interference level into a target level combination.

[0019] Based on the pre-defined correspondence between level combinations and similarity thresholds, the dynamic similarity threshold corresponding to the target level combination is determined. Among them, the higher the speech clarity level, the higher the speech clarity; the higher the noise interference level, the greater the noise interference. Under the same noise interference level, the dynamic similarity threshold increases with the increase of speech clarity level; under the same speech clarity level, the dynamic similarity threshold decreases with the increase of noise interference level.

[0020] In an optional implementation, the vehicle-mounted multimodal vector database is constructed using the following method: Create a vector storage structure to store multiple sets of reference data.

[0021] Acquire multiple reference voice samples, reference instruction texts corresponding to each reference voice sample, and vehicle function operation information; wherein, the multiple reference voice samples include different accent voice samples corresponding to the same vehicle function operation information, and the reference instruction texts include complete instruction texts and simplified instruction texts corresponding to the same vehicle function operation information; the simplified instruction texts are instruction texts that omit the wake word and / or omit some modifiers.

[0022] Each reference speech sample and its corresponding reference instruction text are feature-encoded to obtain the reference speech vector for each reference speech sample and the reference text vector for each reference instruction text.

[0023] Based on the semantic similarity between the reference speech vector and the reference text vector, a semantic association relationship is established between the corresponding reference speech vector and the reference text vector.

[0024] Each set of semantically related reference speech vectors and reference text vectors is associated with the corresponding in-vehicle function operation information to form a set of reference data.

[0025] The generated sets of reference data are written into a vector storage structure to obtain an in-vehicle multimodal vector database.

[0026] In an optional implementation, before associating each set of semantically related reference speech vectors and reference text vectors with corresponding in-vehicle function operation information to form a set of reference data, the method further includes: Construct an objective function to optimize the vector distribution based on the vector distribution of the reference speech vector and the reference text vector.

[0027] Set the regularization coefficient, iteration step size, and convergence threshold. Iterate and update the reference speech vector and reference text vector according to the objective function, and calculate the change between the objective function values ​​corresponding to two adjacent iterations.

[0028] When the change is greater than the convergence threshold, adjust the regularization coefficient and iteration step size and continue iterating; when the change is less than or equal to the convergence threshold, stop iterating.

[0029] The updated reference speech vectors and reference text vectors are quantized, and the quantized reference speech vectors and reference text vectors are used to form the corresponding reference data.

[0030] In an optional implementation, after writing the formed sets of reference data into a vector storage structure to obtain an in-vehicle multimodal vector database, the method further includes: A tree index is constructed based on the reference speech vector and the reference text vector.

[0031] Obtain the retrieval frequency of each reference speech vector and each reference text vector, and construct a hash index for the reference speech vectors and reference text vectors whose retrieval frequency reaches the preset frequency condition.

[0032] Based on the semantic relationship between the reference speech vector and the reference text vector, as well as their relationship with the vehicle function operation information, a semantic relationship index is constructed between the reference speech vector, the reference text vector, and the vehicle function operation information.

[0033] When reference speech vectors, reference text vectors, or vehicle function operation information are inserted, deleted, or updated, the affected indexes in the tree index, hash index, and semantic association index are incrementally updated.

[0034] In optional implementations, the vector storage architecture includes local caching, on-board solid-state storage, and cloud storage.

[0035] The resulting sets of reference data are written into a vector storage structure, including: Each set of reference data is designated as a separate set of stored data.

[0036] Determine the data sensitivity level of each group of stored data, and write each group of stored data into the corresponding storage partition in the vehicle solid-state storage according to the data sensitivity level. Among them, the stored data with a data sensitivity level that reaches the preset sensitivity level is encrypted and then written into the corresponding encrypted storage partition.

[0037] Obtain the retrieval frequency of each group of stored data, and load the stored data whose retrieval frequency reaches the preset frequency condition into the local cache.

[0038] Data that meets the preset backup conditions is identified as data to be synchronized, and the data to be synchronized is synchronized to cloud storage.

[0039] The method also includes: In response to a data access request for stored data, the system performs access permission verification on the requesting module that initiated the data access request. If the verification passes, the system provides the requested stored data to the requesting module. If the verification fails, the system rejects the data access request and records the rejected access event.

[0040] In an optional implementation, the data to be synchronized is synchronized to cloud storage, including: Detect the access point name, channel status, network connection status, network bandwidth, and over-the-air (OTA) upgrade status of the vehicle network.

[0041] When the network connection status is connected, the target access point name channel is determined from the available access point name channels based on the data type, data sensitivity level, and access point name channel status of the data to be synchronized, and the transmission encryption method is determined according to the data sensitivity level.

[0042] The synchronization frequency of the data to be synchronized is determined based on the network bandwidth.

[0043] When the over-the-air download upgrade status indicates that an over-the-air download upgrade task is being performed, the data to be synchronized is divided into data required for basic voice interaction and other data to be synchronized. The transmission priority of other data to be synchronized is reduced, while network bandwidth resources are reserved for data required for basic voice interaction.

[0044] The network address stored in the cloud is resolved through the vehicle's intranet domain name resolution service, and the data to be synchronized is transmitted through the target access point name channel according to the synchronization frequency, transmission encryption method and corresponding transmission priority.

[0045] When the network connection is interrupted, the untransmitted data to be synchronized is stored in the vehicle's solid-state storage and the transmission breakpoint is recorded. After the vehicle network is restored, the target access point name, channel and synchronization frequency are re-determined, and the data to be synchronized is transmitted again according to the transmission breakpoint.

[0046] In an optional implementation, the in-vehicle multimodal vector database also stores multiple user voice habit vectors corresponding to the current user, and each user voice habit vector is associated with the corresponding in-vehicle function operation information.

[0047] User voice habit vectors are generated or updated through the following steps.

[0048] Obtain the current user's historical voice commands within a preset historical time period, as well as the historical in-vehicle function operation information corresponding to each historical voice command.

[0049] Based on the method of feature encoding of voice commands, feature encoding is performed on each historical voice command to obtain a historical voice vector.

[0050] The historical voice vectors are grouped according to the historical in-vehicle function operation information, and each group of historical voice vectors is aggregated to obtain the updated user voice habit vector associated with each historical in-vehicle function operation information.

[0051] For each updated user voice habit vector, if a user voice habit vector associated with the same in-vehicle function operation information exists in the in-vehicle multimodal vector database, the updated user voice habit vector is used to replace the user voice habit vector; if no user voice habit vector associated with the same in-vehicle function operation information exists, the updated user voice habit vector is added to the in-vehicle multimodal vector database and associated with the corresponding in-vehicle function operation information.

[0052] Calculate the first similarity between the speech query vector and each reference speech vector, including: Calculate the basic speech similarity between the speech query vector and each reference speech vector.

[0053] For each reference voice vector, query whether there exists a user voice habit vector that is associated with the same in-vehicle function operation information as that reference voice vector.

[0054] If it exists, calculate the habit similarity between the voice query vector and the queried user voice habit vector, and perform a weighted calculation on the basic voice similarity and habit similarity according to the preset weights to obtain the first similarity corresponding to the reference voice vector.

[0055] If it does not exist, the basic speech similarity will be used as the first similarity corresponding to the reference speech vector.

[0056] Secondly, the present invention provides an in-vehicle voice interaction data processing system, applied to an in-vehicle terminal equipped with an in-vehicle multimodal vector database. The in-vehicle multimodal vector database stores multiple sets of reference data, each set of reference data including semantically related reference voice vectors and reference text vectors, as well as in-vehicle function operation information associated with the reference voice vectors and reference text vectors; the system includes: The data acquisition module is used to acquire the voice commands collected by the in-vehicle voice interaction module and the corresponding command text.

[0057] The feature encoding module is used to encode the features of the voice command and the command text respectively, so as to obtain the voice query vector and the text query vector.

[0058] The threshold determination module is used to determine the dynamic similarity threshold based on the speech quality parameters of the speech command.

[0059] The vector matching module is used to calculate the first similarity between the speech query vector and each reference speech vector, form a first candidate vector set by the reference speech vectors whose first similarity is greater than or equal to the dynamic similarity threshold, and calculate the second similarity between the text query vector and each reference text vector.

[0060] The vector matching module is further configured to determine the reference speech vector with the highest first similarity as the target matching vector if the first candidate vector set is not empty; otherwise, determine the reference text vector with the highest second similarity as the target matching vector.

[0061] The instruction generation module is used to generate vehicle control instructions based on the vehicle function operation information associated with the target matching vector.

[0062] This application provides a method and system for processing in-vehicle voice interaction data. By storing reference voice vectors, reference text vectors, and in-vehicle function operation information in an in-vehicle multimodal vector database, a unified matching basis for voice and text information can be provided. By generating voice query vectors and text query vectors separately, the complementarity of the two modalities can be fully utilized. By determining a dynamic similarity threshold based on voice quality parameters, the adaptability of the matching process to changes in in-vehicle voice quality can be improved. By prioritizing the use of voice matching results when voice vector matching is successful and using text matching results when voice vector matching is unsuccessful, the matching success rate and accuracy of voice commands can be improved, thereby enhancing the robustness of in-vehicle voice interaction in complex in-vehicle acoustic environments and improving the accuracy and reliability of vehicle control command generation.

[0063] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application are realized and obtained through the structures particularly pointed out in the description, claims and drawings.

[0064] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0066] Figure 1A flowchart of the in-vehicle voice interaction data processing method provided in the embodiments of this application; Figure 2 Flowchart of the method for obtaining voice query vectors and text query vectors provided in the embodiments of this application; Figure 3 Flowchart of the dynamic similarity threshold determination method provided in the embodiments of this application; Figure 4 A schematic diagram of an in-vehicle voice interaction data processing system provided in an embodiment of this application.

[0067] Icons: 1-Data acquisition module; 2-Feature encoding module; 3-Threshold determination module; 4-Vector matching module; 5-Instruction generation module. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] To help those skilled in the art better understand this application, a brief introduction to its application scenarios and design concepts is provided.

[0070] During vehicle operation, voice commands are easily interfered with by factors such as engine noise, wind noise, road noise, and passenger conversations. Furthermore, different users may have different accents and simplify their expressions by omitting wake words or some modifiers, leading to significant variations in the clarity, completeness, and recognition results of voice commands. Existing in-vehicle voice interaction technologies typically employ speech feature matching or convert speech to text for text matching, using a fixed similarity threshold to determine the matching result. This makes it difficult to reasonably adjust matching conditions based on changes in speech quality. When speech is affected by noise or accent deviations, speech features are prone to shift, and transcription errors may occur in the command text. Relying solely on any one type of information can easily affect the accuracy of vehicle function matching. Moreover, existing solutions often lack a unified semantic relationship between voice data, text data, and in-vehicle function operation information, making it difficult to supplement matching with text information when voice matching fails.

[0071] Based on this, this application establishes a correlation between reference speech vectors, reference text vectors, and in-vehicle function operation information in an in-vehicle multimodal vector database, and encodes the current voice command and its corresponding command text into speech query vectors and text query vectors, respectively, thereby providing a unified data foundation for the collaborative matching of speech and text information. This application also dynamically determines a similarity threshold based on speech quality parameters, prioritizing the use of speech matching results when the speech vector meets the matching conditions, and supplementing with text matching results when the speech vector does not meet the matching conditions. This adapts to changes in the in-vehicle acoustic environment and speech quality, fully utilizing the complementarity of speech and text information, and improving the success rate and accuracy of matching between voice commands and in-vehicle function operation information.

[0072] Furthermore, this application enhances its adaptability to accent differences and simplified expressions by constructing reference data corresponding to the same in-vehicle function operation information through the collection of different accented speech samples, complete command text, and simplified command text. By optimizing the distribution, quantizing, constructing composite indexes, and incrementally updating the reference speech and text vectors, it improves vector storage and retrieval performance. By combining historical user voice commands to generate user voice habit vectors, it adapts to the pronunciation habits of different users, improving personalized matching capabilities. By hierarchically storing, encrypting, verifying access permissions, and synchronizing the reference data to the cloud, and adjusting the synchronization process according to the in-vehicle network status and over-the-air (OTA) upgrade status, it improves the data security, access efficiency, and operational reliability of the in-vehicle multimodal vector database. Therefore, this application can generate vehicle control commands more accurately and stably in in-vehicle voice scenarios with noise interference, accent differences, and simplified commands.

[0073] To facilitate understanding of this embodiment, the embodiments of this application will be described in detail below.

[0074] This application provides a method and system for processing in-vehicle voice interaction data, which is applied to an in-vehicle terminal equipped with an in-vehicle multimodal vector database. The in-vehicle multimodal vector database stores multiple sets of reference data. Each set of reference data includes a reference voice vector and a reference text vector with semantic association, as well as in-vehicle function operation information that is associated with the reference voice vector and the reference text vector.

[0075] Here, the in-vehicle terminal is an electronic device deployed inside the vehicle, possessing audio data processing, vector computation, data storage, and vehicle communication capabilities. The in-vehicle terminal includes a processor, memory, an audio acquisition interface, and a vehicle communication interface. The processor runs the in-vehicle voice interaction data processing program and completes feature encoding, quality assessment, vector matching, and vehicle control command generation. The memory stores the in-vehicle multimodal vector database, feature encoding model, threshold configuration data, and program execution data. The audio acquisition interface connects to the in-vehicle voice interaction module to receive voice commands and command text. The vehicle communication interface connects to the vehicle control network to provide the generated vehicle control commands to the corresponding vehicle function control modules. The in-vehicle terminal can be an in-vehicle infotainment host, a smart cockpit domain controller, or an in-vehicle computing device with corresponding processing capabilities.

[0076] The in-vehicle voice interaction module includes a microphone, an audio front-end processing unit, and a speech recognition unit. The microphone can be a single in-vehicle microphone or a microphone array positioned at different locations within the vehicle. The audio front-end processing unit performs echo cancellation, beamforming, noise reduction, and voice activity detection on the acquired in-vehicle audio, and extracts audio segments containing the user's control intent from the continuous audio stream. The speech recognition unit performs speech recognition on the audio segments and generates instruction text corresponding to the voice commands. The in-vehicle voice interaction module can be integrated into the in-vehicle terminal or connected to the in-vehicle terminal as a standalone module via an in-vehicle communication network.

[0077] The in-vehicle multimodal vector database is a vectorized data storage structure oriented towards both voice and text modalities. The in-vehicle multimodal vector database can be deployed entirely in the in-vehicle terminal's storage, or it can be deployed using a combination of local storage on the in-vehicle terminal and cloud backup. The in-vehicle terminal retains the reference data required for voice interaction even without a network connection, and reads reference voice vectors, reference text vectors, and in-vehicle function operation information through a database query interface. Reference voice vectors are numerical sequences formed by encoding the voice features of reference voice samples, and reference text vectors are numerical sequences formed by encoding the text features of reference command text. Each set of reference data is assigned a unique reference data identifier, and semantically related reference voice vectors and reference text vectors are recorded in the same set of reference data. Semantic association indicates that the reference voice sample and the reference command text express the same or similar vehicle operation intent, rather than simply indicating that the reference voice vector and reference text vector are adjacent in storage location.

[0078] In-vehicle function operation information describes the vehicle functions that need to be triggered by voice interaction and the corresponding operation content. This information includes at least one of the following: vehicle function identifier, target control module identifier, operation type, control parameters, parameter unit, and parameter allowable range. Vehicle functions include air conditioning control, window control, seat control, navigation control, and audio / video control. Operation types include turning on, turning off, increasing, decreasing, switching, and setting target values. Reference voice vectors and reference text vectors are linked to the same in-vehicle function operation information through associated fields, ensuring that the target matching vector for any modality can be mapped to a specific vehicle operation.

[0079] In one implementation, the same vehicle function operation information corresponds to multiple sets of reference data, and different sets of reference data are used to represent different voice or text expressions corresponding to the same vehicle operation.

[0080] Reference Figure 1 The in-vehicle voice interaction data processing method provided in this application includes: Step S101: Obtain the voice commands collected by the vehicle voice interaction module and the corresponding command text.

[0081] Here, the in-vehicle voice interaction module collects in-vehicle audio and extracts voice commands from it. Voice collection can be triggered by a preset wake-up word, steering wheel buttons, touch controls, or continuous voice activity detection. The in-vehicle voice interaction module uses the voice activity detection results to determine the start and end positions of the voice, and identifies the audio segment between these two positions as the voice command. The in-vehicle voice interaction module inputs the voice command into a voice recognition model to obtain the command text composed of characters or words. The voice recognition model can be deployed locally on the in-vehicle terminal or on a voice server connected to the in-vehicle terminal. The in-vehicle voice interaction module assigns the same interaction session identifier to the same voice command and command text, and records the corresponding collection time, thereby ensuring that the voice command and command text originate from the same audio segment and the same voice interaction.

[0082] Step S102: Perform feature encoding on the voice command and the command text respectively to obtain the voice query vector and the text query vector.

[0083] Here, speech feature encoding and text feature encoding models are used to encode the speech commands and command text, respectively. The speech feature encoding model receives the speech commands processed by the audio front-end and converts the variable-length audio data into a fixed-dimensional speech query vector. The text feature encoding model performs word segmentation or character segmentation on the command text and converts the resulting text unit sequence into a fixed-dimensional text query vector. The speech feature encoding model maintains the same model structure, model parameter version, and output dimension as the model used to generate the reference speech vector, and the text feature encoding model maintains the same model structure, model parameter version, and output dimension as the model used to generate the reference text vector. The speech query vector and the reference speech vector reside in the same speech vector space, and the text query vector and the reference text vector reside in the same text vector space. After feature encoding, the speech query vector and the text query vector are normalized to ensure that the similarity calculation results corresponding to different speech commands are comparable.

[0084] Step S103: Determine the dynamic similarity threshold based on the speech quality parameters of the speech command.

[0085] Here, the voice commands are quality-assessed, and the dynamic similarity threshold used in this voice interaction is determined based on the assessment results. Voice quality parameters include at least speech clarity and noise interference level. Speech clarity characterizes the recognizability of the speech content, while noise interference level characterizes the impact of non-target sounds on the voice commands.

[0086] A pre-established correspondence between voice quality parameters and similarity thresholds is created using sample data from an in-vehicle environment, and this correspondence is stored as threshold configuration data. After obtaining the voice quality parameters of the current voice command, a matching dynamic similarity threshold is retrieved from the threshold configuration data.

[0087] The dynamic similarity threshold is determined individually for each voice interaction, rather than applying the same fixed threshold to voice commands of varying quality. A higher dynamic similarity threshold is used when the voice clarity is high to increase the stringency of the voice matching conditions; a lower dynamic similarity threshold is used when the noise interference is high to increase the tolerance of voice matching for acoustic feature shifts. The dynamic similarity threshold uses the same numerical range as the first similarity threshold.

[0088] Step S104: Calculate the first similarity between the speech query vector and each reference speech vector, form a first candidate vector set with reference speech vectors whose first similarity is greater than or equal to the dynamic similarity threshold, and calculate the second similarity between the text query vector and each reference text vector.

[0089] Here, speech vector similarity calculation and text vector similarity calculation are performed separately. The first similarity represents the degree of similarity between the speech query vector and a reference speech vector, and the second similarity represents the degree of similarity between the text query vector and a reference text vector.

[0090] In one implementation, both the first similarity and the second similarity are calculated using cosine similarity, with higher values ​​indicating greater similarity between the corresponding vectors. Reference speech vectors are read in batches from the in-vehicle multimodal vector database. The first similarity between the speech query vector and each reference speech vector is calculated, and each first similarity is compared to a dynamic similarity threshold. When the first similarity is greater than or equal to the dynamic similarity threshold, the corresponding reference speech vector is added to a first candidate vector set, and the reference speech vector identifier, the first similarity, and the associated in-vehicle function operation information identifier are recorded in the first candidate vector set. When the first similarity is less than the dynamic similarity threshold, the corresponding reference speech vector is not added to the first candidate vector set.

[0091] Calculate the second similarity between the text query vector and each reference text vector, and record the second similarity corresponding to each reference text vector. Speech vector similarity calculation and text vector similarity calculation can be performed in parallel, or in the order of speech vector first, followed by text vector. Both execution methods maintain the first similarity, second similarity, and association relationships unchanged. A dynamic similarity threshold is used to filter reference speech vectors, and the second similarity is used to provide a basis for text matching when the first candidate vector set is empty.

[0092] Step S105: If the first candidate vector set is not empty, the reference speech vector with the highest first similarity is determined as the target matching vector; otherwise, the reference text vector with the highest second similarity is determined as the target matching vector.

[0093] Here, the system first determines whether a reference speech vector exists in the first candidate vector set. If the first candidate vector set is not empty, the first similarity scores in the first candidate vector set are compared, and the reference speech vector with the highest first similarity score is determined as the target matching vector. If multiple reference speech vectors have the same highest first similarity score, a reference speech vector is selected as the target matching vector according to a pre-set data priority. If the first candidate vector set is empty, the second similarity scores are compared, and the reference text vector with the highest second similarity score is determined as the target matching vector. The recorded content of the target matching vector includes the vector type, vector identifier, corresponding similarity score, and associated vehicle function operation information identifier. When the speech vector matching reaches the dynamic similarity threshold, the speech matching result is used first, which can preserve the matching evidence provided by the speech acoustic features and speech semantic features; when the speech vector matching does not reach the dynamic similarity threshold, the text matching result is used, which can use the semantic information expressed by the command text to complete the supplementary matching.

[0094] Step S106: Generate vehicle control commands based on the vehicle function operation information associated with the target matching vector.

[0095] Here, the corresponding vehicle function operation information is queried based on the vehicle function operation information identifier carried by the target matching vector, and the vehicle function operation information is converted into vehicle control commands that can be recognized by the vehicle function control module. The vehicle control command generation process includes reading the target control module identifier and operation type, generating parameter fields according to the data type and allowable range of control parameters, generating command identifiers and data fields according to the vehicle communication protocol, and encapsulating the command identifiers and data fields. Vehicle control commands can be application programming interface call messages within the smart cockpit, or control messages encapsulated based on Controller Area Network (CAN), CAN Flexible Data Rate Protocol (FDR), or vehicle Ethernet protocol.

[0096] After the vehicle control command is generated, the on-board terminal sends the command to the target control module via the vehicle communication interface. The target control module parses the command and executes the corresponding vehicle function operation. For example, if the on-board function operation information associated with the target matching vector indicates that the air conditioning is to be turned on, the generated vehicle control command will include at least the air conditioning control module identifier and the start operation code. Upon receiving the command, the air conditioning control module will activate the air conditioning.

[0097] The in-vehicle voice interaction data processing method provided in this application converts the voice information and text information corresponding to the same voice interaction into two types of query vectors, sets voice vector matching conditions according to the current voice quality, and supplements the voice matching results with text matching results when the voice matching results are insufficient, thereby forming a complete data processing flow from voice acquisition, feature encoding, quality adaptation, bimodal matching to vehicle control command generation.

[0098] In an optional implementation, refer to Figure 2 Step S102 includes the following steps S201-S203. Steps S201 and S202 can be executed in parallel or sequentially.

[0099] Step S201: Extract the acoustic and semantic features of the speech command to obtain the initial speech vector.

[0100] Here, the voice command refers to the valid speech segment after audio preprocessing. The audio preprocessing process includes using Wiener filtering to remove engine noise and wind noise, using Normalized Least Mean Squares (NLMS) to eliminate speech echo, and using an endpoint detection algorithm combining short-time energy and zero-crossing rate to determine valid speech segments. Valid speech segments are mapped to the [0, 1] interval using a minimum-to-maximum normalization algorithm, with a uniform sampling rate of 16 kHz and a quantization bit depth of 16 bits.

[0101] Acoustic features are used to reflect the characteristics of speech commands in terms of frequency, energy, timbre, and rhythm, while semantic features are used to reflect the vehicle function, operation type, and target of the speech command. Speech feature extractors can employ Mel-CNN (Mel Convolutional Neural Network) or Mel-Frequency Cepstral Coefficients (MFCC) feature extractors.

[0102] When using Mel-CNN to extract features, the speech command is first divided into multiple audio frames according to a preset frame length and frame shift. In one implementation, the preset frame length is 25ms and the preset frame shift is 10ms. Windowing and spectral transformation are applied to each audio frame, and the spectral transformation results are input into a Mel filter bank to obtain spectral energy arranged according to Mel frequencies. The spectral energy is then logarithmically processed to form a Mel spectral feature matrix. The Mel spectral feature matrix is ​​input into a convolutional neural network that has undergone pruning and quantization. The convolutional layers extract local variation features between adjacent time periods and adjacent frequency bands, the pooling layers compress the feature matrix and retain significant acoustic features, and the semantic feature extraction layer extracts high-level features related to vehicle functions and operational intentions based on pre-trained weights. The acoustic and semantic features are concatenated, and the concatenation result is converted into a 128-dimensional initial speech vector through a dimension mapping layer.

[0103] When extracting features using the MFCC feature extractor, the audio frames that have undergone framing and windowing are sequentially subjected to spectral transformation, Mel filtering, logarithmic processing, and cepstral transformation to obtain an MFCC feature sequence arranged in chronological order. This MFCC feature sequence is then input into a lightweight semantic feature extraction network to obtain semantic features. The MFCC features and semantic features are then concatenated and their dimensions mapped to obtain a 128-dimensional initial speech vector. The speech feature extractor loads pre-trained weights upon startup of the in-vehicle terminal. Pre-training samples include reference speech samples corresponding to different noise environments, accents, and expressions, including Mandarin, Cantonese, and Sichuanese. Reference speech samples with the same in-vehicle function operation information are used to learn the same or similar semantic representations. The speech feature extractor also loads updated model parameters based on user speech habits and in-vehicle scenario requirements. The updated model parameters are synchronized with the corresponding version of the reference speech vector before being used, avoiding the use of different encoding parameters between the query vector and the reference speech vector. After pruning and quantization, the speech encoding latency of the speech feature extractor on the in-vehicle embedded processor does not exceed 20ms.

[0104] Step S202: Extract the textual semantic features of the instruction text to obtain the initial text vector.

[0105] Here, the instruction text undergoes text preprocessing and semantic feature extraction. Text preprocessing includes unified character encoding, unifying full-width and half-width characters, removing consecutive spaces that do not affect the operational intent, and segmenting the instruction text into multiple text units according to a preset vocabulary. The preset vocabulary includes Chinese characters, English words, vehicle function names, operational terms, and commonly used parameter terms. The text encoding module generates a text unit vector and a position vector for each text unit, and adds the text unit vector to the corresponding position vector to obtain the text input sequence.

[0106] Text semantic feature extraction employs a lightweight large language model encoder derived from a Transformer architecture pruning. The lightweight large language model encoder retains six Transformer layers, each consisting of an eight-head self-attention layer and a simplified feedforward neural network. The self-attention layers assign feature weights based on the correlation between different text units in the instruction text, used to identify vehicle functions, operation types, operation objects, negation relations, and parameter information; the simplified feedforward neural network performs a non-linear mapping on the features output by the self-attention layers. After the final Transformer layer outputs the semantic representation corresponding to each text unit, average pooling is performed on each semantic representation, and a 128-dimensional initial text vector is generated through a dimension mapping layer. The text encoding model dynamically loads pre-trained weights upon startup of the in-vehicle terminal, maintaining consistency with the model parameter version used when generating the reference text vector. The overall latency between speech encoding and text encoding does not exceed 30ms.

[0107] Step S203: The initial speech vector and the initial text vector are mapped to the common semantic space where the reference speech vector and the reference text vector are located by a preset semantic alignment algorithm, respectively, to obtain the speech query vector and the text query vector.

[0108] Here, the common semantic space is a 128-dimensional semantic coordinate space shared by both speech vectors and text vectors. Each vector position in the common semantic space represents the vehicle operation semantics of the corresponding voice command or command text. Speech vectors and text vectors expressing the same in-vehicle function operation have a high degree of semantic similarity in the common semantic space, while speech vectors and text vectors expressing different in-vehicle function operations have a low degree of semantic similarity. The reference speech vectors and reference text vectors in the in-vehicle multimodal vector database are pre-mapped using a semantic alignment algorithm. Therefore, speech query vectors can perform similarity calculations with each reference speech vector, and text query vectors can perform similarity calculations with each reference text vector.

[0109] The pre-defined semantic alignment algorithm includes a speech mapping branch, a text mapping branch, an attention layer, and a normalization layer. The speech mapping branch maps the initial speech vectors to a common semantic space, and the text mapping branch maps the initial text vectors to the common semantic space. The attention layer assigns weights based on the correlation between the speech and text feature dimensions and the semantics of vehicle operation, ensuring that both mapping branches primarily retain features related to vehicle function, operation type, and operation object. The normalization layer normalizes the outputs of both mapping branches, ensuring that the speech query vector, text query vector, and corresponding reference vector are on the same numerical scale.

[0110] The parameters of the semantic alignment algorithm are obtained through training on reference data. Before training, reference speech samples and reference instruction texts are paired according to the vehicle function operation information. Reference speech samples and reference instruction texts associated with the same vehicle function operation information are formed into positive sample pairs, and reference speech samples and reference instruction texts associated with different vehicle function operation information are formed into negative sample pairs. Initial speech vectors and initial text vectors for training are generated separately. The initial speech vectors for training are then input into the speech mapping branch, and the initial text vectors for training are input into the text mapping branch. In each training round, the cosine similarity of positive and negative sample pairs in the common semantic space is calculated. The parameters of the speech mapping branch, text mapping branch, and attention layer are adjusted according to the cosine similarity to increase the cosine similarity of positive sample pairs and decrease the cosine similarity of negative sample pairs. The training process stops when the semantic matching accuracy corresponding to the validation data reaches the preset accuracy or the number of training rounds reaches the preset number of rounds. After training, the speech mapping branch parameters, text mapping branch parameters, attention layer parameters, normalization rules, and model version are saved together as the preset semantic alignment algorithm.

[0111] During online semantic alignment, a preset semantic alignment algorithm consistent with the version of the in-vehicle multimodal vector database is first loaded. Then, the initial speech vector is input into the speech mapping branch and attention weighting and normalization are performed to obtain a 128-dimensional speech query vector. Simultaneously, the initial text vector is input into the text mapping branch and attention weighting and normalization are performed to obtain a 128-dimensional text query vector. After mapping, the dimensions, numerical validity, and model version of the speech and text query vectors are checked. If the model version is inconsistent with the in-vehicle multimodal vector database, the matching model parameters are reloaded and step S203 is executed again. If the dimension of any query vector is not equal to 128 or there are invalid values ​​in the query vector, the current encoding result is discarded and steps S201-S203 are executed again. After the check passes, the speech query vector and text query vector are associated with the same interactive session identifier.

[0112] The vehicle-mounted voice interaction data processing method provided in this application preserves the acoustic differences in voice commands, and maps the vehicle operation semantics expressed by the voice commands and command text to a unified semantic coordinate space. This reduces the impact of noise, accent differences, and text expression differences on vector matching. Simultaneously, the 128-dimensional output and the pruned and quantized encoding model reduce the computational load and storage consumption of the vehicle terminal, which is beneficial for meeting the real-time processing requirements of vehicle-mounted voice interaction.

[0113] In an optional implementation, the speech quality parameters include speech intelligibility and noise interference level.

[0114] Reference Figure 3 Step S103 includes the following steps S301-S304.

[0115] Step S301: Evaluate the quality of the voice command to obtain the voice clarity and noise interference level of the voice command.

[0116] Here, speech intelligibility is used to characterize the ease with which speech content can be accurately identified. A higher speech intelligibility value indicates a more complete pronunciation and a clearer acoustic profile. Noise interference level is used to characterize the influence of engine noise, wind noise, road noise, and passenger conversations on speech commands. A higher noise interference level value indicates stronger interference from non-target sounds. Both speech intelligibility and noise interference level are normalized to the [0, 1] interval.

[0117] The process reads the voice command, the original audio segment retained before audio preprocessing, the endpoint detection results, and the speech recognition confidence score. Wiener filtering noise reduction, NLMS echo cancellation, and endpoint detection have already been explained, so the corresponding audio preprocessing steps will not be repeated here. The quality assessment module divides the original audio segment into effective speech frames and non-speech frames based on the endpoint detection results, calculates the average energy of the effective speech frames and the average energy of the non-speech frames, and determines the signal-to-noise ratio (SNR) based on the two average energies. If the original audio segment lacks a sufficient number of non-speech frames, it reads the in-vehicle ambient audio collected within a preset duration before the speech start position and uses this ambient audio as noise estimation data.

[0118] In one implementation, the quality assessment module employs a quality assessment method based on signal statistics. The module determines speech intelligibility based on SNR, the proportion of effective speech frames, the proportion of clipped frames, and the confidence level of each text unit output by the speech recognition model. The module averages the confidence levels of each text unit and, combined with the proportion of effective speech frames and the proportion of clipped frames, queries a preset intelligibility scoring table to obtain the speech intelligibility. The module also queries a preset noise interference scoring table based on the relative magnitude of the average energy of non-speech frames and the average energy of effective speech frames, along with the SNR, to determine the level of noise interference. Higher SNR and higher speech recognition confidence result in higher speech intelligibility; conversely, higher average energy of non-speech frames and lower SNR indicate a higher level of noise interference.

[0119] Speech quality assessment can be conducted using signal statistics-based methods or pre-trained speech quality assessment models. When training the speech quality assessment model, in-vehicle noise samples are collected under different vehicle speeds, window conditions, and passenger conversation states. These in-vehicle noise samples are then mixed with clear speech samples according to multiple preset mixing intensities to obtain training speech samples. Speech clarity and noise interference levels are labeled according to speech intelligibility and known noise mixing intensities. The labeled training speech samples are used to train the speech quality assessment model until the accuracy of recognizing the corresponding quality level in the validation data reaches the preset accuracy. When applying the speech quality assessment model, the Mel-spectral features of the speech command are input into the model, and the model outputs the speech clarity and noise interference levels within the [0, 1] interval.

[0120] Step S302: Match the speech clarity and noise interference level with the corresponding preset parameter ranges to obtain the speech clarity level and noise interference level.

[0121] Here, the preset parameter ranges include multiple non-overlapping and continuously covering speech intelligibility ranges within the [0, 1] interval, and multiple non-overlapping and continuously covering noise interference ranges within the [0, 1] interval. Each speech intelligibility range corresponds to a speech intelligibility level, and each noise interference range corresponds to a noise interference level. The endpoints of the parameter ranges adopt pre-defined opening and closing interval rules to prevent the same parameter value from falling into two adjacent ranges simultaneously.

[0122] In one embodiment, the speech intelligibility range includes [0, 0.5), [0.5, 0.8), and [0.8, 1], which correspond to a first speech intelligibility level, a second speech intelligibility level, and a third speech intelligibility level, respectively. The third speech intelligibility level is higher than the second speech intelligibility level, and the second speech intelligibility level is higher than the first speech intelligibility level. The noise interference level range includes [0, 0.3), [0.3, 0.6), and [0.6, 1], which correspond to a first noise interference level, a second noise interference level, and a third noise interference level, respectively. The third noise interference level is higher than the second noise interference level, and the second noise interference level is higher than the first noise interference level.

[0123] The speech clarity is compared sequentially with each speech clarity interval, and the level corresponding to the speech clarity interval into which the speech clarity falls is determined as the current speech clarity level. Similarly, the noise interference level is compared sequentially with each noise interference level interval, and the level corresponding to the noise interference level interval into which the noise interference level falls is determined as the current noise interference level. Preset parameter intervals are determined during the vehicle development phase using calibration samples from different in-vehicle acoustic environments and are stored as threshold configuration data in the in-vehicle terminal.

[0124] Step S303: Combine the speech clarity level and noise interference level into a target level combination.

[0125] Here, the target level combinations are ordered pairs of levels, with speech clarity level first and noise interference level second. The level combination process does not add the values ​​of the two levels together; instead, it preserves both the speech clarity level and the noise interference level in their entirety. Using three speech clarity levels and three noise interference levels results in nine possible level combinations. For example, the third speech clarity level and the first noise interference level form a target level combination corresponding to high speech clarity and low noise interference; the first speech clarity level and the third noise interference level form a target level combination corresponding to low speech clarity and high noise interference. The target level combinations are associated with the current interactive session identifier and are used to query the dynamic similarity threshold used in this voice interaction.

[0126] Step S304: Determine the dynamic similarity threshold corresponding to the target level combination based on the preset correspondence between level combinations and similarity thresholds; wherein, the higher the speech clarity level, the higher the speech clarity; the higher the noise interference level, the greater the noise interference; when the noise interference level is the same, the dynamic similarity threshold increases with the increase of speech clarity level; when the speech clarity level is the same, the dynamic similarity threshold decreases with the increase of noise interference level.

[0127] Here, the correspondence between the level combinations and the similarity threshold is obtained through calibration during the vehicle development phase. During calibration, the voice samples collected in the in-vehicle environment are grouped according to each level combination, and a voice query vector is generated for each voice sample. For each group of voice samples, the first similarity between the voice query vector and the correct reference voice vector and the non-target reference voice vector is calculated, and an initial similarity threshold is selected based on the distribution of correct matching results and incorrect matching results. In one implementation, the candidate threshold with the highest correct matching rate is selected as the initial similarity threshold from multiple candidate thresholds whose incorrect matching rate is not higher than a preset incorrect matching rate.

[0128] After selecting the initial similarity thresholds for each level combination, a monotonicity check is performed on each initial similarity threshold. When noise interference levels are the same, the similarity thresholds are increased in order of increasing speech clarity; when speech clarity levels are the same, the similarity thresholds are decreased in order of increasing noise interference. The similarity thresholds after monotonicity check are then written into the threshold configuration data along with their corresponding level combinations.

[0129] In one implementation, when the noise interference level is the first noise interference level, the first, second, and third speech intelligibility levels correspond to 0.80, 0.82, and 0.85, respectively; when the noise interference level is the second noise interference level, the first, second, and third speech intelligibility levels correspond to 0.77, 0.80, and 0.82, respectively; and when the noise interference level is the third noise interference level, the first, second, and third speech intelligibility levels correspond to 0.75, 0.77, and 0.80, respectively. Therefore, when speech intelligibility is high and there is no significant noise interference, the dynamic similarity threshold is 0.85; when speech intelligibility is low and there is significant noise interference, the dynamic similarity threshold decreases to 0.75.

[0130] After determining the target level combination, the similarity threshold associated with the target level combination is read from the threshold configuration data, and the read similarity threshold is determined as the dynamic similarity threshold corresponding to this voice interaction. The dynamic similarity threshold is associated with the current interaction session identifier and is used to filter reference speech vectors whose first similarity reaches the dynamic similarity threshold. Increasing the dynamic similarity threshold corresponding to speech intelligibility at the same noise interference level allows for more stringent matching using the stable acoustic features provided by clear speech; conversely, decreasing the dynamic similarity threshold corresponding to a higher level of noise interference at the same speech intelligibility level expands the tolerance range for acoustic feature shifts caused by noise.

[0131] The in-vehicle voice interaction data processing method provided in this application allows the in-vehicle terminal to set a similarity threshold for the actual voice quality of each voice interaction, so that the voice vector matching conditions change with the voice clarity and noise interference level, thereby improving the adaptability and stability of voice vector matching under different in-vehicle acoustic environments.

[0132] In an optional implementation, the vehicle-mounted multimodal vector database is constructed through the following steps S401-S406.

[0133] Step S401: Create a vector storage structure for storing multiple sets of reference data.

[0134] Here, the vector storage structure is the logical data structure of the in-vehicle multimodal vector database. The vector storage structure includes a reference speech vector storage area, a reference text vector storage area, an in-vehicle function operation information storage area, a semantic relationship storage area, a reference data group storage area, and a database metadata storage area. Each storage area is a logical collection of data, and there is no requirement that each storage area must reside on a different physical storage device.

[0135] The reference speech vector storage area should at least include the following fields: reference speech vector identifier, reference speech vector value, vector dimension, encoding model version, reference speech sample identifier, accent type, and data status. The reference text vector storage area should at least include the following fields: reference text vector identifier, reference text vector value, vector dimension, encoding model version, reference instruction text identifier, instruction expression type, and data status. The vehicle function operation information storage area should at least include the following fields: vehicle function operation information identifier, vehicle function identifier, target control module identifier, operation type, control parameters, parameter units, and parameter allowable range. The semantic association storage area records the reference speech vector identifier, reference text vector identifier, semantic similarity, and association status. The reference data group storage area records the reference data identifier and the association between a set of reference speech vectors, reference text vectors, and vehicle function operation information. The database metadata storage area records the database version, vector dimension, encoding model version, creation time, and data volume.

[0136] When creating the vector storage structure, first determine the field names, field types, and field lengths for each storage area. Then, set unique identifier generation rules for reference speech vectors, reference text vectors, vehicle function operation information, semantic relationships, and reference data. Next, establish the reference relationships between each identifier field. Vector fields are stored according to a 128-dimensional vector, and vector values ​​are set to data types according to the output precision of the encoding model. After completing the field configuration, perform null value constraints, identifier uniqueness checks, and related field validity checks on the vector storage structure. Once the checks pass, open the batch write and query interfaces.

[0137] Step S402: Obtain multiple reference voice samples, reference instruction texts corresponding to each reference voice sample, and vehicle function operation information; wherein, the multiple reference voice samples include different accent voice samples corresponding to the same vehicle function operation information, and the reference instruction texts include complete instruction texts and simplified instruction texts corresponding to the same vehicle function operation information; the simplified instruction texts are instruction texts with wake-up words omitted and / or with some modifier words omitted.

[0138] Here, reference speech samples can be obtained through in-vehicle recordings during the vehicle development phase, or selected from historical voice interaction data for which usage licenses have been obtained. When collecting reference speech samples, the speech acquisition module collects speech data at a sampling rate of 16kHz and a quantization bit depth of 16bit, and records the vehicle being acquired, the vehicle speed range, window status, in-vehicle noise type, speaker's accent, and the acquisition time. The reference speech samples undergo Wiener filtering, NLMS echo cancellation, endpoint detection, normalization, and sampling rate unification processing to obtain effective speech segments for feature encoding.

[0139] For each in-vehicle function operation information, multiple reference speech samples are collected from different speakers with different accents. These accents include Mandarin, Cantonese, Sichuanese, and regional accents determined by the vehicle's sales region. The reference speech samples also cover different speech rates, volumes, and in-vehicle noise environments to reduce the impact of pronunciation habits and acoustic differences on vector matching. Independent reference speech sample labels are assigned to different accent speech samples corresponding to the same in-vehicle function operation information.

[0140] The reference instruction text is a standard text record corresponding to the content expressed in the reference speech sample. The reference instruction text can be transcribed by annotators based on the reference speech sample, or it can be generated based on the vehicle function operation template and then verified by annotators. The reference instruction text includes complete instruction text and simplified instruction text. The complete instruction text retains the wake-up word, operation word, vehicle function name, and necessary modifiers. The simplified instruction text omits at least one of the wake-up word and some modifiers, but retains the core expression that establishes a correspondence with the vehicle function operation information. For example, if the vehicle function operation information indicates that the air conditioning is to be turned on, the complete instruction text is recorded as "Xiao Ben, please turn on the air conditioning," the simplified instruction text after omitting the wake-up word is recorded as "Please turn on the air conditioning," and the simplified instruction text after further omitting some modifiers is recorded as "Air conditioning."

[0141] When constructing samples for the same in-vehicle function operation information, different accented voice recordings corresponding to the complete and simplified command texts are recorded separately. Each reference voice sample is then bound to a reference command text identifier and an in-vehicle function operation information identifier. The in-vehicle function operation information uses structured data to record vehicle functions, target control modules, operation types, and control parameters. The sample verification module checks whether the reference voice sample is complete, whether the reference command text matches the voice content, and whether the in-vehicle function operation information can be uniquely pointed to by the reference command text. Samples that fail verification are recorded with the reason for the failure.

[0142] Step S403: Perform feature encoding on each reference speech sample and its corresponding reference instruction text to obtain the reference speech vector corresponding to each reference speech sample and the reference text vector corresponding to each reference instruction text.

[0143] Here, a feature encoding model and semantic alignment algorithm are employed. Reference speech samples are input into a Mel-CNN or MFCC feature extractor to extract acoustic and semantic features, yielding an initial reference speech vector. Reference instruction text is input into a lightweight Transformer encoder to extract textual semantic features, resulting in an initial reference text vector. Then, the initial reference speech vector and initial reference text vector are mapped to a 128-dimensional common semantic space through speech mapping and text mapping branches, respectively, and normalized to obtain the reference speech vector and reference text vector.

[0144] When generating reference speech vectors and reference text vectors, the encoding model structure, pre-trained weights, model parameter version, vector dimension, and normalization rules are consistent with the configuration used when generating speech query vectors and text query vectors. After encoding, the reference speech sample identifier, reference instruction text identifier, encoding model version, and corresponding vector are bound and stored. If the dimension of the encoding result is not equal to 128 or invalid values ​​exist in the encoding result, the corresponding version of the encoding model is reloaded and feature encoding is performed again; if the re-encoding still fails the check, the corresponding sample is marked as encoding failure and processing of the corresponding sample is stopped. After the encoding model is updated, the affected reference speech vectors and reference text vectors are regenerated according to the updated encoding model to avoid the reference vectors in the database using different feature representations than the query vectors.

[0145] Step S404: Based on the semantic similarity between the reference speech vector and the reference text vector, establish the semantic association between the corresponding reference speech vector and the reference text vector.

[0146] Here, semantic similarity is represented using cosine similarity. For reference speech samples and reference command texts with binding relationships, the corresponding reference speech vectors and reference text vectors are read, and the cosine similarity between the reference speech vectors and reference text vectors is calculated. The larger the cosine similarity, the closer the vehicle operation semantics expressed by the reference speech vectors and reference text vectors are. The semantic association establishment process also checks whether the reference speech samples and reference command texts are associated with the same in-vehicle function operation information, avoiding the association of different vehicle operations solely due to vector proximity.

[0147] In one implementation, a preset semantic association threshold is set to 0.80. When the cosine similarity is greater than or equal to the preset semantic association threshold, and the reference speech sample and the reference instruction text are associated with the same in-vehicle function operation information, a semantic association identifier is generated, and the reference speech vector identifier, reference text vector identifier, cosine similarity, in-vehicle function operation information identifier, and association status are recorded in the semantic association storage area. When the cosine similarity is less than the preset semantic association threshold, the sample annotation, encoding model version, and vector normalization results are re-examined. If the cosine similarity after checking and re-encoding is still less than the preset semantic association threshold, no semantic association is established, and the corresponding sample is transferred to manual review. Semantic similarity calculations are performed on different accent speech samples, complete instruction text, and simplified instruction text to ensure that each set of semantic associations is verified individually.

[0148] Step S405: Associate each set of semantically related reference speech vectors and reference text vectors with corresponding vehicle function operation information to form a set of reference data.

[0149] Here, before generating reference data, the semantic association record is read, and the corresponding vehicle function operation information is queried based on the vehicle function operation information identifier. The data assembly module then checks again whether the identifiers corresponding to the reference speech vector, reference text vector, and vehicle function operation information are consistent. After passing the check, a unique reference data identifier is generated, and the reference data identifier is bound to the reference speech vector identifier, reference text vector identifier, semantic association identifier, and vehicle function operation information identifier. Each set of reference data thus formed logically includes a reference speech vector, a reference text vector, semantic similarity, and commonly associated vehicle function operation information.

[0150] The same in-vehicle function operation information corresponds to multiple sets of reference data. Each set of reference data retains reference vectors corresponding to different accents, speech rates, noise environments, or text expressions. For example, the complete Mandarin command, simplified Mandarin command, complete Cantonese command, and simplified Cantonese command for turning on the air conditioner each form reference data, and all sets of reference data point to the in-vehicle function operation information for turning on the air conditioner. By retaining multiple sets of reference data, the in-vehicle multimodal vector database can provide a matching basis for different pronunciations and different command expressions.

[0151] In one implementation, before execution, the reference speech vector and reference text vector are optimized for vector distribution and quantized, and the processed reference speech vector and reference text vector are used to form reference data.

[0152] Step S406: Write the formed sets of reference data into the vector storage structure to obtain the vehicle-mounted multimodal vector database.

[0153] Here, when writing reference data to each group, the reference data is divided into multiple write batches according to a preset batch number, and a database transaction is created for each write batch. First, it checks whether the corresponding vehicle function operation information already exists in the vehicle function operation information storage area; if the corresponding vehicle function operation information does not exist, it is written to the vehicle function operation information storage area. Then, the reference speech vector and reference text vector are written to their respective vector storage areas, the semantic association is written to the semantic association storage area, and the reference data identifier and each association identifier are written to the reference data group storage area. When the same vehicle function operation information already exists, the corresponding vehicle function operation information identifier is directly reused to avoid repeatedly storing the same vehicle function operation information.

[0154] After each write batch is completed, the actual number of data written, vector dimensions, encoding model version, identifier uniqueness, and integrity of associated fields are checked. If all checks pass, the database transaction is committed, and the corresponding reference data's status is updated to searchable. If any check fails, the current database transaction is rolled back, the abnormal data is corrected, and the current write batch is re-executed to prevent incomplete reference data from entering the vehicle-mounted multimodal vector database. After all reference data has been written, the total amount of reference data, database version, and last update time in the database metadata are updated.

[0155] The sets of reference data that have been logically written and are in a searchable state constitute the vehicle-mounted multimodal vector database. The vehicle terminal can retrieve the corresponding reference data based on the voice query vector and the text query vector, and generate vehicle control commands based on the vehicle function operation information.

[0156] The in-vehicle voice interaction data processing method provided in this application establishes a data closed loop between reference speech vectors, reference text vectors, and in-vehicle function operation information. It also expands the coverage of reference data by using speech samples with different accents, complete command texts, and simplified command texts, thereby improving the adaptability of the in-vehicle multimodal vector database to accent differences and simplified expressions.

[0157] In an optional implementation, before step S405, the method further includes the following steps S501-S504.

[0158] Step S501: Construct an objective function for optimizing the vector distribution based on the vector distribution of the reference speech vector and the reference text vector.

[0159] Here, reference speech vectors and reference text vectors with normal encoding status, consistent model version, and 128 dimensions are read from the reference speech vector storage area and the reference text vector storage area. The vector optimization unit retains the correspondence between each vector according to the vehicle function operation information identifier and semantic association relationship identifier, so as to avoid destroying the association between the reference speech vector, reference text vector and vehicle function operation information during the optimization process.

[0160] Vector distribution is used to represent the positional relationship and clustering degree of reference speech vectors and reference text vectors in the common semantic space. The vector optimization unit forms a set of vectors to be optimized from all reference speech vectors and reference text vectors to be optimized, and assigns a sequence number to each vector in the set according to its vector identifier. For any two vectors in the set of vectors to be optimized, the cosine similarity between the two vectors is calculated, and the cosine similarity is used as the association weight between the two vectors. Vectors expressing the same or similar vehicle operation semantics have a larger association weight, while vectors expressing different vehicle operation semantics have a smaller association weight.

[0161] The objective function is expressed as follows: .

[0162] Among them, A ij Let v be the cosine similarity between the i-th vector and the j-th vector. i and v j The optimized high-dimensional vector is λ, where λ is the regularization coefficient. Indicates v i With v j The square of the L2 norm of the difference Indicates v i The squared norm of the vector is obtained by iterating through the vectors in the set of vectors to be optimized using i and j. This is used to constrain vectors with larger association weights to maintain a smaller distance in the common semantic space. This is used to limit the excessive increase in vector values ​​and improve vector distribution. By minimizing the objective function, reference speech vectors and reference text vectors expressing the same vehicle operation semantics can maintain a high degree of aggregation, and the range of vector values ​​is constrained, thereby reducing the complexity of subsequent similarity calculations.

[0163] For reference speech vectors and reference text vectors with semantic relationships, the vector optimization unit also employs an attention mechanism for assisted fusion. The attention layer assigns weights based on the degree of correlation between speech acoustic features, speech semantic features, and text semantic features and in-vehicle function operation information, fusing the weighted features to obtain a 128-dimensional fused semantic vector. This fused semantic vector is used to verify whether the update directions of the reference speech vector and reference text vector maintain semantic consistency, and does not replace the reference speech vectors and reference text vectors used for speech retrieval and text retrieval, respectively.

[0164] Step S502: Set the regularization coefficient, iteration step size and convergence threshold, iterate and update the reference speech vector and reference text vector according to the objective function, and calculate the change between the objective function values ​​corresponding to two adjacent iterations.

[0165] Here, the initial value of the regularization coefficient is... Set to 0.1. Before the formal iteration, the regularization coefficient to be used in the formal iteration is determined using the parameter pre-calibration process of the preset rounds. The regularization adjustment threshold is set during the parameter pre-calibration process. 1×10 -5 And after each round of pre-calibration iteration, the difference between the current objective function value and the previous objective function value is calculated. The difference is less than... At that time, according to Increase the regularization coefficient, Set to 1.2; difference greater than At that time, according to Reduce the regularization coefficient, Set to 0.8; the difference equals During this process, the current regularization coefficient is maintained. After the parameter precalibration process is completed, the final regularization coefficient is used as the initial value for the regularization coefficient in the formal iteration.

[0166] Iteration step size The initial value is set to 0.01. The convergence threshold can be set to 1×10. -4 It can also be set to 1×10 when higher vector distribution accuracy is required. -5 After setting the parameters, the reference speech vector and reference text vector are used as the vectors corresponding to the zeroth iteration, and the objective function value is calculated for the vectors corresponding to the zeroth iteration. Then, the gradient descent algorithm is used to update each reference speech vector and reference text vector once according to the objective function, obtaining the vectors corresponding to the first iteration, and the vectors corresponding to the first iteration are normalized.

[0167] After each iteration, the cosine similarity and objective function value between each vector are recalculated. The absolute value of the difference between the objective function value of the current iteration and the objective function value of the previous iteration is also calculated, and this absolute value is used as the change in the objective function value between two adjacent iterations. The vector optimization unit simultaneously saves the vector, objective function value, regularization coefficient, and iteration step size for each iteration, facilitating recovery to a historical iteration state with a smaller objective function value in case of iteration anomalies.

[0168] Step S503: When the change is greater than the convergence threshold, adjust the regularization coefficient and iteration step size and continue iterating; when the change is less than or equal to the convergence threshold, stop iterating.

[0169] Here, a change greater than the convergence threshold indicates that the distributions of the reference speech vectors and reference text vectors are still changing. When the current objective function value is lower than the objective function value corresponding to the previous iteration, but there is still significant uneven clustering of semantic association vectors, then according to... Increase the regularization coefficient to strengthen the constraints on vector values ​​and vector distribution. When the current objective function value is higher than the objective function value of the previous iteration, or when the objective function value fluctuates across multiple iterations, follow the... Reduce the regularization coefficient and decrease the current iteration step size to half of the previous iteration step size to reduce the magnitude of a single update. When the current objective function value is steadily decreasing, maintain the current iteration step size; after the objective function value recovers to a stable decrease, gradually increase the iteration step size according to the preset step size recovery rule, but the recovered iteration step size shall not exceed 0.01.

[0170] After adjusting the regularization coefficient and iteration step size, the reference speech vector and reference text vector are updated according to the objective function, and the change is recalculated. When the change is less than or equal to the convergence threshold, it indicates that the vector distribution corresponding to two adjacent iterations has stabilized, the iteration is stopped, and the reference speech vector and reference text vector corresponding to the current iteration are used as the updated reference speech vector and reference text vector.

[0171] To prevent anomalous data from causing the iteration process to fail to converge, the vector optimization unit also sets a preset maximum number of iterations. When the preset maximum number of iterations is reached and the change is still greater than the convergence threshold, the update stops, and the historical iteration state with the smallest objective function value is selected from the saved historical iteration states. The reference speech vector and reference text vector corresponding to the selected historical iteration state are used as the updated reference speech vector and reference text vector. After the iteration ends, the cosine similarity between semantically related reference speech vectors and reference text vectors is checked. If the cosine similarity is lower than the preset semantic association threshold, the corresponding vector is restored to the historical iteration state with a smaller objective function value that meets the preset semantic association threshold, avoiding the vector distribution optimization from destroying the established semantic association relationship.

[0172] Step S504: Perform vector quantization on the iteratively updated reference speech vector and reference text vector, and use the quantized reference speech vector and reference text vector to form the corresponding reference data.

[0173] Here, the K-means vector quantization algorithm is used. Before quantization, the iteratively updated reference speech vectors and reference text vectors are combined into a quantization training set, and initial cluster centers are selected from the quantization training set according to a preset number of clusters. Subsequently, the distance between each vector and each cluster center is calculated, and each vector is assigned to the cluster corresponding to the cluster center with the smallest distance. The cluster centers are recalculated based on the vectors contained in each cluster. Vector assignment and cluster center calculation are performed again until the change in cluster centers obtained from two consecutive calculations is less than or equal to a preset quantization convergence threshold, or the number of quantization iterations reaches a preset number of quantization iterations.

[0174] After quantization iteration, the cluster centers are organized into a quantization codebook, and a corresponding quantization code is assigned to each cluster center. For each reference speech vector and reference text vector, the cluster center with the smallest distance is determined, and the quantization code corresponding to the cluster center is used as a compact storage representation of the vector. The reference speech vector and reference text vector still logically correspond to a 128-dimensional common semantic space. The vector storage structure stores the quantization code, quantization codebook identifier, and quantization model version, thereby reducing the storage footprint caused by directly storing all floating-point vectors.

[0175] When performing similarity retrieval, the corresponding cluster centers can be read from the quantization codebook and the quantization vector can be recovered based on the quantization encoding. Alternatively, the similarity between the query vector and the corresponding cluster centers in the quantization codebook can be directly calculated. The speech query vector is matched with the quantized reference speech vector, and the text query vector is matched with the quantized reference text vector, avoiding confusion between the speech retrieval channel and the text retrieval channel during the quantization process.

[0176] After quantization, the same similarity retrieval is performed on the reference speech vectors and reference text vectors before and after quantization using preset verification data, and the correct matching results and storage usage are compared before and after quantization. If the retrieval accuracy loss after quantization does not exceed 3% and the storage usage is reduced by 75% compared to before quantization, the current quantization codebook is determined to meet the preset quantization requirements. If the retrieval accuracy loss exceeds 3%, the number of clusters is increased and the quantization codebook is retrained; if the storage usage does not meet the requirement of a 75% reduction, the number of quantization bits is reduced or the number of clusters is reselected, and retrieval accuracy verification and storage usage verification are performed again.

[0177] After quantization verification is completed, the corresponding reference vectors output in step S403 are replaced with the quantized reference speech vectors and reference text vectors, while retaining the reference speech vector identifier, reference text vector identifier, semantic association identifier, and vehicle function operation information identifier. The quantized reference speech vectors and reference text vectors are used to form corresponding reference data.

[0178] In an optional implementation, after step S406, the method further includes the following steps S601-S604.

[0179] Step S601: Construct a tree index for the reference speech vector and the reference text vector.

[0180] Here, the index building module reads the reference speech vectors, reference text vectors, and corresponding reference data identifiers that are in a searchable state from the vector storage structure. If the reference vector has been quantized, the corresponding cluster centers are read from the quantization codebook according to the quantization encoding, and the 128-dimensional coordinates of the reference vector are represented by these cluster centers. If the reference vector has not been quantized, its 128-dimensional coordinates are used directly. The index building module maintains separate speech retrieval and text retrieval channels according to vector type, preventing speech query vectors from entering the text index or vice versa.

[0181] The tree-structured index uses a k-dimensional tree (kd tree). The index building module constructs a kd tree for all reference speech vectors and a kd tree for all reference text vectors. Both kd trees use the same construction rules and store the vector identifier, vector coordinates, and reference data identifier for the corresponding modality, respectively.

[0182] When constructing the kd-tree, the 128 vector dimensions are arranged in a fixed order, and a cyclic splitting strategy is used to select the splitting dimension for each level of nodes. The root node uses the first vector dimension, and the next level node uses the second vector dimension; after traversing the 128th vector dimension, it restarts from the first vector dimension. For the current set of vectors to be split, they are sorted according to the coordinate values ​​corresponding to the current splitting dimension. The vector in the middle position is selected as the current node. Vectors with smaller coordinate values ​​are assigned to the left subset, and vectors with larger coordinate values ​​are assigned to the right subset. The same splitting process is then performed on the left and right subsets respectively until the number of vectors to be split reaches the preset leaf node capacity.

[0183] Each kd-tree node records at least the vector identifier, vector type, split dimension, split coordinate value, reference data identifier, left child node pointer, right child node pointer, and node status. When multiple reference vectors have the same split coordinate value, all reference vectors are recorded in the data list of the same leaf node. After the tree index is constructed, an integrity check is performed starting from the root node to confirm that the split coordinate value corresponding to each left child node is not greater than the split coordinate value of the parent node, the split coordinate value corresponding to each right child node is not less than the split coordinate value of the parent node, and to confirm that the number of vectors in the tree index is consistent with the number of searchable vectors in the vector storage structure.

[0184] When performing vector retrieval, the left or right child node is selected based on the coordinates of the query vector in the current split dimension, and candidate reference vectors are searched layer by layer along the selected branch. After obtaining the candidate reference vectors, the cosine similarity between the query vector and the candidate reference vectors is calculated, and the candidate results are determined according to the cosine similarity. The kd-tree is used to handle general retrieval of all reference vectors and is suitable for vector sets with large amounts of data, such as historical voice interaction vectors.

[0185] Step S602: Obtain the retrieval frequency of each reference speech vector and each reference text vector, and construct a hash index for the reference speech vectors and reference text vectors whose retrieval frequency reaches the preset frequency condition.

[0186] Here, the vector storage structure sets a retrieval count and the most recent retrieval time for each reference speech vector and reference text vector. Each time a retrieval request accesses a reference vector or adds a reference vector to the candidate results, the retrieval interaction module increments the corresponding retrieval count and records the access time. The index building module reads the retrieval count of each vector within a one-minute statistical window, converts the retrieval count into retrievals per minute, and obtains the retrieval frequency of each reference speech vector and reference text vector.

[0187] In one implementation, the preset frequency condition is a retrieval frequency greater than or equal to 10 times per minute. The index building module identifies reference speech vectors and reference text vectors that meet the preset frequency condition as high-frequency reference vectors. High-frequency reference vectors include reference vectors corresponding to commonly used voice commands such as turning on the air conditioner, navigating to the company, and playing music. Building a hash index for high-frequency reference vectors avoids having to search layer by layer from the root node of the kd-tree for each retrieval.

[0188] The size of the hash table is set to 2. 16 The hash key can be obtained from the quantization code generated in step S504, or from the unquantized reference vector through a preset symbol mapping. When using the preset symbol mapping, the coordinate values ​​of each dimension of the reference vector are compared with the average coordinate value of the corresponding dimension. Dimensions not less than the average coordinate value are recorded as the first binary value, and dimensions less than the average coordinate value are recorded as the second binary value. The binary values ​​are then combined in dimensional order to obtain the vector feature code of the reference vector. The vector type is combined with the quantization code or vector feature code as the hash key, and then the hash key is mapped to a storage location in the hash table using a preset hash function.

[0189] Each index entry in the hash table records the hash key, vector identifier, vector type, reference data identifier, retrieval frequency, and data status. When multiple high-frequency reference vectors are mapped to the same storage location, an index entry list is created in the corresponding storage location, and a cosine similarity check is performed on each reference vector in the index entry list during retrieval. After the query vector generates a hash key according to the same rules, the storage location corresponding to the hash key is directly accessed, enabling the candidate positioning of high-frequency reference vectors to achieve a constant-time complexity of O(1).

[0190] The index building module recalculates the retrieval frequency according to a preset period. If a reference vector fails to meet the preset frequency condition for several consecutive statistical windows, the corresponding index entry is removed from the hash index, but the corresponding reference vector is retained in the kd-tree. Therefore, the hash index only occupies storage resources corresponding to high-frequency reference vectors, while low-frequency reference vectors can still be retrieved using the tree index.

[0191] Step S603: Based on the semantic relationship between the reference speech vector and the reference text vector, and the relationship between the two and the vehicle function operation information, construct a semantic relationship index between the reference speech vector, the reference text vector, and the vehicle function operation information.

[0192] Here, the index building module sequentially reads each group of reference data and checks whether the reference speech vector identifier, reference text vector identifier, semantic association identifier, and vehicle function operation information identifier are complete. After passing the check, the reference data identifier is used as the group identifier for the semantic association index. A first mapping is established from the reference speech vector identifier to the reference text vector identifier and the vehicle function operation information identifier; a second mapping is established from the reference text vector identifier to the reference speech vector identifier and the vehicle function operation information identifier; and a third mapping is established from the vehicle function operation information identifier to each reference speech vector identifier and each reference text vector identifier.

[0193] Each semantic association index entry records at least the reference data identifier, reference speech vector identifier, reference text vector identifier, vehicle function operation information identifier, semantic similarity level, encoding model version, and data status. Within the same set of reference data, reference speech vectors and reference text vectors form a corresponding mapping according to the semantic association established in step S404. When the same vehicle function operation information corresponds to multiple sets of reference data, the third mapping records all reference speech vector identifiers and reference text vector identifiers associated with the vehicle function operation information.

[0194] After a voice query vector matches a target reference voice vector, the corresponding reference text vector and vehicle function operation information are directly retrieved based on the first mapping. Similarly, after a text query vector matches a target reference text vector, the corresponding reference voice vector and vehicle function operation information are directly retrieved based on the second mapping. This semantic association index transforms the vector matching results into explicit vehicle operations, avoiding the need to traverse all reference data again to search for vehicle function operation information.

[0195] Step S604: When the reference speech vector, reference text vector, or vehicle function operation information is inserted, deleted, or updated, the affected indexes in the tree index, hash index, and semantic association index are incrementally updated.

[0196] Here, when data in the vector storage structure changes, the data maintenance module generates a data change record. The data change record includes the change type, the identifier of the changed object, the version before the change, the version after the change, and the change time. The index building module determines the affected tree index nodes, hash index entries, and semantically related index entries based on the data change record, processing only the index content affected by the current data change.

[0197] When inserting a new reference speech vector or reference text vector, starting from the root node of the corresponding kd-tree, the coordinate values ​​of the new vector are compared according to the split dimensions of each level until the leaf node corresponding to the new vector is determined. The new vector identifier and the reference data identifier are then written to the corresponding leaf node. When the retrieval frequency of the new vector reaches a preset frequency condition, the index entry corresponding to the new vector is written to the hash table according to the hash key. When the new reference data has established a semantic association and is associated with vehicle function operation information, the newly added first mapping, second mapping, and third mapping are written to the semantic association index.

[0198] When deleting a reference speech vector or reference text vector, first mark the node status in the corresponding kd-tree node as invalid, then delete the corresponding index entry from the hash index, and delete the mapping involving the corresponding vector identifier from the semantic association index. When the number of invalid nodes reaches the preset node cleanup condition, only the subtree containing invalid nodes is rebuilt, and the complete kd-tree is not rebuilt. When deleting vehicle function operation information, first check whether the vehicle function operation information is still associated with valid reference data; if a valid association still exists, do not perform deletion and output the association check result; if no valid association exists, delete the third mapping corresponding to the vehicle function operation information from the semantic association index.

[0199] When updating the vector values, quantization codes, or model versions of reference speech or text vectors, first delete the kd-tree nodes and hash index entries corresponding to the old vectors, then re-insert the corresponding indexes according to the updated vector coordinates and hash keys. When updating the operation type or control parameters of in-vehicle function operation information, keep the tree index and hash index unchanged, and update the index entries in the semantic association index that involve the corresponding in-vehicle function operation information identifiers. When updating the semantic association relationship between reference speech vectors and reference text vectors, only update the corresponding first and second mappings in the semantic association index. When the retrieval frequency of a reference vector crosses a preset frequency condition, only add or remove the corresponding index entries in the hash index.

[0200] When performing an incremental update, an updated index entry is first generated outside the current index version, and a consistency check is performed on the vector identifier, reference data identifier, and vehicle function operation information identifier. If the check passes, the updated index entry is switched to the current valid version; if the check fails, the index version before the update is retained and the reason for the update failure is recorded. The processing latency for a single incremental update does not exceed 100ms, ensuring that new voice commands, changes in user voice habits, and adjustments to vehicle function operation information are promptly reflected in the search results.

[0201] In optional implementations, the vector storage architecture includes local caching, on-board solid-state storage, and cloud storage.

[0202] Here, local caching is used to store frequently retrieved data that requires fast access; in-vehicle solid-state storage is used to persistently store reference data, historical data, and security control information; and cloud storage is used to store historical data, redundant backup data, and data that needs to be synchronized between different in-vehicle terminals. All three storage resources use unified storage data identifiers, data versions, and association identifiers to ensure that the same set of storage data can still be associated with the corresponding reference data and in-vehicle function operation information after being transferred between different storage levels.

[0203] Step S406 includes the following steps S701-S704.

[0204] Step S701: Each set of reference data is determined as a set of stored data.

[0205] Here, the reference data identifier, reference speech vector, reference text vector, semantic relationships, and vehicle function operation information corresponding to a set of reference data are read, and a unique storage data identifier is generated for the set of reference data. When the reference speech vector and reference text vector have been quantized, the storage data records the quantization encoding, quantization codebook identifier, and quantization model version; when the reference speech vector and reference text vector have not been quantized, the storage data records the corresponding vector value, vector dimension, and encoding model version.

[0206] Each set of stored data includes at least a stored data identifier, a reference data identifier, a reference speech vector identifier, a reference text vector identifier, an in-vehicle function operation information identifier, a semantic relationship identifier, a data source, a creation time, an update time, a data version, a data size, a number of retrievals, and a data status. The fields in each set of reference data are arranged according to a preset serialization rule, and an integrity check value is calculated. The stored data identifier, data version, and integrity check value are written together into the stored data header, while the reference vectors and in-vehicle function operation information are written into the stored data body. Thus, a set of stored data forms a data unit that can be independently written, read, migrated, and verified, avoiding field mismatches between different reference data sets.

[0207] Step S702: Determine the data sensitivity level of each group of stored data, and write each group of stored data into the corresponding storage partition in the vehicle solid-state storage according to the data sensitivity level. Among them, the stored data with a data sensitivity level that reaches the preset sensitivity level is encrypted and then written into the corresponding encrypted storage partition.

[0208] Here, the data sensitivity level is determined based on the data source, vector type, vehicle function operation type, whether it contains user identifiers, and whether it involves vehicle control parameters. In one implementation, the data sensitivity level includes a first data sensitivity level, a second data sensitivity level, and a third data sensitivity level. The first data sensitivity level corresponds to non-sensitive data, the second data sensitivity level corresponds to sensitive data, and the third data sensitivity level corresponds to voice privacy data. The third data sensitivity level is higher than the second data sensitivity level, and the second data sensitivity level is higher than the first data sensitivity level. The preset sensitivity level is set to the second data sensitivity level.

[0209] General voice command vectors, entertainment-related voice vectors, and in-vehicle interaction data without user identification are classified as the first data sensitivity level. Autonomous driving perception data vectors, vehicle control parameter vectors, and data related to vehicle location or operating status are classified as the second data sensitivity level. User voice habit vectors, privacy-related voice segment vectors, and voice data capable of identifying specific users are classified as the third data sensitivity level. User voice habit vectors and privacy-related voice segment vectors are processed according to the same data sensitivity level determination rules when subsequently written into the in-vehicle multimodal vector database.

[0210] The in-vehicle solid-state storage uses a solid-state drive supporting the Non-Volatile Memory Express (NVMe) protocol, with a storage capacity of 128GB. The in-vehicle solid-state storage is divided into three partitions: a non-sensitive data partition, a sensitive data encryption partition, and a voice privacy data encryption partition. Data corresponding to the first data sensitivity level is written to the non-sensitive data partition, data corresponding to the second data sensitivity level is written to the sensitive data encryption partition, and data corresponding to the third data sensitivity level is written to the voice privacy data encryption partition.

[0211] The stored data corresponding to the second and third data sensitivity levels are encrypted using Advanced Encryption Standard 256-bit (AES-256). During encrypted write operations, the data to be encrypted is first generated according to the serialization rules determined in step S701, and then a key access request is sent to the Hardware Security Module (HSM). The HSM generates or reads the currently valid data encryption key and performs the encryption operation internally. The security control unit writes the encrypted data, key identifier, encryption algorithm version, and integrity check value to the corresponding encrypted storage partition, and does not store the plaintext key in the ordinary storage area of ​​the vehicle terminal.

[0212] HSM updates its data encryption key every 7 days. After the key is updated, newly written storage data is encrypted using the new key; existing data is decrypted in batches according to a preset number and then re-encrypted using the new key. The old key is discontinued after all corresponding data has been re-encrypted to avoid data malfunctions during key switching.

[0213] The system monitors the used capacity of the vehicle's solid-state storage in real time. When the used capacity reaches 90% of the total capacity, a storage threshold warning is triggered, and historical storage data retrieved less than once per day is filtered out. Historical storage data that has been backed up to the cloud and has not been marked as core data is cleaned up in order of retrieval frequency from low to high. Frequently used voice command vectors, user voice habit vectors, and data still associated with effective vehicle function operation information are prioritized for retention.

[0214] Step S703: Obtain the retrieval frequency of each group of stored data, and load the stored data whose retrieval frequency reaches the preset frequency condition into the local cache.

[0215] Here, the statistical method for retrieval frequency is consistent with the statistical method for reference vector retrieval frequency in step S602. Each time the retrieval interaction module reads a set of stored data, it increments the corresponding retrieval count and records the retrieval time. The storage management module calculates the retrieval count per minute for each set of stored data according to a one-minute statistical window, and combines this with the retrieval records from the most recent hour to determine whether the stored data continuously meets the preset frequency conditions.

[0216] The local cache uses a Level 1 cache (L1 cache) with a capacity of 16GB. The preset frequency condition is a retrieval frequency of 10 times per minute within the past hour. When the preset frequency condition is met, the corresponding stored data is read from the vehicle's solid-state storage, its integrity check value is verified, and the verified data is loaded into the local cache. When encrypted stored data needs to be loaded into the local cache, access permission verification is performed first, then HSM is called to decrypt it, and the decryption result is written to the protected cache area. The protected cache area clears the corresponding plaintext data when the vehicle is powered off or the cache expires.

[0217] Each cached entry records the stored data identifier, data version, retrieval frequency, last access time, cache load time, and data sensitivity level. When the remaining cache capacity is insufficient, cached entries with lower retrieval frequencies and earlier last access times are removed first. If stored data fails to meet the preset frequency condition for multiple consecutive statistical windows, the corresponding cached entry is removed from the local cache, but the corresponding stored data continues to be retained in the vehicle's solid-state storage. Frequently used voice command vectors, recent user voice interaction feature vectors, and frequently used vehicle function operation information are read through the local cache, with a retrieval response time of no more than 50ms.

[0218] Step S704: The stored data that meets the preset backup conditions is identified as the data to be synchronized, and the data to be synchronized is synchronized to the cloud storage.

[0219] Here, the preset backup conditions include at least one of the following: the stored data has not yet been fully backed up to the cloud; the stored data version is higher than the most recent cloud backup version; the stored data is marked as redundant backup data; the stored data belongs to user voice habit data or historical voice interaction data; and the in-vehicle solid-state storage is about to undergo capacity cleanup. Each group of stored data is checked according to the preset backup conditions. Stored data that meets any of the preset backup conditions is marked as data to be synchronized, and the identifier of the data to be synchronized, the local data version, the data sensitivity level, and the synchronization status are recorded.

[0220] Cloud storage is deployed on the car manufacturer's private cloud server. Non-sensitive voice data and general voice command vectors are synchronized via the Access Point Name (APN) entertainment channel APN3; voice privacy data and sensitive data are synchronized via the control channel APN2. Data to be synchronized corresponding to the second and third data sensitivity levels is kept AES-256 encrypted before transmission and an encrypted transmission connection is established using Transport Layer Security 1.3 (TLS 1.3). Specific channel selection, synchronization frequency adjustment, and network interruption handling procedures are described in subsequent implementation details.

[0221] The storage management module sets the synchronization frequency to between once per minute and once every 10 minutes based on the current network bandwidth, prioritizing the synchronization of user voice habit vectors and core voice data whose data versions have changed. Upon receiving the data to be synchronized, the cloud storage returns the cloud data version and integrity verification result. If the integrity verification passes, the local synchronization status is updated to "synchronization complete," and the most recent synchronization time is recorded; if the integrity verification fails or transmission is interrupted, the pending synchronization status is retained, and synchronization is awaited again.

[0222] The vehicle terminal performs a synchronization check every 7 days between locally stored data and cloud backup data. The synchronization check uses Message-Digest Algorithm 5 (MD5) to generate a data digest and compares the local data digest with the cloud data digest. If the data digests are inconsistent, the data copy that needs to be repaired is determined according to the data version, and data synchronization is re-executed, with a focus on verifying the integrity of user voice habit vectors and historical voice interaction data.

[0223] During the operation of the in-vehicle multimodal vector database, the in-vehicle voice interaction data processing method also includes a storage data access control process. Upon receiving a data access request for stored data, the security control unit parses the data access request to obtain the requesting module identifier, the target stored data identifier, the request operation type, the request time, and the identity verification information. Based on the requesting module identifier, the corresponding access permission level is queried, and based on the target stored data identifier, the corresponding data sensitivity level is queried. Finally, it is determined whether the access permission level allows the requesting module to perform the requested operation on the target stored data.

[0224] The method also includes: In response to a data access request for stored data, the system performs access permission verification on the requesting module that initiated the data access request. If the verification passes, the system provides the requested stored data to the requesting module. If the verification fails, the system rejects the data access request and records the rejected access event.

[0225] Here, access permission levels include Level 1, Level 2, and Level 3 permissions. The domain controller is configured with Level 1 permissions, which allow access to the stored data corresponding to each data sensitivity level. The in-vehicle central control unit is configured with Level 2 permissions, which allow access to non-sensitive data, general voice command vectors, and some authorized sensitive data, but not to voice privacy data. The in-vehicle voice interaction module and third-party applications are configured with Level 3 permissions, which only allow access to general voice command vectors and corresponding voice data collected and authorized for use by the request module, but not to voice privacy data or unauthorized sensitive data.

[0226] If access permission verification passes, the target storage data is searched in the order of local cache, in-vehicle solid-state storage, and cloud storage. If the target storage data is encrypted, HSM is called to decrypt it, and only the data fields required for the request operation are provided to the requesting module. If access permission verification fails, the data access request is rejected, the target storage data content is not returned to the requesting module, and the denied access event is recorded.

[0227] A denied access event must include at least the request module identifier, the target stored data identifier, the requested operation type, the request time, the data sensitivity level, and the reason for the denial. Insertion, deletion, retrieval, and modification operations on stored data are recorded to form a data operation log. The data operation log is retained for at least 30 days and stored in isolation according to log access permissions to facilitate traceability of abnormal access and data changes.

[0228] The in-vehicle voice interaction data processing method provided in this application configures storage data among local cache, in-vehicle solid-state storage and cloud storage according to data sensitivity level, retrieval frequency and backup requirements. While improving the reading speed of high-frequency data and the reliability of data backup, it realizes encrypted storage, permission isolation and access traceability of sensitive voice data.

[0229] In an optional implementation, step S704 synchronizes the data to be synchronized to cloud storage, including the following steps S801-S806.

[0230] Step S801: Detect the access point name channel status, network connection status, network bandwidth, and over-the-air (OTA) upgrade status of the vehicle network.

[0231] Here, the vehicle network status is obtained according to a preset detection cycle. In one implementation, the preset detection cycle is 1 second. The vehicle network status includes the configuration status, activation status, and data transmission status of each APN channel, the current network standard, whether the vehicle terminal has obtained a valid network address, whether the connection between the vehicle terminal and cloud storage is reachable, the current network bandwidth, and the operating status of the Over-the-Air (OTA) upgrade module.

[0232] When checking the APN channel status, the network interface information of the vehicle communication module is read through the Android Debug Bridge (ADB), and modem control commands (AT commands) are sent to the vehicle communication module. AT+CGDCONT is used to query the configured APN channels and their corresponding packet data protocol parameters, and AT+CGACT is used to query whether each APN channel is active. APN channels that are configured, activated, and capable of transmitting data are marked as available, while APN channels that are not configured, inactive, or unable to transmit data are marked as unavailable.

[0233] When detecting network connectivity, the interface status of 3G, 4G, 5G, and Wi-Fi networks is obtained. If at least one network interface is active, the vehicle terminal has obtained a valid network address, and the connection test for cloud storage passes, the network connectivity status is determined to be connected. If all network interfaces are inactive, the vehicle terminal has not obtained a valid network address, or the cloud storage connection fails multiple times consecutively, the network connectivity status is determined to be interrupted.

[0234] When testing network bandwidth, the amount of data sent and received by the network interface is recorded within a preset measurement period. The effective data transmission volume per unit time is determined based on the data volume at the beginning and end of the measurement period, and then converted into megabits per second (Mbps). The bandwidth results obtained from multiple consecutive measurement periods are averaged to reduce the impact of instantaneous fluctuations on the synchronization frequency.

[0235] When checking the over-the-air (OTA) upgrade status, the system reads the task status, upgrade package download progress, and upgrade package installation status output by the OTA upgrade module. If the task status is "Idle" or "Upgrade Completed," it indicates that no OTA upgrade task is currently being executed. If the task status is "Downloading Upgrade Package," "Verifying Upgrade Package," or "Installing Upgrade Package," it indicates that an OTA upgrade task is currently being executed. After the check is complete, a vehicle network status record with the check time is generated.

[0236] Step S802: When the network connection status is connected, determine the target access point name channel from the available access point name channels based on the data type, data sensitivity level, and access point name channel status of the data to be synchronized, and determine the transmission encryption method according to the data sensitivity level.

[0237] Here, the identifier, data type, and data sensitivity level of the data to be synchronized are read first, and then the current status of the APN2 control channel and the APN3 entertainment channel is queried. General voice command vectors, entertainment voice data, and data to be synchronized corresponding to the first data sensitivity level are preferentially selected through the APN3 entertainment channel. User voice habit vectors, privacy voice data, vehicle control parameters, and data to be synchronized corresponding to the second or third data sensitivity level are selected through the APN2 control channel.

[0238] When the APN3 entertainment channel is unavailable and the APN2 control channel is allowed to carry non-sensitive data, the APN2 control channel is designated as the target APN channel for the non-sensitive data to be synchronized. When the APN2 control channel is unavailable, the data to be synchronized corresponding to the second or third data sensitivity level is not transmitted through the APN3 entertainment channel; instead, the corresponding data to be synchronized remains in a pending synchronization state until the APN2 control channel becomes available again. When a switch from the APN3 entertainment channel to the APN2 control channel or vice versa is detected, the target APN channel selection is re-executed to avoid continuing to use an expired APN channel.

[0239] The data to be synchronized corresponding to the first data sensitivity level uses TLS 1.3 to establish an encrypted transmission connection. The data to be synchronized corresponding to the second and third data sensitivity levels remains AES-256 encrypted, and a TLS 1.3 encrypted transmission connection is established on top of AES-256 encryption, forming a dual encryption method combining storage data encryption and transmission channel encryption. Before establishing the transmission connection, the security control unit checks the key identifier, encryption algorithm version, and data integrity check value; if the check fails, the transmission of the corresponding data to be synchronized is stopped and the encryption check failure event is recorded.

[0240] Step S803: Determine the synchronization frequency of the data to be synchronized based on the network bandwidth.

[0241] Here, the synchronization frequency refers to the time interval between two consecutive synchronization tasks. In one implementation, when the network bandwidth is greater than or equal to 5 Mbps, the synchronization frequency is set to once per minute; when the network bandwidth is greater than or equal to 1 Mbps but less than 5 Mbps, the synchronization frequency is set to once every 5 minutes; when the network bandwidth is less than 1 Mbps, the synchronization frequency is set to once every 10 minutes, and the synchronization of non-core historical data and entertainment data is suspended, while core voice command data and user voice habit data are transmitted first.

[0242] After acquiring new network bandwidth over several consecutive detection cycles, the corresponding synchronization frequency is re-matched. When network bandwidth increases and consistently reaches a higher bandwidth range, the synchronization frequency is increased; when network bandwidth decreases to a lower bandwidth range, the synchronization frequency is immediately decreased to prevent the data to be synchronized from consuming network resources required by vehicle control or voice interaction services. The synchronization frequency adjustment results are correlated with the current network status record.

[0243] In step S804, when the over-the-air download upgrade status indicates that an over-the-air download upgrade task is being performed, the data to be synchronized is divided into data required for basic voice interaction and other data to be synchronized. The transmission priority of other data to be synchronized is reduced, and network bandwidth resources are reserved for data required for basic voice interaction.

[0244] Here, the data required for basic voice interaction includes general voice command vectors, reference text vectors, in-vehicle function operation information, currently valid index data, and basic voice interaction configuration data for common vehicle control functions such as turning on the air conditioner, closing the windows, and starting the navigation. Non-basic voice interaction data to be synchronized includes historical voice interaction data, entertainment-related voice data, redundant backup data, and data that does not affect the current voice control functions.

[0245] During an OTA upgrade task, the transmission priority of OTA upgrade data is set to the highest level, the transmission priority of data required for basic voice interaction is set to the reserved level, and the transmission priority of non-basic voice interaction data to be synchronized is reduced to the lowest level. Active synchronization of the lowest-level data is paused, and bandwidth resources not exceeding 10% of the currently available network bandwidth are reserved for data required for basic voice interaction. The remaining network bandwidth is prioritized for use by the OTA upgrade task. After the OTA upgrade task is completed, the synchronization frequency and the original transmission priorities of various types of data to be synchronized are restored, and the data to be synchronized accumulated during the OTA upgrade continues to be processed.

[0246] Step S805: The network address stored in the cloud is resolved through the vehicle intranet domain name resolution service, and the data to be synchronized is transmitted through the target access point name channel according to the synchronization frequency, transmission encryption method and corresponding transmission priority.

[0247] Here, the in-vehicle intranet domain name resolution service uses a preset Domain Name System (DNS) server address. The domain name stored in the cloud is sent to the primary DNS server, which receives the corresponding network address. If the primary DNS server does not return a valid network address within the preset resolution time, the domain name stored in the cloud is sent to the backup DNS server. If neither the primary nor the backup DNS server returns a valid network address, the current synchronization task is paused, the data to be synchronized is retained, and the domain name resolution failure event is recorded.

[0248] After obtaining the network address of the cloud storage, a communication connection is established with the cloud storage through the target APN channel, and identity verification and encrypted connection establishment are completed according to the transmission encryption method determined in step S802. The data transmission module divides the data to be synchronized into multiple data blocks according to the preset data block size, and adds the data identifier to be synchronized, data version, data block sequence number, total number of data blocks, starting byte position, data length, and integrity check value to each data block. Each data block inherits the transmission priority corresponding to the data to be synchronized.

[0249] The synchronization task is initiated according to the synchronization frequency, and data blocks are read from the synchronization queue in descending order of transmission priority. After each data block is sent, the system waits for confirmation of receipt from the cloud storage. Upon receiving confirmation, the sequence number and starting byte position of the data block that has been successfully transmitted are recorded, and the next data block is sent. After all data blocks have been transmitted, the cloud storage performs version and integrity checks on the complete data to be synchronized and returns a synchronization completion result. If both version and integrity checks pass, the synchronization status of the data to be synchronized is updated to synchronized complete.

[0250] Step S806: When the network connection is interrupted, the untransmitted data to be synchronized is stored in the vehicle solid-state storage and the transmission breakpoint is recorded; after the vehicle network is restored, the target access point name channel and synchronization frequency are re-determined, and the data to be synchronized is transmitted according to the transmission breakpoint.

[0251] Here, when the network connection status changes from connected to interrupted, the transmission of new data blocks stops, and untransmitted data blocks, data blocks that have been sent but for which no receipt confirmation has been received, and their corresponding data to be synchronized are retained in the vehicle's solid-state storage. User voice habit data is written separately to a priority recovery queue, while non-basic voice interaction data to be synchronized is written to a normal recovery queue. The vehicle's solid-state storage retains data to be synchronized during the interruption state for at least 24 hours.

[0252] The transmission breakpoint record includes the identifier of the data to be synchronized, the data version, the target APN channel, the transmission encryption method, the total number of data blocks, the sequence number of the last data block to receive a confirmation, the starting byte position of the next data block, the synchronization frequency, the interruption time, and the number of retries. The transmission breakpoint is bound to the corresponding data to be synchronized and stored, and written to the vehicle's solid-state storage to prevent the loss of transmission progress after the vehicle is powered off.

[0253] After the network connection is restored to a connected state, steps S802 and S803 are re-executed to determine the restored target APN channel, transmission encryption method, and synchronization frequency. The network address of the cloud storage is then re-resolved using the vehicle's intranet DNS. After establishing a new encrypted connection, the sequence number of the received data block corresponding to the data to be synchronized is queried from the cloud storage, and the sequence number returned by the cloud storage is compared with the local transmission breakpoint. If the two data block sequence numbers match, transmission continues from the starting byte position of the next data block; if the two data block sequence numbers do not match, the local transmission breakpoint is updated based on the sequence number of the data block already confirmed as received by the cloud storage.

[0254] If the cloud data version is inconsistent with the local data version to be synchronized, or if the integrity check of the transmitted data fails, the current transmission breakpoint is cleared, and the corresponding data to be synchronized is retransmitted from the first data block. When resuming transmission, user voice habit data and data required for basic voice interaction are processed first, followed by data to be synchronized in the normal recovery queue. When the target APN channel is switched, a new TLS 1.3 connection is established, and the transmission encryption method corresponding to the data sensitivity level continues to be used.

[0255] The in-vehicle voice interaction data processing method provided in this application can dynamically select a transmission strategy based on the APN channel status, network bandwidth, data sensitivity level, and OTA upgrade task, and continue synchronization from the confirmed transmission position after network interruption, thereby reducing duplicate transmission and synchronization data loss, while ensuring the data transmission required for basic voice interaction functions during OTA upgrade.

[0256] In an optional implementation, the in-vehicle multimodal vector database also stores multiple user voice habit vectors corresponding to the current user, and each user voice habit vector is associated with the corresponding in-vehicle function operation information.

[0257] Here, the user's voice habit vector is used to represent the stable voice expression features formed by the current user for corresponding in-vehicle function operations. Stable voice expression features include the current user's accent, pronunciation style, speech rate, frequently used words, command omission methods, and usage habits of wake words and modifiers. In-vehicle function operation information includes function identifiers, operation types, and operation parameters. For example, in-vehicle function operation information related to air conditioning control includes turning on the air conditioning, turning off the air conditioning, and adjusting the air conditioning temperature to the target temperature; in-vehicle function operation information related to navigation includes starting navigation and navigating to the target location. When the in-vehicle function operation information includes operation parameters, the association relationship also retains the operation parameters, thereby recording the current user's commonly used air conditioning temperature, preferred navigation destination or route, and commonly used music style. The user's voice habit vector is associated with the standardized in-vehicle function operation information, not directly with the original text of historical voice commands, thus allowing different expressions to be categorized under the same in-vehicle function operation.

[0258] The vehicle terminal reads the user identifier from the currently logged-in vehicle account and identifies the user corresponding to the user identifier as the current user. The vehicle multimodal vector database distinguishes different users' voice habit vectors using a combination of user identifier and vehicle function operation identifier. When the vehicle terminal does not obtain a valid user identifier, it does not read the user's voice habit vector corresponding to any registered user, and completes voice command matching according to general reference data to prevent the mixing of voice habit data between different users.

[0259] User voice habit vectors are stored as voice privacy data in a voice privacy encrypted partition of the in-vehicle solid-state storage, and are encrypted, verified, and synchronized to the cloud according to data sensitivity levels, access permissions, and synchronization policies. The generation or updating of user voice habit vectors is performed according to a preset update cycle. In one embodiment, the preset update cycle is 24 hours. Each time a generation or update task is performed, the valid voice interaction data generated by the current user within the new historical time period is processed according to the following steps S901-S904.

[0260] User voice habit vectors are generated or updated through the following steps S901-S904.

[0261] Step S901: Obtain the current user's historical voice commands within a preset historical time period and the historical vehicle function operation information corresponding to each historical voice command.

[0262] Here, the preset historical time period is used to limit the range of historical voice interaction data participating in this aggregation. In one implementation, the preset historical time period is 30 days prior to the current update time. The vehicle terminal reads the historical voice commands issued by the current user within the preset historical time period from the voice interaction log based on the current user's user ID, the collection time of historical voice commands, and the voice interaction event ID, and reads the vehicle control execution results corresponding to each voice interaction event ID from the vehicle control log.

[0263] Voice interaction event identifiers are generated when the in-vehicle voice interaction module receives a voice command. These identifiers are recorded along with the voice command, recognition result, in-vehicle function operation information, and vehicle control execution result, thus establishing a correspondence between historical voice commands and actual in-vehicle function operations. When vehicle control is successfully executed, the actually executed in-vehicle function operation information is identified as historical in-vehicle function operation information. If the current user cancels an operation before the vehicle execution, the corresponding historical voice command is not considered a valid sample. When the current user corrects the recognition result during a voice interaction, the successfully executed in-vehicle function operation information after correction is identified as historical in-vehicle function operation information, and the in-vehicle function operation information that was not executed before the correction is marked as invalid.

[0264] After acquiring historical voice commands, their validity is filtered. Validity filtering includes checking the completeness of the voice data, the consistency of user identifiers, the success of vehicle control execution, and whether the voice quality meets minimum quality requirements. Samples are deleted if the voice data lacks audio segments, the voice interaction event identifier does not correspond to a vehicle control record, the voice clarity level is lower than the preset minimum level, or the noise interference level is higher than the preset maximum level. For duplicate records with identical content, consecutive acquisition times, and the same voice interaction event identifier, only one record is retained. The filtered historical voice commands and their corresponding historical vehicle function operation information constitute a historical habit sample.

[0265] Step S902: In accordance with the method of feature encoding for voice commands, feature encoding is performed on each historical voice command to obtain a historical voice vector.

[0266] Here, historical voice commands are encoded using the same voice preprocessing flow, voice feature extractor, semantic alignment algorithm, model parameter version, and vector dimension as in step S102, ensuring that historical voice vectors, voice query vectors, and reference voice vectors reside in the same common semantic space. Specifically, historical voice commands are first converted into audio data with a sampling rate of 16kHz and a quantization bit depth of 16bit. Then, Wiener filtering algorithm is used to reduce engine noise and wind noise, normalized least mean square algorithm is used to eliminate echoes generated by in-vehicle speakers, and effective voice segments are extracted using endpoint detection combining short-time energy and zero-crossing rate, and these effective voice segments are then normalized.

[0267] After preprocessing, the valid speech segments are input into the Mel-CNN feature extractor to extract Mel spectral and semantic features, resulting in a 128-dimensional initial historical speech vector. Then, using the preset semantic alignment algorithm employed in step S203, the initial historical speech vector is mapped to the common semantic space containing the reference speech vector, and the mapping result is normalized to obtain the final historical speech vector. Each historical speech vector is recorded along with its corresponding user identifier, voice interaction event identifier, acquisition time, historical in-vehicle function operation information, speech quality level, and coding model version.

[0268] When the encoding model version changes, the valid historical speech instructions within the preset historical period are re-encoded according to the updated encoding model before subsequent aggregation is performed, so as to avoid vectors generated by different encoding model versions from directly participating in the same aggregation.

[0269] Step S903: Group the historical voice vectors according to the historical vehicle function operation information, and aggregate each group of historical voice vectors to obtain the updated user voice habit vectors associated with each historical vehicle function operation information.

[0270] Before grouping, historical in-vehicle function operation information is standardized. The standardization process maps function names to unified function identifiers, colloquial operation descriptions to unified operation types, and operation parameters to corresponding parameter formats. Historical voice vectors with the same function identifier, operation type, and operation parameters are grouped into the same vector group. For example, if the vehicle control execution result for "turn on the air conditioning," "turn on the air conditioning," and "turn on the air conditioning briefly" is "turn on the air conditioning," the corresponding historical voice vectors are grouped into the vector group corresponding to "turn on the air conditioning." When continuously adjusting parameters are involved, the operation parameters are merged according to a preset parameter granularity. For example, if the preset parameter granularity for air conditioning temperature is 1℃, historical voice vectors with the same target temperature are grouped into the same vector group.

[0271] In one implementation, each group of historical speech vectors is aggregated according to the following process: First, the average of each historical speech vector within the same vector group is calculated along its vector dimension to obtain an initial center vector. Then, the cosine similarity between each historical speech vector and the initial center vector is calculated. Historical speech vectors with a cosine similarity lower than a preset aggregation threshold are identified as outliers and removed from the current vector group. The preset aggregation threshold is set to 0.75. If the number of valid historical speech vectors is less than 3 after removing outliers, no new user speech habit vector is generated; if the same user speech habit vector corresponding to the same in-vehicle function operation information already exists in the in-vehicle multimodal vector database, the original user speech habit vector is retained.

[0272] When the number of valid historical speech vectors reaches three, time weights are assigned to these vectors based on their acquisition time. Valid historical speech vectors no more than 7 days after the current update correspond to a first time weight, while those more than 7 days after the update correspond to a second time weight, with the first time weight being greater than the second. In one implementation, the first time weight is 0.7, and the second time weight is 0.3. The vectors are first averaged across the two time ranges, then multiplied by their respective time weights and summed to obtain the aggregated vector. If no valid historical speech vector exists within a time range, the average result from the other time range is directly used as the aggregated vector. The aggregated vector is then normalized to obtain a 128-dimensional updated user speech habit vector. This updated user speech habit vector allows speech expressions closer to the current moment to significantly influence the aggregation result, thus reflecting changes in the user's accent, preferred expressions, and instruction omission methods.

[0273] Step S904: For each updated user voice habit vector, if there is a user voice habit vector associated with the same vehicle function operation information in the vehicle multimodal vector database, replace the user voice habit vector with the updated user voice habit vector; if there is no user voice habit vector associated with the same vehicle function operation information, add the updated user voice habit vector to the vehicle multimodal vector database and associate it with the corresponding vehicle function operation information.

[0274] Here, before writing the updated user voice habit vector, the dimension, data format, normalization result, number of valid samples, and encoding model version of the updated user voice habit vector are checked. If the check results do not meet the data specifications of the current in-vehicle multimodal vector database, the writing process is terminated and the original user voice habit vector is retained. If the check results meet the data specifications, the in-vehicle multimodal vector database is queried based on the combined identifier composed of the current user's user identifier and the in-vehicle function operation identifier.

[0275] When the original user voice habit vector corresponding to the combined identifier is found, a new data version and update time are generated for the updated user voice habit vector. First, the updated user voice habit vector is written to a temporary record, and then an integrity check is performed on the temporary record. After the integrity check passes, the data record pointed to by the combined identifier is atomically replaced with the temporary record, and the original user voice habit vector is marked as invalid. Atomic replacement means that after a database write transaction is completed, the retrieval request can only read either the original user voice habit vector or the updated user voice habit vector, and cannot read intermediate data that has not yet been fully written.

[0276] When no user voice habit vector corresponding to the combined identifier is found, a vector identifier and an initial data version are generated for the updated user voice habit vector. The updated user voice habit vector is written to the voice privacy encrypted partition, and an association is established between the updated user voice habit vector and the user identifier and in-vehicle function operation information. After the replacement or addition is completed, incremental updates are performed on the tree index, hash index, and semantic association index corresponding to the user voice habit vector, and user voice habit vectors with high retrieval frequency are loaded into the local cache. The incremental update time is controlled within 100ms. When synchronization to cloud storage is required, synchronization is completed through the APN2 control channel using the encryption method described above.

[0277] Every 7 days, the in-vehicle terminal performs a consistency check on the user voice habit vectors stored locally and in the cloud. The in-vehicle terminal uses the MD5 hash checksum algorithm to calculate the checksums for the local and cloud-based user voice habit vectors respectively, and compares the two checksums. If the two checksums are inconsistent, the user voice habit vector with the newest version and passed the integrity check in the in-vehicle solid-state storage is used as the synchronization source, triggering data synchronization and repair. When the in-vehicle solid-state storage reaches the preset cleanup conditions, valid user voice habit vectors are retained first to avoid frequently used voice habit data being deleted along with low-frequency historical interaction data.

[0278] In step S104, the first similarity between the speech query vector and each reference speech vector is calculated, including the following steps S1001-S1004.

[0279] Step S1001: Calculate the basic speech similarity between the speech query vector and each reference speech vector.

[0280] Here, the speech query vector and the reference speech vector use the same vector dimension, encoding model version, and normalization method. After locating the reference speech vector to be compared using a tree index or hash index, the cosine similarity algorithm is used to calculate the similarity between the speech query vector and each reference speech vector, and the calculation result is determined as the basic speech similarity of the corresponding reference speech vector. After the speech query vector and the reference speech vector have been normalized, they are multiplied item by item according to the same vector dimension and then summed to obtain the basic speech similarity. The larger the basic speech similarity, the closer the current speech command is to the reference speech sample in terms of acoustic and semantic features.

[0281] Step S1002: For each reference voice vector, query whether there exists a user voice habit vector that is associated with the same vehicle function operation information as the reference voice vector.

[0282] Here, the semantic association index is first queried based on the vector identifier of the reference speech vector to obtain the vehicle function operation identifier associated with the reference speech vector. Then, a combined query condition is formed based on the current user's user identifier and the vehicle function operation identifier to query the user's speech habit vector from the vehicle multimodal vector database. If the data status corresponding to the query result is valid, the vector dimension matches the speech query vector, and the encoding model version matches the speech query vector, it is determined that a user speech habit vector exists that can participate in this matching. If no corresponding record is found, the corresponding record is invalid, the vector dimension is inconsistent, or the encoding model version is inconsistent, it is determined that no user speech habit vector exists that can participate in this matching.

[0283] The query process is only performed within the data range corresponding to the current user. When the reference voice vector is associated with another user's voice habit vector for the same in-vehicle function operation information, the other user's voice habit vector is not read to avoid the influence of different users' accents and expression habits on the matching results of the current user. High-frequency user voice habit vectors are directly provided by the local cache, while low-frequency user voice habit vectors are read from the in-vehicle solid-state storage through semantic association index, so that the overall response time of voice commands after personalized matching is kept within 50ms.

[0284] Step S1003: If it exists, calculate the habit similarity between the voice query vector and the queried user voice habit vector, and perform weighted calculation on the basic voice similarity and habit similarity according to the preset weight to obtain the first similarity corresponding to the reference voice vector.

[0285] Here, habitual similarity is calculated using the same cosine similarity algorithm as basic speech similarity, reflecting the degree of closeness between the current voice command and the user's historical expression habits for the corresponding in-vehicle function operations. Basic speech similarity and habitual similarity are calculated using the same numerical range before weighting to avoid any unexpected impact on the calculation results due to different numerical ranges.

[0286] The preset weights include a basic speech weight and a habit weight, the sum of which is 1. In one implementation, the basic speech weight is 0.7 and the habit weight is 0.3. The basic speech similarity is multiplied by the basic speech weight, and the habit similarity is multiplied by the habit weight. The two products are then added together to obtain the first similarity corresponding to the reference speech vector. The basic speech weight is higher than the habit weight, ensuring that the general semantic matching result remains dominant; the habit weight personalizes the matching result, ensuring that speech commands with the current user's accent, common abbreviations, or omitted expressions obtain matching results consistent with historical usage habits.

[0287] After obtaining the first similarity corresponding to each reference speech vector, the first candidate vector set is filtered according to the dynamic similarity threshold determined in step S104. The user's speech habit vector only participates in the first similarity calculation of the speech retrieval path and does not change the second similarity calculation method between the text query vector and the reference text vector.

[0288] Step S1004: If it does not exist, then the basic speech similarity is used as the first similarity corresponding to the reference speech vector.

[0289] Here, when a user is using the in-vehicle voice interaction function for the first time, there are insufficient valid historical habit samples, the corresponding in-vehicle function operation has not yet formed a user voice habit vector, or the user voice habit vector cannot participate in matching due to data status, vector dimension, or encoding model version not meeting the requirements, the basic voice similarity is directly used to complete the first candidate vector set selection. The basic voice similarity is not reduced or compensated due to the lack of user voice habit vectors, so that voice command recognition can still be completed through general reference voice vectors when personalized data is missing.

[0290] After a vehicle function operation is successfully executed, the voice command, the current user's identifier, the actual vehicle function operation information, and the vehicle control execution result are written to the voice interaction log. This voice command, after being processed by steps S901-S904 in the next update cycle, participates in the user's voice habit vector update. If a vehicle function operation is not executed or the current user corrects the vehicle function operation, only the information of the successfully executed vehicle function operation is used for updating, thus forming an update process from historical interaction learning, personalized matching to the feedback of new interaction data.

[0291] By generating, updating, and weighted matching user voice habit vectors, the vehicle terminal can adapt to different users' accents, common abbreviations, vague expressions, and command omissions while retaining the general reference voice vector matching capability. This reduces the probability of personalized voice commands being mismatched and continuously corrects the matching results when user voice habits change.

[0292] In one specific embodiment, the in-vehicle multimodal vector database is applied to a passenger vehicle traveling on a highway. Reference data for air conditioning control, navigation, window control, and multimedia control has been written into the vehicle before it leaves the factory. Taking the air conditioning temperature adjustment function as an example, the in-vehicle multimodal vector database stores reference voice vectors corresponding to Mandarin, Sichuan accent, and Cantonese accent, and reference text vectors corresponding to complete or simplified command texts such as "Please adjust the air conditioning to 24 degrees," "Air conditioning 24 degrees," and "24-degree air conditioning." The aforementioned reference voice vectors and reference text vectors are collectively associated with the in-vehicle function operation information for setting the target air conditioning temperature to 24°C. The in-vehicle multimodal vector database also stores reference data corresponding to in-vehicle function operations such as setting the target air conditioning temperature to 14°C, turning on the air conditioning, and turning off the air conditioning.

[0293] After the driver logs into the vehicle account with user identifier U001, the vehicle terminal identifies the driver as the current user. The vehicle terminal records 15 historical voice commands related to setting the air conditioning to 24 degrees Celsius issued by the current user within the last 30 days. After voice data integrity checks, execution result checks, and voice quality screening, 12 historical voice commands are identified as valid historical habit samples. These 12 historical voice commands include expressions such as "air conditioning to 24," "adjust to 24," and "temperature set to 24." The vehicle terminal performs feature encoding and grouping aggregation on these 12 historical voice commands to obtain a user voice habit vector associated with setting the target air conditioning temperature to 24°C, and writes this user voice habit vector into the voice privacy encryption partition.

[0294] When the vehicle is traveling at 100 km / h, wind noise, tire noise, and music are present inside the vehicle. The user currently issues a voice command: "Air conditioning 24 degrees." The in-vehicle voice interaction module collects the voice command and transcribes it into the text "Air conditioning 24 degrees." The in-vehicle terminal performs feature encoding on both the voice command and the text command, obtaining a 128-dimensional voice query vector and a 128-dimensional text query vector.

[0295] The vehicle-mounted terminal maps the speech clarity parameter and noise interference level parameter to a numerical range of 0 to 1. The terminal performs a quality assessment on the current voice command, obtaining a speech clarity parameter of 0.63 and a noise interference level parameter of 0.66. Based on the preset parameter range, the terminal classifies the current voice command as a low-clarity, high-noise-interference combination, and, according to the correspondence between the combination and the similarity threshold, sets the dynamic similarity threshold to 0.75.

[0296] The in-vehicle terminal calculates the basic voice similarity between the voice query vector and each reference voice vector. The basic voice similarity between the voice query vector and the reference voice vector corresponding to the air conditioning temperature being set to 24℃ is 0.78, the basic voice similarity between the voice query vector and the reference voice vector corresponding to the air conditioning temperature being set to 14℃ is 0.74, and the basic voice similarity between the voice query vector and the reference voice vector corresponding to turning on the air conditioning is 0.71. The in-vehicle terminal finds that the current user has a user voice habit vector associated with the air conditioning temperature being set to 24℃, and calculates a habit similarity of 0.91. After weighting according to a basic voice weight of 0.7 and a habit weight of 0.3, the first similarity corresponding to the air conditioning temperature being set to 24℃ is 0.819. The current user does not have any user voice habit vectors associated with the air conditioning temperature being set to 14℃ or turning on the air conditioning; therefore, the first similarities corresponding to the air conditioning temperature being set to 14℃ and turning on the air conditioning remain at 0.74 and 0.71, respectively.

[0297] The reference speech vectors whose first similarity reaches the dynamic similarity threshold only include the reference speech vector corresponding to the air conditioning temperature being set to 24℃, and the first candidate vector set is not empty. Although the vehicle terminal simultaneously calculates the second similarity between the text query vector and the reference text vectors corresponding to the air conditioning temperature being set to 24℃, 14℃, and the air conditioning being turned on as 0.94, 0.76, and 0.72 respectively, the vehicle terminal determines the reference speech vector in the first candidate vector set as the target matching vector according to the matching order prioritizing speech vectors. The vehicle terminal reads the vehicle function operation information associated with the target matching vector, generates a vehicle control command to set the target air conditioning temperature to 24℃, and sends the vehicle control command to the air conditioning controller. After the air conditioning controller sets the target temperature to 24℃ and returns a success message, the vehicle voice interaction module outputs feedback that the air conditioning has been adjusted to twenty-four degrees.

[0298] After the vehicle continues driving and enters the tunnel, the current user issues a voice command to navigate to the company. The voice clarity and noise interference level of this command still fall into the low-clarity, high-noise-interference combination, and the vehicle terminal sets the dynamic similarity threshold to 0.75. The highest first similarity calculated by the vehicle terminal is 0.73. Since the first similarity of all reference voice vectors is lower than the dynamic similarity threshold, the first candidate vector set is empty. The vehicle terminal performs text vector matching on the transcribed command text to navigate to the company. The second similarity between the text query vector and the reference text vector corresponding to the navigation to the company is 0.92, which is higher than the second similarity of the reference text vectors other than the navigation to the company. The vehicle terminal determines the reference text vector corresponding to the navigation to the company as the target matching vector. Based on the vehicle function operation information associated with the target matching vector, it reads the company address pre-saved by the current user and generates a vehicle control command to start navigation and set the company address as the destination. The navigation module begins planning the route based on the vehicle control command. Through this text vector backtracking matching, the vehicle terminal was able to complete the recognition of vehicle function operations even when the voice acoustic features could not reach the dynamic similarity threshold due to the influence of tunnel noise.

[0299] After successful execution of air conditioning control and navigation operations, the in-vehicle terminal records the current user's voice command, voice interaction event identifier, actual in-vehicle function operation information, and execution result. When the next user voice habit vector update cycle arrives, the in-vehicle terminal includes the two voice interaction data into the historical habit sample screening range, and updates the corresponding user voice habit vector and associated index based on the valid samples.

[0300] When the vehicle enters the tunnel, the in-vehicle network is interrupted. Unsynchronized user voice habit data and voice interaction records are stored in the in-vehicle solid-state storage, along with the location of data blocks that have already been transmitted. After the vehicle exits the tunnel and the in-vehicle network is restored, the in-vehicle terminal re-detects the network bandwidth and APN channel status, and resumes synchronization of user voice habit data from the point of interruption via the APN2 control channel. If the vehicle is simultaneously performing an OTA upgrade and the network bandwidth is below 1Mbps, the in-vehicle terminal reduces the synchronization frequency of non-basic voice interaction data to once every 10 minutes, and reserves no more than 10% of the currently available network bandwidth for the data required for basic voice interaction, thereby preventing cloud synchronization and OTA upgrades from affecting the current voice control process.

[0301] This application's embodiments demonstrate two processing results—voice vector priority matching and text vector fallback matching—through the same driving process. User voice habit vectors improve the matching degree between voice commands with accents and simplified expressions and corresponding in-vehicle function operations, while text query vectors provide supplementary matching paths when voice candidates are empty. Hierarchical storage and breakpoint resume mechanisms ensure that user voice habit data is saved and subsequently synchronized during in-vehicle network interruptions and OTA upgrades.

[0302] Based on this, embodiments of this application provide an in-vehicle voice interaction data processing system, applied to an in-vehicle terminal equipped with an in-vehicle multimodal vector database. The in-vehicle multimodal vector database stores multiple sets of reference data, each set of reference data including semantically related reference speech vectors and reference text vectors, as well as in-vehicle function operation information associated with the reference speech vectors and reference text vectors. (Refer to...) Figure 4 The in-vehicle voice interaction data processing system provided in this application includes: Data acquisition module 1 is used to acquire the voice commands collected by the in-vehicle voice interaction module and the corresponding command text.

[0303] Feature encoding module 2 is used to encode the features of the voice command and the command text respectively, so as to obtain the voice query vector and the text query vector.

[0304] The threshold determination module 3 is used to determine the dynamic similarity threshold based on the speech quality parameters of the speech command.

[0305] Vector matching module 4 is used to calculate the first similarity between the speech query vector and each reference speech vector, form a first candidate vector set with reference speech vectors whose first similarity is greater than or equal to the dynamic similarity threshold, and calculate the second similarity between the text query vector and each reference text vector.

[0306] The vector matching module 4 is further configured to determine the reference speech vector with the highest first similarity as the target matching vector if the first candidate vector set is not empty; otherwise, determine the reference text vector with the highest second similarity as the target matching vector.

[0307] The instruction generation module 5 is used to generate vehicle control instructions based on the vehicle function operation information associated with the target matching vector.

[0308] The in-vehicle voice interaction data processing system provided in this application embodiment processes voice commands and corresponding command texts collaboratively through a data acquisition module, a feature encoding module, a threshold determination module, a vector matching module, and an instruction generation module. It can dynamically adjust the matching conditions of voice vectors according to voice quality. When there are valid voice candidates, it prioritizes the voice vector matching results. When there are no valid voice candidates due to in-vehicle noise, accent deviation, or ambiguous expression, it uses the text vector matching results. This fully utilizes the complementarity between voice acoustic features and text semantic features, reduces the probability of mismatch of in-vehicle function operation information, improves the accuracy, adaptability, and stability of voice command recognition in complex in-vehicle environments, and quickly generates vehicle control commands based on the matching results, thereby improving the response efficiency and user experience of in-vehicle voice interaction.

[0309] In an optional implementation, feature encoding module 2 is further configured to: The acoustic and semantic features of the speech commands are extracted to obtain the initial speech vector.

[0310] Extract the semantic features of the instruction text to obtain the initial text vector.

[0311] The initial speech vector and initial text vector are mapped to the common semantic space where the reference speech vector and reference text vector are located by a preset semantic alignment algorithm, respectively, to obtain the speech query vector and text query vector.

[0312] In an optional implementation, the speech quality parameters include speech intelligibility and noise interference level. The threshold determination module 3 is further configured to: The quality of voice commands is evaluated to obtain the speech clarity and noise interference level of the voice commands.

[0313] The speech clarity and noise interference levels are matched with corresponding preset parameter ranges to obtain the speech clarity level and noise interference level.

[0314] Combine the speech clarity level and noise interference level into a target level combination.

[0315] Based on the pre-defined correspondence between level combinations and similarity thresholds, the dynamic similarity threshold corresponding to the target level combination is determined. Among them, the higher the speech clarity level, the higher the speech clarity; the higher the noise interference level, the greater the noise interference. Under the same noise interference level, the dynamic similarity threshold increases with the increase of speech clarity level; under the same speech clarity level, the dynamic similarity threshold decreases with the increase of noise interference level.

[0316] In an optional implementation, the in-vehicle voice interaction data processing system further includes a storage module, which is used for: Create a vector storage structure to store multiple sets of reference data.

[0317] Acquire multiple reference voice samples, reference instruction texts corresponding to each reference voice sample, and vehicle function operation information; wherein, the multiple reference voice samples include different accent voice samples corresponding to the same vehicle function operation information, and the reference instruction texts include complete instruction texts and simplified instruction texts corresponding to the same vehicle function operation information; the simplified instruction texts are instruction texts that omit the wake word and / or omit some modifiers.

[0318] Each reference speech sample and its corresponding reference instruction text are feature-encoded to obtain the reference speech vector for each reference speech sample and the reference text vector for each reference instruction text.

[0319] Based on the semantic similarity between the reference speech vector and the reference text vector, a semantic association relationship is established between the corresponding reference speech vector and the reference text vector.

[0320] Each set of semantically related reference speech vectors and reference text vectors is associated with the corresponding in-vehicle function operation information to form a set of reference data.

[0321] The generated sets of reference data are written into a vector storage structure to obtain an in-vehicle multimodal vector database.

[0322] In an optional implementation, the storage module is further configured to: Construct an objective function to optimize the vector distribution based on the vector distribution of the reference speech vector and the reference text vector.

[0323] Set the regularization coefficient, iteration step size, and convergence threshold. Iterate and update the reference speech vector and reference text vector according to the objective function, and calculate the change between the objective function values ​​corresponding to two adjacent iterations.

[0324] When the change is greater than the convergence threshold, adjust the regularization coefficient and iteration step size and continue iterating; when the change is less than or equal to the convergence threshold, stop iterating.

[0325] The updated reference speech vectors and reference text vectors are quantized, and the quantized reference speech vectors and reference text vectors are used to form the corresponding reference data.

[0326] In an optional implementation, the storage module is further configured to: A tree index is constructed based on the reference speech vector and the reference text vector.

[0327] Obtain the retrieval frequency of each reference speech vector and each reference text vector, and construct a hash index for the reference speech vectors and reference text vectors whose retrieval frequency reaches the preset frequency condition.

[0328] Based on the semantic relationship between the reference speech vector and the reference text vector, as well as their relationship with the vehicle function operation information, a semantic relationship index is constructed between the reference speech vector, the reference text vector, and the vehicle function operation information.

[0329] When reference speech vectors, reference text vectors, or vehicle function operation information are inserted, deleted, or updated, the affected indexes in the tree index, hash index, and semantic association index are incrementally updated.

[0330] In optional implementations, the vector storage architecture includes local caching, on-board solid-state storage, and cloud storage. The storage module is also used for: Each set of reference data is designated as a separate set of stored data.

[0331] Determine the data sensitivity level of each group of stored data, and write each group of stored data into the corresponding storage partition in the vehicle solid-state storage according to the data sensitivity level. Among them, the stored data with a data sensitivity level that reaches the preset sensitivity level is encrypted and then written into the corresponding encrypted storage partition.

[0332] Obtain the retrieval frequency of each group of stored data, and load the stored data whose retrieval frequency reaches the preset frequency condition into the local cache.

[0333] Data that meets the preset backup conditions is identified as data to be synchronized, and the data to be synchronized is synchronized to cloud storage.

[0334] The storage module is also used for: In response to a data access request for stored data, the system performs access permission verification on the requesting module that initiated the data access request. If the verification passes, the system provides the requested stored data to the requesting module. If the verification fails, the system rejects the data access request and records the rejected access event.

[0335] In an optional implementation, the storage module is further configured to: Detect the access point name, channel status, network connection status, network bandwidth, and over-the-air (OTA) upgrade status of the vehicle network.

[0336] When the network connection status is connected, the target access point name channel is determined from the available access point name channels based on the data type, data sensitivity level, and access point name channel status of the data to be synchronized, and the transmission encryption method is determined according to the data sensitivity level.

[0337] The synchronization frequency of the data to be synchronized is determined based on the network bandwidth.

[0338] When the over-the-air download upgrade status indicates that an over-the-air download upgrade task is being performed, the data to be synchronized is divided into data required for basic voice interaction and other data to be synchronized. The transmission priority of other data to be synchronized is reduced, while network bandwidth resources are reserved for data required for basic voice interaction.

[0339] The network address stored in the cloud is resolved through the vehicle's intranet domain name resolution service, and the data to be synchronized is transmitted through the target access point name channel according to the synchronization frequency, transmission encryption method and corresponding transmission priority.

[0340] When the network connection is interrupted, the untransmitted data to be synchronized is stored in the vehicle's solid-state storage and the transmission breakpoint is recorded. After the vehicle network is restored, the target access point name, channel and synchronization frequency are re-determined, and the data to be synchronized is transmitted again according to the transmission breakpoint.

[0341] In an optional implementation, the in-vehicle multimodal vector database also stores multiple user voice habit vectors corresponding to the current user, each user voice habit vector being associated with corresponding in-vehicle function operation information. The system provided in this application embodiment also includes a network adaptation module, which is used for: Obtain the current user's historical voice commands within a preset historical time period, as well as the historical in-vehicle function operation information corresponding to each historical voice command.

[0342] Based on the method of feature encoding of voice commands, feature encoding is performed on each historical voice command to obtain a historical voice vector.

[0343] The historical voice vectors are grouped according to the historical in-vehicle function operation information, and each group of historical voice vectors is aggregated to obtain the updated user voice habit vector associated with each historical in-vehicle function operation information.

[0344] For each updated user voice habit vector, if a user voice habit vector associated with the same in-vehicle function operation information exists in the in-vehicle multimodal vector database, the updated user voice habit vector is used to replace the user voice habit vector; if no user voice habit vector associated with the same in-vehicle function operation information exists, the updated user voice habit vector is added to the in-vehicle multimodal vector database and associated with the corresponding in-vehicle function operation information.

[0345] Vector matching module 4 is also used for: Calculate the basic speech similarity between the speech query vector and each reference speech vector.

[0346] For each reference voice vector, query whether there exists a user voice habit vector that is associated with the same in-vehicle function operation information as that reference voice vector.

[0347] If it exists, calculate the habit similarity between the voice query vector and the queried user voice habit vector, and perform a weighted calculation on the basic voice similarity and habit similarity according to the preset weights to obtain the first similarity corresponding to the reference voice vector.

[0348] If it does not exist, the basic speech similarity will be used as the first similarity corresponding to the reference speech vector.

[0349] The computer program product provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0350] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0351] Furthermore, in the description of the embodiments of this application, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0352] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0353] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0354] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A method for processing in-vehicle voice interaction data, characterized in that, The method is applied to an in-vehicle terminal equipped with an in-vehicle multimodal vector database, wherein the in-vehicle multimodal vector database stores multiple sets of reference data, each set of reference data including semantically related reference speech vectors and reference text vectors, as well as in-vehicle function operation information jointly associated with the reference speech vectors and the reference text vectors; the method includes: Acquire the voice commands collected by the in-vehicle voice interaction module and the corresponding command text; The voice command and the command text are respectively feature-encoded to obtain a voice query vector and a text query vector; A dynamic similarity threshold is determined based on the voice quality parameters of the voice command; Calculate the first similarity between the speech query vector and each of the reference speech vectors, form a first candidate vector set with reference speech vectors whose first similarity is greater than or equal to the dynamic similarity threshold, and calculate the second similarity between the text query vector and each of the reference text vectors; If the first candidate vector set is not empty, the reference speech vector with the highest first similarity is determined as the target matching vector; otherwise, the reference text vector with the highest second similarity is determined as the target matching vector. Vehicle control commands are generated based on the vehicle function operation information associated with the target matching vector.

2. The in-vehicle voice interaction data processing method according to claim 1, characterized in that, The step of performing feature encoding on the voice command and the command text respectively to obtain a voice query vector and a text query vector includes: The acoustic and semantic features of the voice command are extracted to obtain an initial voice vector; Extract the textual semantic features of the instruction text to obtain an initial text vector; The initial speech vector and the initial text vector are mapped to the common semantic space where the reference speech vector and the reference text vector are located by a preset semantic alignment algorithm, respectively, to obtain the speech query vector and the text query vector.

3. The in-vehicle voice interaction data processing method according to claim 1, characterized in that, The speech quality parameters include speech clarity and noise interference level; The step of determining the dynamic similarity threshold based on the speech quality parameters of the speech command includes: The voice command is evaluated for quality to obtain the voice clarity and noise interference level of the voice command; The speech clarity and the noise interference level are matched with corresponding preset parameter ranges to obtain the speech clarity level and the noise interference level. The speech clarity level and the noise interference level are combined into a target level combination; Based on the preset correspondence between level combinations and similarity thresholds, a dynamic similarity threshold corresponding to the target level combination is determined; wherein, a higher speech clarity level indicates higher speech clarity; a higher noise interference level indicates a greater degree of noise interference; when the noise interference levels are the same, the dynamic similarity threshold increases with the increase of the speech clarity level; when the speech clarity levels are the same, the dynamic similarity threshold decreases with the increase of the noise interference level.

4. The in-vehicle voice interaction data processing method according to claim 1, characterized in that, The vehicle-mounted multimodal vector database is constructed using the following method: Create a vector storage structure for storing multiple sets of the reference data; Acquire multiple reference voice samples, reference instruction texts corresponding to each reference voice sample, and vehicle function operation information; wherein, the multiple reference voice samples include different accent voice samples corresponding to the same vehicle function operation information, and the reference instruction texts include complete instruction texts and simplified instruction texts corresponding to the same vehicle function operation information; the simplified instruction texts are instruction texts that omit wake words and / or omit some modifiers; Each of the reference speech samples and its corresponding reference instruction text is feature-encoded to obtain a reference speech vector for each reference speech sample and a reference text vector for each reference instruction text. Based on the semantic similarity between the reference speech vector and the reference text vector, establish the corresponding semantic association between the reference speech vector and the reference text vector; Each set of reference speech vectors and reference text vectors with the aforementioned semantic relationship are jointly associated with the corresponding in-vehicle function operation information to form a set of reference data; The generated sets of reference data are written into the vector storage structure to obtain the vehicle-mounted multimodal vector database.

5. The in-vehicle voice interaction data processing method according to claim 4, characterized in that, Before associating each set of reference speech vectors and reference text vectors with the aforementioned semantic relationship to the corresponding in-vehicle function operation information to form a set of reference data, the method further includes: Construct an objective function to optimize the vector distribution based on the vector distribution of the reference speech vector and the reference text vector; Set the regularization coefficient, iteration step size, and convergence threshold; iterate and update the reference speech vector and the reference text vector according to the objective function; and calculate the change between the objective function values ​​corresponding to two adjacent iterations. When the change is greater than the convergence threshold, the regularization coefficient and the iteration step size are adjusted and iteration continues; when the change is less than or equal to the convergence threshold, iteration stops. The updated reference speech vectors and reference text vectors are quantized, and the quantized reference speech vectors and reference text vectors are used to form the corresponding reference data.

6. The in-vehicle voice interaction data processing method according to claim 4, characterized in that, After writing the formed sets of reference data into the vector storage structure to obtain the vehicle-mounted multimodal vector database, the method further includes: Construct a tree index for the reference speech vector and the reference text vector; Obtain the retrieval frequency of each of the reference speech vectors and each of the reference text vectors, and construct a hash index for the reference speech vectors and reference text vectors whose retrieval frequency reaches a preset frequency condition; Based on the semantic association between the reference speech vector and the reference text vector, and the association between the two and the vehicle function operation information, a semantic association index is constructed between the reference speech vector, the reference text vector, and the vehicle function operation information; When the reference speech vector, the reference text vector, or the vehicle function operation information is inserted, deleted, or updated, the affected indexes in the tree index, the hash index, and the semantic association index are incrementally updated.

7. The in-vehicle voice interaction data processing method according to claim 4, characterized in that, The vector storage structure includes local cache, in-vehicle solid-state storage, and cloud storage; The step of writing the formed sets of reference data into the vector storage structure includes: Each set of reference data is respectively identified as a set of stored data; Determine the data sensitivity level of each group of stored data, and write each group of stored data into the corresponding storage partition in the vehicle solid-state storage according to the data sensitivity level. Among them, the stored data with a data sensitivity level that reaches the preset sensitivity level is encrypted and then written into the corresponding encrypted storage partition. Obtain the retrieval frequency of the stored data in each group, and load the stored data whose retrieval frequency reaches the preset frequency condition into the local cache; The stored data that meets the preset backup conditions is identified as the data to be synchronized, and the data to be synchronized is synchronized to the cloud storage; The method further includes: In response to a data access request for the stored data, the requesting module that initiated the data access request performs an access permission check. If the check passes, the requested stored data is provided to the requesting module. If the check fails, the data access request is rejected, and the rejected access event is recorded.

8. The in-vehicle voice interaction data processing method according to claim 7, characterized in that, The step of synchronizing the data to be synchronized to the cloud storage includes: Detect the access point name, channel status, network connection status, network bandwidth, and over-the-air (OTA) upgrade status of the vehicle network; When the network connection status is connected, the target access point name channel is determined from the available access point name channels based on the data type, data sensitivity level, and access point name channel status of the data to be synchronized, and the transmission encryption method is determined according to the data sensitivity level. The synchronization frequency of the data to be synchronized is determined based on the network bandwidth. When the over-the-air download upgrade status indicates that an over-the-air download upgrade task is being performed, the data to be synchronized is divided into data required for basic voice interaction and other data to be synchronized. The transmission priority of the other data to be synchronized is reduced, and network bandwidth resources are reserved for the data required for basic voice interaction. The network address stored in the cloud is resolved through the vehicle intranet domain name resolution service, and the data to be synchronized is transmitted through the target access point name channel according to the synchronization frequency, the transmission encryption method and the corresponding transmission priority. When the network connection is interrupted, the untransmitted data to be synchronized is stored in the vehicle solid-state storage and the transmission breakpoint is recorded; after the vehicle network is restored, the target access point name channel and the synchronization frequency are re-determined, and the data to be synchronized is transmitted again according to the transmission breakpoint.

9. The in-vehicle voice interaction data processing method according to claim 1, characterized in that, The in-vehicle multimodal vector database also stores multiple user voice habit vectors corresponding to the current user, and each user voice habit vector is associated with the corresponding in-vehicle function operation information. The user's voice habit vector is generated or updated through the following steps: Obtain the current user's historical voice commands within a preset historical time period and the historical vehicle function operation information corresponding to each historical voice command; According to the method of feature encoding of the voice command, feature encoding is performed on each of the historical voice commands to obtain a historical voice vector; The historical voice vectors are grouped according to the historical vehicle function operation information, and each group of historical voice vectors is aggregated to obtain an updated user voice habit vector associated with each of the historical vehicle function operation information. For each of the updated user voice habit vectors, if a user voice habit vector associated with the same vehicle function operation information exists in the vehicle multimodal vector database, the updated user voice habit vector replaces the original user voice habit vector; if no user voice habit vector associated with the same vehicle function operation information exists, the updated user voice habit vector is added to the vehicle multimodal vector database and associated with the corresponding vehicle function operation information. The calculation of the first similarity between the speech query vector and each of the reference speech vectors includes: Calculate the basic speech similarity between the speech query vector and each of the reference speech vectors; For each of the reference voice vectors, query whether there exists a user voice habit vector associated with the same in-vehicle function operation information as that reference voice vector; If it exists, calculate the habit similarity between the voice query vector and the queried user voice habit vector, and perform a weighted calculation on the basic voice similarity and the habit similarity according to the preset weight to obtain the first similarity corresponding to the reference voice vector; If it does not exist, the basic speech similarity is used as the first similarity corresponding to the reference speech vector.

10. A vehicle-mounted voice interaction data processing system, characterized in that, An application is made to an in-vehicle terminal equipped with an in-vehicle multimodal vector database, wherein the in-vehicle multimodal vector database stores multiple sets of reference data, each set of reference data including semantically related reference speech vectors and reference text vectors, as well as in-vehicle function operation information associated with the reference speech vectors and the reference text vectors; the system includes: The data acquisition module is used to acquire the voice commands collected by the in-vehicle voice interaction module and the corresponding command text of the voice commands; The feature encoding module is used to encode the features of the voice command and the command text respectively to obtain the voice query vector and the text query vector. The threshold determination module is used to determine a dynamic similarity threshold based on the voice quality parameters of the voice command; The vector matching module is used to calculate the first similarity between the speech query vector and each of the reference speech vectors, form a first candidate vector set by the reference speech vectors whose first similarity is greater than or equal to the dynamic similarity threshold, and calculate the second similarity between the text query vector and each of the reference text vectors. The vector matching module is further configured to, if the first candidate vector set is not empty, determine the reference speech vector with the highest first similarity as the target matching vector; otherwise, determine the reference text vector with the highest second similarity as the target matching vector. The instruction generation module is used to generate vehicle control instructions based on the vehicle function operation information associated with the target matching vector.

Images