An in-vehicle task prediction method, system, device and storage medium

Abstract

The application discloses a kind of vehicle-mounted task prediction method, system, equipment and storage medium, the method is converted to the text semantic representation of the multimodal sensor data of vehicle terminal and vehicle-mounted task, respectively constructs mode embedding vector and task embedding vector, calculates the semantic correlation index of each mode sensor data to vehicle-mounted task;According to the mode text and task text of any time, calculate the time value index of each mode to vehicle-mounted task;According to semantic correlation index and time value index, calculate the comprehensive income of each mode to vehicle-mounted task;Under the constraint of resource budget, according to comprehensive income, select optimal mode combination;According to the sensor data of optimal mode combination, execute vehicle-mounted task prediction.The application carries out semantic compression in vehicle terminal, carries out mode screening and vehicle-mounted task prediction in edge server, can reduce communication overhead and edge server computing overhead, improve task prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle technology, and in particular to in-vehicle task prediction methods, systems, devices and storage media. Background Technology

[0002] Intelligent connected vehicles are a significant product of the deep integration of automotive engineering, communication technology, artificial intelligence, and transportation systems, possessing multiple functions such as environmental perception, path planning, decision support, autonomous driving, and remote operation and maintenance. With the continuous development of technologies such as 5G, V2X, and edge computing, intelligent connected vehicles are gradually evolving from single-vehicle intelligence to vehicle-road-cloud collaboration, enabling the integration of information from roads, vehicles, pedestrians, and infrastructure on a larger scale, thereby improving traffic efficiency, driving safety, and the level of service intelligence.

[0003] Multimodal sensor systems are a crucial foundation in the field of intelligent sensing. They acquire multidimensional information about the vehicle's surrounding environment and its own status through the combined deployment of various heterogeneous sensors, such as cameras, LiDAR, millimeter-wave radar, ultrasonic sensors, inertial measurement units, and positioning devices. Different types of sensors have their own advantages in terms of sensing distance, resolution, anti-interference capabilities, and applicable scenarios. By leveraging multimodal information fusion technology, the limitations of single sensors in complex weather, occluded environments, or changing lighting conditions can be overcome, improving the accuracy and robustness of target detection, scene understanding, behavior prediction, and decision control.

[0004] However, due to significant differences in sampling frequency, data dimensionality, representation, and noise characteristics among different modal data, multimodal sensor systems face considerable challenges in data acquisition, time synchronization, spatial registration, feature extraction, and fusion decision-making. On the one hand, information redundancy exists between some modalities, while key semantic information is often provided only by specific modalities. Improperly designed fusion strategies can easily lead to wasted computational resources or insufficient utilization of critical information. On the other hand, multimodal data processing typically involves high storage and computational overhead. In scenarios with limited onboard resources and stringent real-time requirements, systems are prone to problems such as complex processing chains, increased inference latency, and decreased fusion efficiency.

[0005] Furthermore, intelligent connected vehicles need to perform various onboard tasks during operation, including environmental perception, target detection, trajectory prediction, path planning, driving control, fault diagnosis, data uploading, and human-machine interaction. Vehicles typically operate in complex environments characterized by high-speed movement, dynamic network topology changes, and limited computing resources. Existing technologies largely rely on local processing of core onboard tasks. During operation, vehicles need to continuously process large-scale data from multiple sources while simultaneously performing various computational tasks such as perception, recognition, prediction, and control, placing high demands on system real-time performance, stability, and reliability. Under high concurrency, multi-tasking, and complex scenarios, the limited computing power of onboard terminals often struggles to meet real-time processing requirements. If all tasks are executed locally, it leads to increased processing latency and rapidly rising energy consumption. These problems become more pronounced when the task scale increases or the external environment changes drastically, ultimately affecting the overall efficiency of the vehicle system's collaborative operation. Uploading all multimodal perception data to edge servers for processing introduces data privacy and security risks. Simultaneously, the large amount of raw data transmission consumes excessive communication bandwidth, significantly increasing the communication cost between the vehicle and the edge server. Summary of the Invention

[0006] Purpose of the Invention: The purpose of this invention is to provide an in-vehicle task prediction method, system, device, and storage medium that, while ensuring the data privacy of intelligent connected vehicles, reduces communication overhead and vehicle terminal computing overhead, fully utilizes the high computing power of edge servers, improves the accuracy of task prediction, and supports the implementation of multimodal task prediction in intelligent connected vehicle environments.

[0007] Technical solution: The in-vehicle task prediction method of the present invention includes the following steps:

[0008] Acquire multimodal sensor data from the vehicle terminal, and perform text semantic representation conversion on the multimodal sensor data and the vehicle task to obtain modal text and task text;

[0009] Based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively, and the semantic relevance index of each modal sensor data to the vehicle task is calculated.

[0010] Based on the modal text and task text at any given time, calculate the time value index of each modality for the onboard task.

[0011] Based on the semantic relevance index and the time value index, calculate the comprehensive benefit of each modality for the vehicle-mounted task;

[0012] Under resource budget constraints, the optimal mode combination is selected based on the comprehensive benefits.

[0013] Vehicle task prediction is performed based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

[0014] Furthermore, based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively, and the semantic relevance index of each modality to the in-vehicle task is calculated, including:

[0015] The modal text and task text are converted into a set of lexical units. Each lexical unit is converted into a vector using an embedding model, and modal embedding vectors and task embedding vectors are constructed respectively.

[0016] Furthermore, based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively, and the semantic relevance index of each modality to the in-vehicle task is calculated, including:

[0017] The semantic relevance index is calculated based on the bidirectional semantic coverage index and semantic discrimination index between each modality and the vehicle-mounted task;

[0018] The bidirectional semantic coverage metric is calculated based on the cosine similarity between the modality embedding vector and the task embedding vector.

[0019] The semantic discrimination metric is calculated based on the InfoNCE loss of sample pairs constructed from modality embedding vectors and task embedding vectors.

[0020] Furthermore, the semantic relevance index is the sigmoid mapping value of the weighted sum of the bidirectional semantic coverage index and the semantic discrimination index.

[0021] Furthermore, the calculation method for the bidirectional semantic coverage index includes:

[0022] Take the top few word vectors with the largest values ​​in the task embedding vector, calculate their first cosine similarity with the modality embedding vector, and calculate the mean of the first cosine similarity to obtain the first average similarity from the modality to the task direction.

[0023] Take the top few word vectors with the largest values ​​in the modality embedding vector, calculate their second cosine similarity with the task embedding vector, and calculate the mean of the second cosine similarity to obtain the second average similarity from the task to the modality direction.

[0024] The bidirectional semantic coverage index is obtained by calculating the mean of the first average similarity and the second average similarity.

[0025] Furthermore, the method for calculating the semantic discrimination index includes:

[0026] A semantic similarity matrix is ​​constructed based on the cosine similarity of each word vector between the modality embedding vector and the task embedding vector.

[0027] Positive sample pairs are constructed by taking the modal word vectors and task word vectors corresponding to the top few elements with the largest values ​​in the semantic similarity matrix, and negative sample pairs are constructed by randomly sampling from the remaining elements.

[0028] Calculate the InfoNCE loss value for positive sample pairs;

[0029] The semantic discrimination index is calculated based on the InfoNCE loss values ​​of all positive sample pairs.

[0030] Furthermore, the comprehensive benefit is the Sigmoid mapping value of the weighted sum of the semantic relevance index and the time value index.

[0031] Furthermore, based on the modal text and task text at any given time, the time value index of each modality for the onboard task is calculated, including:

[0032] The modal text and task text are converted into a lexical set, and the text hit strength of each modality for the vehicle task at any time is calculated based on the lexical set.

[0033] Calculate the original value score at the current moment based on the text hit strength;

[0034] The time value index of each modality for the onboard task at the current moment is calculated based on the original value score and the median absolute deviation of the original value score.

[0035] Furthermore, the method for calculating the text hit strength includes:

[0036] The text hit strength is obtained by calculating the ratio of the number of elements in the intersection set to the number of elements in the second word set.

[0037] The modal text is transformed into a first lexical set, and the task text is transformed into a second lexical set. The intersection of the first lexical set and the second lexical set is the intersection set.

[0038] Furthermore, the method for calculating the original value score includes:

[0039] The original value score is calculated based on the text hit strength and the time interval between the current time and any time within the observation window.

[0040] Furthermore, the modal text and task text are normalized and then converted into a lexical set;

[0041] The normalization process includes one or more of the following: stop word filtering, domain keyword retention, redundant phrase deletion, and threshold filtering.

[0042] Furthermore, under resource budget constraints, selecting the optimal mode combination based on the comprehensive benefits includes:

[0043] An objective function is established based on the comprehensive benefits, and the resource budget constraint is that the total cost of acquiring, processing and transmitting sensor data for all modes does not exceed the total resource budget. The optimal mode combination is obtained by solving the objective function.

[0044] The objective function is to maximize the overall benefit, which includes the coverage benefit of all modalities for all in-vehicle tasks, the discrimination benefit of all modalities for any in-vehicle task, and the redundancy between any two modalities.

[0045] Furthermore, the method for calculating the coverage gain of all modes for all onboard tasks includes:

[0046] Calculate the maximum overall benefit of each mode for any vehicle-mounted task;

[0047] The coverage revenue is obtained by summing the maximum combined revenue of all vehicle-mounted tasks.

[0048] Furthermore, the method for calculating the discrimination benefit of all modes for any in-vehicle task includes:

[0049] The cumulative benefit is obtained by summing the combined benefits of all modes for any vehicle-mounted task;

[0050] Construct the posterior probabilities of all modalities for the vehicle-mounted task based on the cumulative revenue;

[0051] The discrimination benefit is calculated based on the information entropy of the posterior probability.

[0052] Furthermore, the method for calculating the redundancy between any two modes includes:

[0053] Calculate the cosine similarity of the combined benefits of any two modalities for all vehicle-mounted tasks;

[0054] The redundancy is calculated based on the cosine similarity and the number of modes.

[0055] Furthermore, the objective function also considers the specified cumulative benefit of all modes for a given onboard task, and the method for calculating the specified cumulative benefit includes:

[0056] For a specific vehicle-mounted task of particular interest, the sum of the total benefits of all modalities for the specified vehicle-mounted task is calculated to obtain the specified cumulative benefit.

[0057] Furthermore, the objective function is solved using a greedy strategy that maximizes marginal revenue per unit cost.

[0058] The vehicle-mounted task prediction system of the present invention includes:

[0059] The semantic conversion unit is used to acquire multimodal sensor data from the vehicle terminal, and to convert the multimodal sensor data and the vehicle task into text semantic representation to obtain modal text and task text.

[0060] The semantic relevance calculation unit is used to construct modal embedding vectors and task embedding vectors based on the modal text and task text, respectively, and calculate the semantic relevance index of each modal sensor data to the vehicle task.

[0061] The time value calculation unit is used to calculate the time value index of each modality for the vehicle-mounted task based on the modal text and task text at any given time.

[0062] The comprehensive benefit calculation unit is used to calculate the comprehensive benefit of each modality for the vehicle-mounted task based on the semantic relevance index and the time value index.

[0063] The optimal mode combination selection unit is used to select the optimal mode combination based on the comprehensive benefits under resource budget constraints.

[0064] The vehicle-mounted task prediction unit is used to perform vehicle-mounted task prediction based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

[0065] Furthermore, based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively, and the semantic relevance index of each modality to the in-vehicle task is calculated, including:

[0066] The modal text and task text are converted into a set of lexical units. Each lexical unit is converted into a vector using an embedding model, and modal embedding vectors and task embedding vectors are constructed respectively.

[0067] Furthermore, based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively, and the semantic relevance index of each modality to the in-vehicle task is calculated, including:

[0068] The semantic relevance index is calculated based on the bidirectional semantic coverage index and semantic discrimination index between each modality and the vehicle-mounted task;

[0069] The bidirectional semantic coverage metric is calculated based on the cosine similarity between the modality embedding vector and the task embedding vector.

[0070] The semantic discrimination metric is calculated based on the InfoNCE loss of sample pairs constructed from modality embedding vectors and task embedding vectors.

[0071] Furthermore, the semantic relevance index is the sigmoid mapping value of the weighted sum of the bidirectional semantic coverage index and the semantic discrimination index.

[0072] Furthermore, the calculation method for the bidirectional semantic coverage index includes:

[0073] Take the top few word vectors with the largest values ​​in the task embedding vector, calculate their first cosine similarity with the modality embedding vector, and calculate the mean of the first cosine similarity to obtain the first average similarity from the modality to the task direction.

[0074] Take the top few word vectors with the largest values ​​in the modality embedding vector, calculate their second cosine similarity with the task embedding vector, and calculate the mean of the second cosine similarity to obtain the second average similarity from the task to the modality direction.

[0075] The bidirectional semantic coverage index is obtained by calculating the mean of the first average similarity and the second average similarity.

[0076] Furthermore, the method for calculating the semantic discrimination index includes:

[0077] A semantic similarity matrix is ​​constructed based on the cosine similarity of each word vector between the modality embedding vector and the task embedding vector.

[0078] Positive sample pairs are constructed by taking the modal word vectors and task word vectors corresponding to the top few elements with the largest values ​​in the semantic similarity matrix, and negative sample pairs are constructed by randomly sampling from the remaining elements.

[0079] Calculate the InfoNCE loss value for positive sample pairs;

[0080] The semantic discrimination index is calculated based on the InfoNCE loss values ​​of all positive sample pairs.

[0081] Furthermore, the comprehensive benefit is the Sigmoid mapping value of the weighted sum of the semantic relevance index and the time value index.

[0082] Furthermore, based on the modal text and task text at any given time, the time value index of each modality for the onboard task is calculated, including:

[0083] The modal text and task text are converted into a lexical set, and the text hit strength of each modality for the vehicle task at any time is calculated based on the lexical set.

[0084] Calculate the original value score at the current moment based on the text hit strength;

[0085] The time value index of each modality for the onboard task at the current moment is calculated based on the original value score and the median absolute deviation of the original value score.

[0086] Furthermore, the method for calculating the text hit strength includes:

[0087] The text hit strength is obtained by calculating the ratio of the number of elements in the intersection set to the number of elements in the second word set.

[0088] The modal text is transformed into a first lexical set, and the task text is transformed into a second lexical set. The intersection of the first lexical set and the second lexical set is the intersection set.

[0089] Furthermore, the method for calculating the original value score includes:

[0090] The original value score is calculated based on the text hit strength and the time interval between the current time and any time within the observation window.

[0091] Furthermore, the modal text and task text are normalized and then converted into a lexical set;

[0092] The normalization process includes one or more of the following: stop word filtering, domain keyword retention, redundant phrase deletion, and threshold filtering.

[0093] Furthermore, under resource budget constraints, selecting the optimal mode combination based on the comprehensive benefits includes:

[0094] An objective function is established based on the comprehensive benefits, and the resource budget constraint is that the total cost of acquiring, processing and transmitting sensor data for all modes does not exceed the total resource budget. The optimal mode combination is obtained by solving the objective function.

[0095] The objective function is to maximize the overall benefit, which includes the coverage benefit of all modalities for all in-vehicle tasks, the discrimination benefit of all modalities for any in-vehicle task, and the redundancy between any two modalities.

[0096] Furthermore, the method for calculating the coverage gain of all modes for all onboard tasks includes:

[0097] Calculate the maximum overall benefit of each mode for any vehicle-mounted task;

[0098] The coverage revenue is obtained by summing the maximum combined revenue of all vehicle-mounted tasks.

[0099] Furthermore, the method for calculating the discrimination benefit of all modes for any in-vehicle task includes:

[0100] The cumulative benefit is obtained by summing the combined benefits of all modes for any vehicle-mounted task;

[0101] Construct the posterior probabilities of all modalities for the vehicle-mounted task based on the cumulative revenue;

[0102] The discrimination benefit is calculated based on the information entropy of the posterior probability.

[0103] Furthermore, the method for calculating the redundancy between any two modes includes:

[0104] Calculate the cosine similarity of the combined benefits of any two modalities for all vehicle-mounted tasks;

[0105] The redundancy is calculated based on the cosine similarity and the number of modes.

[0106] Furthermore, the objective function also considers the specified cumulative benefit of all modes for a given onboard task, and the method for calculating the specified cumulative benefit includes:

[0107] For a specific vehicle-mounted task of particular interest, the sum of the total benefits of all modalities for the specified vehicle-mounted task is calculated to obtain the specified cumulative benefit.

[0108] Furthermore, the objective function is solved using a greedy strategy that maximizes marginal revenue per unit cost.

[0109] Another vehicle-mounted task prediction method described in the invention includes the following steps:

[0110] The vehicle terminal acquires multimodal sensor data, performs text semantic representation conversion on the multimodal sensor data to obtain modal text, and uploads it to the edge server;

[0111] The edge server performs text semantic representation transformation on the vehicle-mounted task to obtain task text; constructs modal embedding vectors and task embedding vectors based on the modal text and task text respectively, and calculates the semantic relevance index of each modal sensor data to the vehicle-mounted task; calculates the time value index of each modality to the vehicle-mounted task based on the modal text and task text at any time; calculates the comprehensive benefit of each modality to the vehicle-mounted task based on the semantic relevance index and the time value index; selects the optimal modal combination based on the comprehensive benefit under resource budget constraints; performs vehicle-mounted task prediction based on the task embedding vector and the modal embedding vector of the sensor data of the optimal modal combination to obtain the vehicle-mounted task prediction result, and sends the vehicle-mounted task prediction result to the vehicle terminal.

[0112] Another vehicle-mounted task prediction system according to the present invention includes a vehicle terminal and an edge server. The vehicle terminal includes a data acquisition module and a sensing semantic conversion unit. The edge server includes a task semantic conversion unit, a semantic relevance calculation unit, a time value calculation unit, a comprehensive benefit calculation unit, an optimal modality combination selection unit, and a vehicle-mounted task prediction unit.

[0113] The data acquisition module is used to acquire multimodal sensor data from the vehicle terminal.

[0114] The sensing semantic conversion unit is used to convert the multimodal sensor data into text semantic representation to obtain modal text;

[0115] The in-vehicle semantic conversion unit is used to convert the in-vehicle task into a text semantic representation to obtain the task text.

[0116] The semantic relevance calculation unit is used to construct modal embedding vectors and task embedding vectors based on the modal text and task text, respectively, and calculate the semantic relevance index of each modal sensor data to the vehicle task.

[0117] The time value calculation unit is used to calculate the time value index of each modality for the vehicle-mounted task based on the modal text and task text at any given time.

[0118] The comprehensive benefit calculation unit is used to calculate the comprehensive benefit of each modality for the vehicle-mounted task based on the semantic relevance index and the time value index.

[0119] The optimal mode combination selection unit is used to select the optimal mode combination based on the comprehensive benefits under resource budget constraints.

[0120] The vehicle-mounted task prediction unit is used to perform vehicle-mounted task prediction based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

[0121] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the vehicle task prediction method.

[0122] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the vehicle-mounted task prediction method.

[0123] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the vehicle task prediction method.

[0124] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

[0125] 1) Reducing the privacy and security risks associated with the external transmission of raw multimodal data. Existing technologies typically upload raw image, video, and audio modal data directly to edge servers for processing. This data contains a large amount of sensitive information, posing a significant risk of privacy leakage. This invention first converts multimodal sensor data into unified textual semantics at the vehicle terminal and encapsulates it into modal semantic triples. These are then further converted into modal lexical triples and modal embedding triples. Only task-related semantic representations are uploaded, rather than the raw sensor data directly. Because the uploaded object changes from raw data to an abstracted and compressed semantic representation, the risk of direct leakage of sensitive information is significantly reduced.

[0126] 2) Reduce communication bandwidth usage and transmission overhead. Raw image, video, and audio data are typically characterized by high dimensionality, high sampling rates, and large data volumes. Modal semantic triples, modal lexical triples, and modal embedding triples are all compact representations after abstraction. By converting raw modal data into unified semantics and performing lexicalization and embedding, highly redundant, multi-format heterogeneous data can be compressed into a unified and smaller representation, thus significantly reducing the scale of uploaded data and communication costs.

[0127] 3) Improving the relevance of task prediction: In existing technologies, a common approach is to assume that all modalities are equally important to all tasks, thus employing full modality fusion or fixed modality combinations. However, in reality, different tasks have significantly different degrees of dependence on modal information. This invention constructs task semantic triples, task lexical triples, and task embedding triples, and calculates the relevance of each modality to each task, thereby performing modality selection based on task-driven principles. Therefore, it can improve the relevance and effectiveness of task prediction.

[0128] 4) Improving resource utilization efficiency: Simply selecting modes based on correlation still suffers from problems such as excessively high costs for highly correlated modes or information duplication among multiple highly correlated modes. This invention considers mode benefits, budget constraints, and redundancy penalties simultaneously during the mode combination selection process, thereby selecting the mode combination with better overall benefits under resource constraints. Therefore, it can better balance prediction performance and system resource consumption, improving overall resource utilization efficiency.

[0129] 5) Improved system adaptability to task and environmental changes: This invention does not perform modality selection all at once. Instead, after obtaining the optimal modality combination for the current task at the edge server, the selection result is fed back to the vehicle terminal to guide the modality acquisition, processing, and uploading strategy for the next cycle. Simultaneously, the system updates the revenue function parameters or modality priorities based on task prediction results, modality contribution, and time-varying trends. This gives the system stronger adaptive and continuous optimization capabilities. Attached Figure Description

[0130] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0131] Figure 2 This is a flowchart of the generative artificial intelligence processing in Embodiment 1 of the present invention.

[0132] Figure 3 This is a system architecture diagram of Embodiment 2 of the present invention.

[0133] Figure 4 This is a flowchart of the method in Embodiment 3 of the present invention.

[0134] Figure 5 This is a system architecture diagram of Embodiment 4 of the present invention. Detailed Implementation

[0135] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0136] Example 1

[0137] like Figure 1 As shown, the vehicle-mounted task prediction method includes the following steps:

[0138] Step 1: Acquire multimodal sensor data from the vehicle terminal, and perform text semantic representation conversion on the multimodal sensor data and the vehicle task to obtain modal text and task text;

[0139] Step 2: Construct modal embedding vectors and task embedding vectors based on the modal text and task text, respectively;

[0140] Step 3: Calculate the semantic relevance index of each modal sensor data to the vehicle-mounted task;

[0141] Based on the modal text and task text at any given time, calculate the time value index of each modality for the onboard task.

[0142] Based on the semantic relevance index and the time value index, calculate the comprehensive benefit of each modality for the vehicle-mounted task;

[0143] Step 4: Under resource budget constraints, select the optimal mode combination based on the comprehensive benefits;

[0144] Step 5: Perform vehicle task prediction based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

[0145] Further, in step 1, the multimodal sensor data includes image, video, audio, vehicle acceleration, gyroscope, angular acceleration, and environmental state sensing data. The vehicle acceleration, gyroscope, angular acceleration, and environmental state sensing data are one-dimensional data representing one or more values ​​at a given moment. Since the data formats of different modalities are different, direct cross-modal use is difficult. Therefore, semantic extraction is performed on each modal data separately. A large multimodal model (such as the visual language model Qwen2.5-VL-7B-Instruct, which is used in this embodiment) is used as a semantic encoder, or rules are used as a semantic encoder to convert the multimodal sensing data into a unified text semantic representation. This unified text semantic representation is encapsulated into a modal semantic triple, which includes a timestamp, sensor category, and semantic text information.

[0146] Through the above processing, heterogeneous multimodal raw data can be transformed into comparable representations under a unified semantic modality, laying the foundation for subsequent modality-task matching and modality combination selection.

[0147] Furthermore, in step 2, to reduce the communication burden, the modal semantic triples are further filtered. (Refer to...) Figure 2 The semantic text is normalized based on a generative artificial intelligence model and / or preset rules to obtain modal lexical triples. Then, an embedding model (e.g., Qwen3-Embedding-4B, Qwen3-Embedding-8B, Qwen3-Embedding-4B in this embodiment) is used to convert the lexical units into vector form to obtain modal embedding triples. The preset rules may include one or more of the following: stop word filtering rules, domain keyword retention rules, redundant phrase deletion rules, and threshold filtering rules.

[0148] Preferably, the modal lexical triples and modal embedding triples still retain the timestamp and sensor category, and represent the semantic text content as a set of lexical units; the modal embedding triples further map each lexical unit in the lexical set into an embedding vector.

[0149] Furthermore, in step 1, for the vehicle-mounted task set... , Represents the set of task numbers. Indicates the total number of tasks; any task number is denoted as ,in Each task includes a task number and a task description. These tasks encompass various vehicle-mounted tasks such as environmental perception, target detection, trajectory prediction, path planning, driving control, fault diagnosis, data uploading, and human-machine interaction. Examples include driver fatigue detection, forward risk event prediction, interaction intent recognition, and driving behavior prediction. Referring to the processing methods for various modal sensor data, task semantic triples are constructed for each vehicle-mounted task.

[0150] Furthermore, in step 2, referring to the processing method for each modal sensing data, task-related triples and task embedding triples are constructed for each vehicle-mounted task. The task-related triples do not contain timestamps; instead, task numbers are used to distinguish them from the modality-related triples.

[0151] Examples of each triplet are given below:

[0152] Example of a task semantic triple:

[0153] {"Serial number":1,"Content":"There is a vehicle ahead while movingforward.","value":"Slow down and proceed"}, {"Serial number":2,"Content":"Left-turn sign.","value":"Left turn allowed"};

[0154] Example of a task term triple:

[0155] {"Serial number": 1, "Token": ["ahead", "forward", "moving", "vehicle", "while"], "value": "Slow down and proceed"}{"Serial number": 2, "Token": ["leftturn", "sign"], "value": "Left turn allowed"};

[0156] Example of task embedding triples:

[0157] {"token": "closes","embedding": [[-1.3470649719238281e-05,0.00494384765625,……, -1.3470649719238281e-05],…]}; Each element of the token has a corresponding D-dimensional (D elements) embedding vector.

[0158] Example of modal semantic triples:

[0159] {"ts_ms": 0, "sensor": "image", "value": "The image contains the following objects:\n\n- person: A person is visible in the kitchen area.\n-chair: There are chairs around the dining table.\n- dining table: A table with chairs is present in the dining area.\n- tv: A television set is on the stand.\n- vase: There are vases with flowers on the dining table and on the floor.\n- refrigerator: A refrigerator is located in the kitchen.\n- clock: A clock is mounted on the wall.\n- cabinet: Cabinets are visible in the kitchen area.\n- fireplace: A fireplace is present in the living room.\n\nNo other objects from the provided list are present in the image.", "ts_ms": 60, "sensor": "image", "value": "The image contains a bear. According to the provided `COCO80_CLASS_NAMES`, the category for a bear is:\n\n- bear"}

[0160] Modal word element triple example:

[0161] {"ts_ms": 0, "sensor": "image", "semantics": ["area", "around", "cabinet", "cabinets", "chair", "chairs", "clock", "contains", "dining", "fireplace", "floor", "flowers", "following", "image", "kitchen", "list", "living", "located", "mounted", "objects", "other", "person", "present", "provided", "refrigerator", "room", "set", "stand", "table", "television", "there", "tv", "vase", "vases", "visible", "wall"]};

[0162] {"ts_ms": 60, "sensor": "image", "semantics": ["according", "bear", "category", "class", "coco80", "contains", "image", "names", "provided"]};

[0163] Example of modal embedding triples:

[0164] {"token": "closes","embedding": [[-1.3470649719238281e-05,0.00494384765625,…, -1.3470649719238281e-05],…]}.

[0165] Furthermore, for the sake of consistency in this embodiment, the following definitions are given:

[0166] The candidate mode set is , Represents the set of candidate mode numbers. This represents the total number of candidate modes, and the number of any mode is denoted as . , For any mode Its lexical set is denoted as , Representing modes The corresponding set of lexical units, Representing modes The Each word element, For word index, Representing modes The number of lexical units.

[0167] For any task Its lexical set is denoted as , Indicates task The corresponding set of lexical units, Indicates task The Each word element, For word index, Indicates task The number of lexical units. For modality lexical set Its normalized embedded vector set is denoted as , Representing modes The set of embedded vectors, Indicates word elements The corresponding embedding vector, Indicate 3D real vector space, Represents the embedding dimension, and , Let represent the L2 norm. For the task... lexical set Its normalized set of embedded vectors is denoted as , Indicates task The set of embedded vectors, Indicates word elements The corresponding embedding vector, and .

[0168] Furthermore, in step 3, the modality embedding representation is matched with the task embedding representation to calculate the comprehensive benefit of each modality for each task, specifically including the following process.

[0169] (3.1) Construct a semantic similarity matrix between modalities and tasks

[0170] For any mode and any task Define mode With the task The semantic similarity matrix between them is:

[0171] ,

[0172] ;

[0173] in, Representing modes With the task The semantic similarity matrix, This represents the vector transpose operation. Represents the first in the matrix Line number Column elements, simultaneously representing modalities The lexical embedding vectors and tasks The The cosine similarity between the word embedding vectors. Since all embedding vectors are normalized to L2, we have .

[0174] (3.2) Calculate the bidirectional semantic coverage index

[0175] Define operator For: Take the first element with the largest value from the input vector. The set of indices corresponding to each element, where... These are preset positive integer hyperparameters, which can be pre-set based on the number of modal lemmas, the number of task lemmas, or the performance on the validation set. The average similarity from modality to task is defined as:

[0176] ;

[0177] in, Indicates from modality To the mission Average similarity in direction Representation matrix The Okay. Define the average similarity from task to modality as:

[0178] ;

[0179] in, Indicates from the task To mode Average similarity in direction Representation matrix The List.

[0180] Furthermore, the bidirectional semantic coverage metric is defined as follows:

[0181] ;

[0182] in, Representing modes With the task Bidirectional semantic coverage metrics between them. The larger the value, the more modal it represents. With the task The higher the degree of bidirectional semantic coverage between them.

[0183] (3.3) Calculate semantic discrimination index

[0184] Define operator For: Take the first element with the largest value from the input vector. The set of indices corresponding to each element, where... This is a preset positive integer hyperparameter used to control the number of highly similar word pairs retained when constructing positive sample pairs. .

[0185] Based on mode With the task Similarity of all word pairs between Construct a set of positive sample pairs:

[0186] ;

[0187] in, Representing modes With the task The set of positive sample pairs indices, where each element is an index pair. For any positive sample pair From the task Except index Random sampling from the remaining lexical indexes Each index constitutes a negative sample index set. Indicates positive sample pairs The corresponding set of negative sample indices, This represents the number of negative samples corresponding to each positive sample, and The preset positive integer hyperparameter is used to control the size of the corresponding negative samples for each positive sample pair, so as to enhance the semantic discrimination ability. , All settings can be configured based on the size of the training samples, the characteristics of word distribution, or the performance of the validation set.

[0188] Define the corresponding InfoNCE loss as:

[0189] ;

[0190] in, Indicates positive sample pairs InfoNCE loss, This represents the natural exponential function. Represents the natural logarithm function. This indicates temperature hyperparameters. This represents the negative sample index.

[0191] Furthermore, the semantic discrimination index is defined as follows:

[0192] ;

[0193] in, Representing modes For the task semantic discrimination index, Represents a set The number of elements. Because Therefore, there is .

[0194] (3.4) Constructing semantic indicators

[0195] Define the Sigmoid function as follows:

[0196] ;

[0197] in, This represents the Sigmoid mapping function. Represent the independent variable. Define the modality. For the task The semantic metrics are:

[0198] ;

[0199] in, Representing modes For the task semantic metrics, Indicates task The corresponding semantic fusion weights can be initially obtained through offline training using historical task samples, and can be updated online or periodically re-evaluated during system operation based on prediction results, modal contribution, and environmental changes. The larger the value, the more modal it represents. With the task The stronger the semantic relevance.

[0200] (3.5) Calculate the value index based on time utility

[0201] Let the current decision time be... ,in This indicates the moment when the modality selection decision is being made. Let the modality be... The set of time indices within the current observation window is Representing modes The set of valid time indices within the current observation window, where any time index is denoted as... ,in Define the mode At any moment For the task The text hit strength is

[0202] ;

[0203] in, Indicates time Lower mode With the task Text hit strength between them This represents the intersection operation of sets. For sets, Indicates the number of elements in the set. This indicates that the larger value is taken in the calculation. The denominator takes... This is used to avoid a denominator of zero. The original value fraction after time decay is defined as...

[0204] ;

[0205] in, Representing modes For the task The original value fraction decays over time. This represents the time decay coefficient, which can be preset based on the task's sensitivity to timeliness, or updated based on the predicted gain within the historical window. Indicates time Relative to the current decision-making moment The time interval. To improve the comparability of value scores between different tasks, define the task. The median of the raw value scores for the lower transmodal is:

[0206] ;

[0207] in, Indicates task The median of the original value scores. This represents median operations. Define the task. The median absolute deviation of the original value scores for the lower transmodal is:

[0208] ;

[0209] in, Indicates task The median absolute deviation of the original value scores, for real numbers, This represents the operation of absolute value of real numbers. Further, we define modalities. For the task The value indicators are:

[0210] ;

[0211] in, Representing modes For the task Value indicators This represents the numerical stability constant, used to avoid the denominator being zero. The larger the value, the more modal it represents. At the current moment, regarding the task The higher the time utility.

[0212] (3.6) Calculate the overall benefit of a single mode for a single task.

[0213] Define mode For the task The overall return function is:

[0214] ;

[0215] in, Representing modes For the task The overall benefits Indicates task The weighting of semantic and value metrics is integrated. Indicates task The corresponding gating strength parameters can be set or updated based on historical sample statistics, validation set performance, or online feedback results. The larger the value, the more modal it represents. For the task The higher the overall contribution, the better. Furthermore, define modal... The task reward vector is , indicating mode The revenue vector across all tasks. Define the overall revenue matrix as follows: This represents the comprehensive reward matrix for all candidate modes across all tasks. This serves as the input for the modal combination optimization selection in step 4.

[0216] Through the processes described in 3.1 to 3.6 above, this step maps the semantic coverage relationship, semantic discrimination ability, and temporal utility between modalities and tasks into a unified comprehensive benefit. This allows the contributions of different modalities to be uniformly quantified and directly compared across different tasks. Because ,and , , All have undergone Mapping, therefore we have

[0217]

[0218] Therefore, the comprehensive benefit matrix All elements in the system are bounded, thus ensuring good numerical stability across different tasks and modes.

[0219] make Then there is .because The derivative satisfies Therefore:

[0220] , ;

[0221] Therefore, it can be seen that when the semantic matching degree of the modality to the task is improved or the time value is increased, the overall benefit is higher. The monotony ensures that "the more relevant and timely the modality, the higher the reward."

[0222] Furthermore, in step 4, the comprehensive profit matrix output in step 3 is obtained. Then, under resource constraints, candidate modes are combined and selected to determine the optimal mode combination. For any given mode... The total cost of its acquisition, processing, and transmission is defined as in, Representing modes The total cost.

[0223] Define the total system budget as , representing the total resource budget allowed in each round of modality selection. The set of selected modes is defined as... This represents the currently selected subset of modalities. The empty set... This represents the initial set of modes that have not been selected. Define the mode set. The total cost is , Represents a set The corresponding total cost.

[0224] To simultaneously consider task coverage, target task specification, task discrimination, and redundancy suppression, a modality set is defined. The objective function is:

[0225] ;

[0226] in, Represents modal combination The overall benefits, Indicates task coverage items, Indicates the specified target task item. Indicates the label information item. Indicates redundant items. and Let represent the weight coefficients of the objective function, and satisfy . , This represents the redundancy penalty coefficient. The following sections will describe each objective individually.

[0227] (a) Task Coverage: Define the normalized task weight as... . Indicates task The weights in the global coverage target, and satisfying:

[0228] ;

[0229] Define the task coverage items as:

[0230] ;

[0231] in, Represents the modal set Benefits covering all tasks. Indicates the selected mode set In the middle, regarding the task The single-modal overall benefit contributes the most. To avoid The formula is undefined, by convention. .

[0232] (b) Specify target task item: Optionally, when the system has a target task that is currently of priority, let the target task number be . . This indicates the currently specified target task number. The specified target task item is defined as:

[0233] ;

[0234] in, Represents the modal set For the specified target task The specified cumulative earnings. When no specified target task exists, let... In this case, the specified target task item will not participate in the optimization.

[0235] (c) Tag information item: Define modality set For the task The cumulative score is:

[0236] ;

[0237] in, Represents the modal set In the mission The cumulative earnings score.

[0238] Construct the posterior distribution based on the cumulative score:

[0239] ;

[0240] in, In the modality set Task under conditions The posterior probability, This represents the posterior temperature distribution hyperparameter. This represents the task index dummy variable in the summation.

[0241] Define the tag information item as follows:

[0242] ;

[0243] Among them, Represents the modal set The corresponding tag information items, The information entropy represents the posterior distribution. Used for normalization. Because the entropy range satisfies... Therefore, there is .

[0244] (d) Redundancy: To avoid high repetition of the effects of multiple selected modalities on different tasks, define any two modalities and The redundancy between them is ;

[0245] in, Representing modes With mode Cosine similarity in the task reward space and These represent the modal task reward vectors defined in step 3. Since... and All components are positive, therefore we have .

[0246] Define modal set The average redundancy is:

[0247] ;

[0248] in, Represents the modal set Internal average redundancy Represents a set The number of elements, constraints Used to avoid counting the same pair of modes repeatedly. achievable .

[0249] Based on the above (a) to (d), under budget constraints, the optimal mode combination is defined as:

[0250] ;

[0251] in, This indicates that the objective function is satisfied when the budget constraints are met. Maximum optimal mode combination Let represent the set of independent variables that maximize the objective function. Since the above problem is a discrete combinatorial optimization problem, a greedy strategy based on maximizing marginal revenue per unit cost is preferred for approximate solution.

[0252] Initialize to ,in, This represents the set of modes for which no modes have been selected at the initial time. Let the greedy iteration round index be... ,in, In the first In each iteration, from the modes that have not yet been selected and meet the budget constraints, the mode with the largest marginal revenue 'a' per unit cost is selected:

[0253] ;

[0254] in, This represents the set of candidate modes that have not yet been selected. This represents the set union operation. Indicates the first The selected modality is then selected. Subsequently, the modality set is updated. When there are no candidate modes that satisfy the budget constraint, or for all candidate modes that satisfy the budget constraint:

[0255] ;

[0256] Then stop the iteration and denote the current set of modes as the optimal mode combination. .

[0257] Through the above processing, under budget constraints, it is possible to select a combination of modes that is more advantageous to multi-task scenarios or the current specified target task and has low internal redundancy from multiple candidate modes, so as to perform task prediction and policy feedback in step 5.

[0258] Furthermore, step 4 transforms the mode selection problem into a combinatorial optimization problem under budget constraints, ensuring that the selected mode combination has a clear optimization objective and an executable solution path. Define the basic objective function:

[0259] ;

[0260] Among them, task coverage items Each item in Both are functions that "find the maximum value" on the set. For any set satisfying... Two sets, and any ,have:

[0261] ;

[0262] The above inequality shows that the gain from adding the same mode to a smaller set is no less than the gain from adding the same mode to a larger set. Therefore, It satisfies the property of diminishing marginal returns, i.e., it is a monotonic submodular function. Because... Therefore As a non-negative weighted sum, it remains a monotonic submodular function.

[0263] Furthermore, specify the target task item. The form is the summation of the elements of a set, which is a modular function, and therefore also a submodular function. Therefore, the fundamental objective function... Let be a monotonic submodular function. When only budget constraints are considered, a greedy choice of the monotonic submodular function based on marginal revenue per unit cost can yield a feasible solution with a constant factor approximation guarantee.

[0264] Both the label information item and the redundancy item are bounded adjustment items, satisfying the following conditions respectively:

[0265] ;

[0266] Therefore, the difference between the complete objective function and the basic objective function satisfies:

[0267] .

[0268] This formula shows that the impact of label information items and redundant items on the basic submodule structure is a bounded perturbation. Its function is to enhance task discrimination concentration and suppress modal redundancy without violating budget feasibility conditions. Furthermore, since each round of selection only occurs when the following conditions are met... The process is carried out within the candidate modes, therefore the final output... Budget constraints are always met. This step prioritizes modes that yield greater benefits per unit of resource consumption, while ensuring resource feasibility, and enhances the complementarity between selected modes through redundancy penalties, thereby achieving a better balance between prediction performance, resource consumption, and information diversity.

[0269] Furthermore, in step 5, task prediction is performed based on the semantic representation corresponding to the optimal modality combination to obtain task output results, which include category labels, risk levels, interaction intent results, behavior prediction results, or control suggestions.

[0270] Task prediction relies on the embedding representations corresponding to the modal and task semantic representations, and prediction is performed using the semantic similarity matrix calculated from them. Here, the optimal modal benefit combination is used; other unselected modalities are not subsequently included in the semantic similarity matrix calculation. The triplets of the optimal modal combination are combined into a new triplet. The set of words in this triplet is the union of the set of words in each modal triplet, and this set will not contain duplicate words. Then, the merged words are used to generate one-to-one embedding vectors, forming a set of embedding vectors, which are then encapsulated into the merged embedding triplet. For ease of representation, the synthesized triplet is considered as a virtual modality. The generated triples. Modality Embedded triples with each task The embedded triplet calculation yields ,Pick The largest task as a modality The prediction task, i.e., the task corresponding to the optimal mode combination, is defined as follows: the value corresponding to the value key in the semantic triple of this task is used as the predicted value.

[0271] Task predictions can be numeric or other types. For example, a set of tasks. This is a COCO80 classification task. Each category has a corresponding index value, so the task prediction value is the index of the category (e.g., for the category "dog", the prediction value is its corresponding index 76); task set These are the situations encountered during autonomous driving, and each situation has a corresponding behavioral decision. The task prediction value is the behavior (e.g., if there is a pedestrian in front of the vehicle, the prediction value is the corresponding behavior "braking").

[0272] Preferably, based on the optimal modality combination result, the selected modality is prioritized for acquisition, processing, and uploading in the next sampling period, while resource scheduling is implemented for unselected modalities, including reduced sampling frequency, delayed uploading, or paused uploading. During multi-round task execution, the parameters in the benefit function are dynamically updated based on historical prediction results, modality contribution, task type switching, and environmental changes to achieve adaptive optimization of the modality selection strategy. The semantic fusion weights, fusion weights of semantic and value indicators, time decay coefficients, and gating strength parameters corresponding to the task are updated to adjust the comprehensive benefit calculation results and selection priorities of each modality under different tasks. Specifically, when a certain type of task has a high dependence on a specific modality, the semantic fusion weight, fusion weight, or gating strength parameter of that specific modality under the corresponding task is increased; when a modality is unstable, its timeliness decreases, or its contribution to the prediction results weakens under a specific environment, the relevant weight parameters under the corresponding task of that modality are reduced, the time decay effect is increased, or the gating strength is weakened. Through the above feedback and update process, a closed-loop adaptive mechanism for task prediction and resource scheduling is formed.

[0273] Example 2

[0274] Reference Figure 3 The vehicle-mounted task prediction system described in this embodiment includes:

[0275] The semantic conversion unit is used to acquire multimodal sensor data from the vehicle terminal, and to convert the multimodal sensor data and the vehicle task into text semantic representation to obtain modal text and task text.

[0276] The semantic relevance calculation unit is used to construct modal embedding vectors and task embedding vectors based on the modal text and task text, respectively, and calculate the semantic relevance index of each modal sensor data to the vehicle task.

[0277] The time value calculation unit is used to calculate the time value index of each modality for the vehicle-mounted task based on the modal text and task text at any given time.

[0278] The comprehensive benefit calculation unit is used to calculate the comprehensive benefit of each modality for the vehicle-mounted task based on the semantic relevance index and the time value index.

[0279] The optimal mode combination selection unit is used to select the optimal mode combination based on the comprehensive benefits under resource budget constraints.

[0280] The vehicle-mounted task prediction unit is used to perform vehicle-mounted task prediction based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

[0281] Example 3

[0282] Reference Figure 4 The vehicle-mounted task prediction method described in this embodiment uses an intelligent connected vehicle as the vehicle terminal. The vehicle terminal acquires multimodal sensor data and sends it to an edge server for task prediction. The vehicle terminal does not need to upload the original video stream, audio stream, and high-dimensional perception data, but only needs to upload the compressed semantic expression results, thereby reducing bandwidth consumption and minimizing the risk of original sensitive data being transmitted. Specifically, it includes the following steps:

[0283] The vehicle terminal acquires multimodal sensor data, performs text semantic representation conversion on the multimodal sensor data to obtain modal text, and uploads it to the edge server;

[0284] The edge server performs text semantic representation transformation on the vehicle-mounted task to obtain task text; constructs modal embedding vectors and task embedding vectors based on the modal text and task text respectively, and calculates the semantic relevance index of each modal sensor data to the vehicle-mounted task; calculates the time value index of each modality to the vehicle-mounted task based on the modal text and task text at any time; calculates the comprehensive benefit of each modality to the vehicle-mounted task based on the semantic relevance index and the time value index; selects the optimal modal combination based on the comprehensive benefit under resource budget constraints; performs vehicle-mounted task prediction based on the task embedding vector and the modal embedding vector of the sensor data of the optimal modal combination to obtain the vehicle-mounted task prediction result, and sends the vehicle-mounted task prediction result to the vehicle terminal.

[0285] Furthermore, the edge server periodically feeds back the optimal modal combination to the vehicle terminal to guide the modal acquisition, processing, and uploading strategy for the next cycle.

[0286] Example 4

[0287] The vehicle-mounted task prediction system described in this embodiment includes a vehicle terminal and an edge server. The vehicle terminal includes a data acquisition module and a sensing semantic conversion unit. The edge server includes a task semantic conversion unit, a semantic relevance calculation unit, a time value calculation unit, a comprehensive benefit calculation unit, an optimal modality combination selection unit, and a vehicle-mounted task prediction unit.

[0288] The data acquisition unit is used to acquire multimodal sensor data from the vehicle terminal.

[0289] The sensing semantic conversion unit is used to convert the multimodal sensor data into text semantic representation to obtain modal text;

[0290] The in-vehicle semantic conversion unit is used to convert the in-vehicle task into a text semantic representation to obtain the task text.

[0291] The semantic relevance calculation unit is used to construct modal embedding vectors and task embedding vectors based on the modal text and task text, respectively, and calculate the semantic relevance index of each modal sensor data to the vehicle task.

[0292] The time value calculation unit is used to calculate the time value index of each modality for the vehicle-mounted task based on the modal text and task text at any given time.

[0293] The comprehensive benefit calculation unit is used to calculate the comprehensive benefit of each modality for the vehicle-mounted task based on the semantic relevance index and the time value index.

[0294] The optimal mode combination selection unit is used to select the optimal mode combination based on the comprehensive benefits under resource budget constraints.

[0295] The vehicle-mounted task prediction unit is used to perform vehicle-mounted task prediction based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

[0296] Furthermore, referring to Figure 5 The vehicle terminal may also include a modal execution control unit, used to receive the optimal modal combination results from the edge server and adjust the modal acquisition, processing, and uploading strategy for the next cycle accordingly. The system also includes a communication unit that connects the vehicle terminal and the edge server via a wireless network, used to send the modal text from the vehicle terminal to the edge server, and to send the optimal modal combination and onboard task prediction results from the edge server to the vehicle terminal.

[0297] Example 5

[0298] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the vehicle task prediction method.

[0299] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

[0300] Example 6

[0301] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the vehicle-mounted task prediction method.

[0302] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store program code in the form of instructions or data structures and is accessible by a computer.

[0303] Example 7

[0304] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the vehicle task prediction method.

Claims

1. A method for predicting vehicle-mounted tasks, characterized in that, Includes the following steps: Acquire multimodal sensor data from the vehicle terminal, and perform text semantic representation conversion on the multimodal sensor data and the vehicle task to obtain modal text and task text; Based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively, and the semantic relevance index of each modal sensor data to the vehicle task is calculated. Based on the modal text and task text at any given time, calculate the time value index of each modality for the onboard task. Based on the semantic relevance index and the time value index, calculate the comprehensive benefit of each modality for the vehicle-mounted task; Under resource budget constraints, the optimal mode combination is selected based on the comprehensive benefits. Vehicle task prediction is performed based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

2. The vehicle-mounted task prediction method according to claim 1, characterized in that, Based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively. The semantic relevance index of each modality to the in-vehicle task is calculated, including: The modal text and task text are converted into a set of lexical units. Each lexical unit is converted into a vector using an embedding model, and modal embedding vectors and task embedding vectors are constructed respectively.

3. The vehicle-mounted task prediction method according to claim 2, characterized in that, Based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively. The semantic relevance index of each modality to the in-vehicle task is calculated, including: The semantic relevance index is calculated based on the bidirectional semantic coverage index and semantic discrimination index between each modality and the vehicle-mounted task; The bidirectional semantic coverage metric is calculated based on the cosine similarity between the modality embedding vector and the task embedding vector. The semantic discrimination metric is calculated based on the InfoNCE loss of sample pairs constructed from modality embedding vectors and task embedding vectors.

4. The vehicle-mounted task prediction method according to claim 3, characterized in that, The semantic relevance index is the Sigmoid mapping value of the weighted sum of the bidirectional semantic coverage index and the semantic discrimination index.

5. The vehicle-mounted task prediction method according to claim 3, characterized in that, The calculation method for the bidirectional semantic coverage index includes: Take the top few word vectors with the largest values ​​in the task embedding vector, calculate their first cosine similarity with the modality embedding vector, and calculate the mean of the first cosine similarity to obtain the first average similarity from the modality to the task direction. Take the top few word vectors with the largest values ​​in the modality embedding vector, calculate their second cosine similarity with the task embedding vector, and calculate the mean of the second cosine similarity to obtain the second average similarity from the task to the modality direction. The bidirectional semantic coverage index is obtained by calculating the mean of the first average similarity and the second average similarity.

6. The vehicle-mounted task prediction method according to claim 3, characterized in that, The calculation method for the semantic discrimination index includes: A semantic similarity matrix is ​​constructed based on the cosine similarity of each word vector between the modality embedding vector and the task embedding vector. Positive sample pairs are constructed by taking the modal word vectors and task word vectors corresponding to the top few elements with the largest values ​​in the semantic similarity matrix, and negative sample pairs are constructed by randomly sampling from the remaining elements. Calculate the InfoNCE loss value for positive sample pairs; The semantic discrimination index is calculated based on the InfoNCE loss values ​​of all positive sample pairs.

7. The vehicle-mounted task prediction method according to claim 1, characterized in that, The overall benefit is the Sigmoid mapping value of the weighted sum of semantic relevance indicators and time value indicators.

8. The vehicle-mounted task prediction method according to claim 1, characterized in that, Based on the modal text and task text at any given time, the time value index of each modality for the onboard task is calculated, including: The modal text and task text are converted into a lexical set, and the text hit strength of each modality for the vehicle task at any time is calculated based on the lexical set. Calculate the original value score at the current moment based on the text hit strength; The time value index of each modality for the onboard task at the current moment is calculated based on the original value score and the median absolute deviation of the original value score.

9. The vehicle-mounted task prediction method according to claim 8, characterized in that, The method for calculating the text hit strength includes: The text hit strength is obtained by calculating the ratio of the number of elements in the intersection set to the number of elements in the second word set. The modal text is transformed into a first lexical set, and the task text is transformed into a second lexical set. The intersection of the first lexical set and the second lexical set is the intersection set.

10. The vehicle-mounted task prediction method according to claim 8 or 9, characterized in that, The method for calculating the original value score includes: The original value score is calculated based on the text hit strength and the time interval between the current time and any time within the observation window.

11. The vehicle-mounted task prediction method according to claim 3 or 9, characterized in that, The modal text and task text are then normalized and converted into a lexical set; The normalization process includes one or more of the following: stop word filtering, domain keyword retention, redundant phrase deletion, and threshold filtering.

12. The vehicle-mounted task prediction method according to claim 1, characterized in that, Under resource budget constraints, selecting the optimal mode combination based on the comprehensive benefits includes: An objective function is established based on the comprehensive benefits, and the resource budget constraint is that the total cost of acquiring, processing and transmitting sensor data for all modes does not exceed the total resource budget. The optimal mode combination is obtained by solving the objective function. The objective function is to maximize the overall benefit, which includes the coverage benefit of all modalities for all in-vehicle tasks, the discrimination benefit of all modalities for any in-vehicle task, and the redundancy between any two modalities.

13. The vehicle-mounted task prediction method according to claim 12, characterized in that, The calculation method for the coverage benefit of all modalities for all onboard tasks includes: Calculate the maximum overall benefit of each mode for any vehicle-mounted task; The coverage revenue is obtained by summing the maximum combined revenue of all vehicle-mounted tasks.

14. The vehicle-mounted task prediction method according to claim 12, characterized in that, The method for calculating the discrimination benefit of all modes for any vehicle-mounted task includes: The cumulative benefit is obtained by summing the combined benefits of all modes for any vehicle-mounted task; Construct the posterior probabilities of all modalities for the vehicle-mounted task based on the cumulative revenue; The discrimination benefit is calculated based on the information entropy of the posterior probability.

15. The vehicle-mounted task prediction method according to claim 12, characterized in that, The method for calculating the redundancy between any two modes includes: Calculate the cosine similarity of the combined benefits of any two modalities for all vehicle-mounted tasks; The redundancy is calculated based on the cosine similarity and the number of modes.

16. The vehicle-mounted task prediction method according to claim 12, characterized in that, The objective function also considers the specified cumulative benefit of all modes for a given onboard task, and the method for calculating the specified cumulative benefit includes: For a specific vehicle-mounted task of particular interest, the sum of the total benefits of all modalities for the specified vehicle-mounted task is calculated to obtain the specified cumulative benefit.

17. The vehicle-mounted task prediction method according to claim 12, characterized in that, The objective function is solved using a greedy strategy that maximizes marginal revenue per unit cost.

18. A vehicle-mounted task prediction system, characterized in that, include: The semantic conversion unit is used to acquire multimodal sensor data from the vehicle terminal, and to convert the multimodal sensor data and the vehicle task into text semantic representation to obtain modal text and task text. The semantic relevance calculation unit is used to construct modal embedding vectors and task embedding vectors based on the modal text and task text, respectively, and calculate the semantic relevance index of each modal sensor data to the vehicle task. The time value calculation unit is used to calculate the time value index of each modality for the vehicle-mounted task based on the modal text and task text at any given time. The comprehensive benefit calculation unit is used to calculate the comprehensive benefit of each modality for the vehicle-mounted task based on the semantic relevance index and the time value index. The optimal mode combination selection unit is used to select the optimal mode combination based on the comprehensive benefits under resource budget constraints. The vehicle-mounted task prediction unit is used to perform vehicle-mounted task prediction based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

19. The vehicle-mounted task prediction system according to claim 18, characterized in that, Based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively. The semantic relevance index of each modality to the in-vehicle task is calculated, including: The modal text and task text are converted into a set of lexical units. Each lexical unit is converted into a vector using an embedding model, and modal embedding vectors and task embedding vectors are constructed respectively.

20. The vehicle-mounted task prediction system according to claim 19, characterized in that, Based on the modal text and task text, modal embedding vectors and task embedding vectors are constructed respectively. The semantic relevance index of each modality to the in-vehicle task is calculated, including: The semantic relevance index is calculated based on the bidirectional semantic coverage index and semantic discrimination index between each modality and the vehicle-mounted task; The bidirectional semantic coverage metric is calculated based on the cosine similarity between the modality embedding vector and the task embedding vector. The semantic discrimination metric is calculated based on the InfoNCE loss of sample pairs constructed from modality embedding vectors and task embedding vectors.

21. The vehicle-mounted task prediction system according to claim 20, characterized in that, The semantic relevance index is the Sigmoid mapping value of the weighted sum of the bidirectional semantic coverage index and the semantic discrimination index.

22. The vehicle-mounted task prediction system according to claim 20, characterized in that, The calculation method for the bidirectional semantic coverage index includes: Take the top few word vectors with the largest values ​​in the task embedding vector, calculate their first cosine similarity with the modality embedding vector, and calculate the mean of the first cosine similarity to obtain the first average similarity from the modality to the task direction. Take the top few word vectors with the largest values ​​in the modality embedding vector, calculate their second cosine similarity with the task embedding vector, and calculate the mean of the second cosine similarity to obtain the second average similarity from the task to the modality direction. The bidirectional semantic coverage index is obtained by calculating the mean of the first average similarity and the second average similarity.

23. The vehicle-mounted task prediction system according to claim 20, characterized in that, The calculation method for the semantic discrimination index includes: A semantic similarity matrix is ​​constructed based on the cosine similarity of each word vector between the modality embedding vector and the task embedding vector. Positive sample pairs are constructed by taking the modal word vectors and task word vectors corresponding to the top few elements with the largest values ​​in the semantic similarity matrix, and negative sample pairs are constructed by randomly sampling from the remaining elements. Calculate the InfoNCE loss value for positive sample pairs; The semantic discrimination index is calculated based on the InfoNCE loss values ​​of all positive sample pairs.

24. The vehicle-mounted task prediction system according to claim 18, characterized in that, The overall benefit is the Sigmoid mapping value of the weighted sum of semantic relevance indicators and time value indicators.

25. The vehicle-mounted task prediction system according to claim 18, characterized in that, Based on the modal text and task text at any given time, the time value index of each modality for the onboard task is calculated, including: The modal text and task text are converted into a lexical set, and the text hit strength of each modality for the vehicle task at any time is calculated based on the lexical set. Calculate the original value score at the current moment based on the text hit strength; The time value index of each modality for the onboard task at the current moment is calculated based on the original value score and the median absolute deviation of the original value score.

26. The vehicle-mounted task prediction system according to claim 25, characterized in that, The method for calculating the text hit strength includes: The text hit strength is obtained by calculating the ratio of the number of elements in the intersection set to the number of elements in the second word set. The modal text is transformed into a first lexical set, and the task text is transformed into a second lexical set. The intersection of the first lexical set and the second lexical set is the intersection set.

27. The vehicle-mounted task prediction system according to claim 25 or 26, characterized in that, The method for calculating the original value score includes: The original value score is calculated based on the text hit strength and the time interval between the current time and any time within the observation window.

28. The vehicle-mounted task prediction system according to claim 20 or 26, characterized in that, The modal text and task text are then normalized and converted into a lexical set; The normalization process includes one or more of the following: stop word filtering, domain keyword retention, redundant phrase deletion, and threshold filtering.

29. The vehicle-mounted task prediction system according to claim 18, characterized in that, Under resource budget constraints, selecting the optimal mode combination based on the comprehensive benefits includes: An objective function is established based on the comprehensive benefits, and the resource budget constraint is that the total cost of acquiring, processing and transmitting sensor data for all modes does not exceed the total resource budget. The optimal mode combination is obtained by solving the objective function. The objective function is to maximize the overall benefit, which includes the coverage benefit of all modalities for all in-vehicle tasks, the discrimination benefit of all modalities for any in-vehicle task, and the redundancy between any two modalities.

30. The vehicle-mounted task prediction system according to claim 29, characterized in that, The calculation method for the coverage benefit of all modalities for all onboard tasks includes: Calculate the maximum overall benefit of each mode for any vehicle-mounted task; The coverage revenue is obtained by summing the maximum combined revenue of all vehicle-mounted tasks.

31. The vehicle-mounted task prediction system according to claim 29, characterized in that, The method for calculating the discrimination benefit of all modes for any vehicle-mounted task includes: The cumulative benefit is obtained by summing the combined benefits of all modes for any vehicle-mounted task; Construct the posterior probabilities of all modalities for the vehicle-mounted task based on the cumulative revenue; The discrimination benefit is calculated based on the information entropy of the posterior probability.

32. The vehicle-mounted task prediction system according to claim 29, characterized in that, The method for calculating the redundancy between any two modes includes: Calculate the cosine similarity of the combined benefits of any two modalities for all vehicle-mounted tasks; The redundancy is calculated based on the cosine similarity and the number of modes.

33. The vehicle-mounted task prediction system according to claim 29, characterized in that, The objective function also considers the specified cumulative benefit of all modes for a given onboard task, and the method for calculating the specified cumulative benefit includes: For a specific vehicle-mounted task of particular interest, the sum of the total benefits of all modalities for the specified vehicle-mounted task is calculated to obtain the specified cumulative benefit.

34. The vehicle-mounted task prediction system according to claim 29, characterized in that, The objective function is solved using a greedy strategy that maximizes marginal revenue per unit cost.

35. A method for predicting vehicle-mounted tasks, characterized in that, Includes the following steps: The vehicle terminal acquires multimodal sensor data, performs text semantic representation conversion on the multimodal sensor data to obtain modal text, and uploads it to the edge server; The edge server performs text semantic representation transformation on the vehicle-mounted task to obtain task text; constructs modal embedding vectors and task embedding vectors based on the modal text and task text respectively, and calculates the semantic relevance index of each modal sensor data to the vehicle-mounted task; calculates the time value index of each modality to the vehicle-mounted task based on the modal text and task text at any time; calculates the comprehensive benefit of each modality to the vehicle-mounted task based on the semantic relevance index and the time value index; selects the optimal modal combination based on the comprehensive benefit under resource budget constraints; performs vehicle-mounted task prediction based on the task embedding vector and the modal embedding vector of the sensor data of the optimal modal combination to obtain the vehicle-mounted task prediction result, and sends the vehicle-mounted task prediction result to the vehicle terminal.

36. A vehicle-mounted task prediction system, characterized in that, It includes a vehicle terminal and an edge server. The vehicle terminal includes a data acquisition module and a sensor semantic conversion unit. The edge server includes a task semantic conversion unit, a semantic relevance calculation unit, a time value calculation unit, a comprehensive benefit calculation unit, an optimal mode combination selection unit, and an on-board task prediction unit. The data acquisition module is used to acquire multimodal sensor data from the vehicle terminal. The sensing semantic conversion unit is used to convert the multimodal sensor data into text semantic representation to obtain modal text; The in-vehicle semantic conversion unit is used to convert the in-vehicle task into a text semantic representation to obtain the task text. The semantic relevance calculation unit is used to construct modal embedding vectors and task embedding vectors based on the modal text and task text, respectively, and calculate the semantic relevance index of each modal sensor data to the vehicle task. The time value calculation unit is used to calculate the time value index of each modality for the vehicle-mounted task based on the modal text and task text at any given time. The comprehensive benefit calculation unit is used to calculate the comprehensive benefit of each modality for the vehicle-mounted task based on the semantic relevance index and the time value index. The optimal mode combination selection unit is used to select the optimal mode combination based on the comprehensive benefits under resource budget constraints. The vehicle-mounted task prediction unit is used to perform vehicle-mounted task prediction based on the task embedding vector and the mode embedding vector of the sensor data of the optimal mode combination.

37. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the vehicle task prediction method according to any one of claims 1-17, 35.

38. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the vehicle task prediction method according to any one of claims 1-17 and 35.

39. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the vehicle task prediction method according to any one of claims 1-17 and 35.

Images