Intelligent cabin system, vehicle control method, electronic equipment and vehicle

Abstract

The invention provides an intelligent cabin system, a vehicle control method, electronic equipment and a vehicle, relates to the technical field of vehicles, and can collect and respond to multi-source data in real time, analyze the driving demand of a user, provide corresponding services and improve the driving experience of the vehicle. The intelligent cabin system comprises a data collection layer, a data processing layer and an intelligent decision-making layer; the data acquisition layer is used for acquiring multi-modal sensor data; the multi-modal sensor data comprises physiological data, behavior data and environment data; the data processing layer is used for carrying out preprocessing and data integration on the multi-modal sensor data to obtain processed data; the current driving scene of the vehicle is identified based on the processed data; and the intelligent decision-making layer is used for carrying out resource allocation based on the current driving scene of the vehicle, analyzing user demands based on the current driving scene of the vehicle and the historical behavior data of the user, and executing tasks related to the user demands.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to an intelligent cockpit system, a vehicle control method, an electronic device, and a vehicle. Background Art

[0002] The smart cockpit system is equipped with advanced software and hardware, and has the ability to integrate human-machine-environment, providing drivers and passengers with a safe, intelligent, efficient and pleasant comprehensive experience of the car mobile space. The smart cockpit is equipped with advanced software and hardware systems, and has the ability to integrate human-machine-environment, human-machine-environment, human-computer interaction, network services, and scene expansion. It can create a safe, intelligent, efficient and pleasant mobile space for drivers and passengers. For example, it can automatically adjust seats and ambient lights according to the habits of the driver and passengers, and can also link with smart homes to prepare the environment in advance.

[0003] At present, the intelligent cockpit system makes the car no longer just a means of transportation, but becomes a mobile intelligent space that can sense needs, proactively provide services, and continuously evolve, reshaping people's travel experience. However, the structural design of most intelligent cockpit systems is simple, the functional design is confusing, and it is difficult to meet complex driving needs. Summary of the Invention

[0004] The present invention aims to provide an intelligent cockpit system, a vehicle control method, an electronic device and a vehicle, which can provide rich driving services and enhance the vehicle driving experience.

[0005] In the first aspect, the present application provides an intelligent cockpit system, which includes: a data acquisition layer, a data processing layer, and an intelligent decision-making layer; the data acquisition layer is used to collect multimodal sensor data; the multimodal sensor data includes: physiological data, behavioral data and environmental data; the data processing layer is used to pre-process and integrate the multimodal sensor data to obtain processed data; and then identify the current driving scene of the vehicle based on the processed data; the intelligent decision-making layer is used to allocate resources based on the current driving scene of the vehicle, and analyze user needs based on the current driving scene of the vehicle and the user's historical behavior data, and perform tasks related to user needs.

[0006] The intelligent cockpit system provided in the present application utilizes a hierarchical architecture comprised of a "data acquisition layer, data processing layer, and intelligent decision-making layer" to uniformly collect, preprocess, and fuse multimodal sensory information, including physiological, behavioral, and environmental data, to generate a scene label corresponding to the current driving scenario. The intelligent decision-making layer dynamically allocates computing resources based on this scene label, draws upon a pre-stored database of user historical behavior, and outputs task instructions based on a demand prediction model to drive in-vehicle functions relevant to user needs. This allows for analysis of complex user needs through multi-source data and timely responses, thereby enhancing the user experience of driving the vehicle.

[0007] In combination with the first aspect above, in one possible implementation method, the intelligent decision-making layer is specifically used to determine user needs based on the vehicle's current driving scenario, the user's historical behavior data and the demand analysis model; wherein the demand analysis model is used to analyze the user's historical behavior data through federated learning, and determine user needs in combination with the vehicle's current driving scenario.

[0008] In combination with the first aspect above, in one possible implementation, the intelligent decision-making layer is also used to predict the destination based on the vehicle's current driving scenario, the user's historical behavior data and environmental data, and to perform route planning based on the destination.

[0009] In combination with the first aspect above, in one possible implementation, the data processing layer is specifically used to determine the current driving scene of the vehicle based on the processed data and the driving scene recognition model; wherein the driving scene recognition model is used to determine the current driving scene of the vehicle based on the processed data and preset scene matching rules.

[0010] In combination with the first aspect above, in one possible implementation, the data acquisition layer is also used to obtain the user's multimodal input; the data acquisition layer is also used to detect the driver's driving status and, when it is detected that the driver is in the target driving state, issue a reminder to the driver.

[0011] In combination with the first aspect above, in one possible implementation, the data acquisition layer includes a semantic microphone matrix; when the user's multimodal input includes voice commands, the semantic microphone matrix is ​​used to obtain the user's voice commands and distinguish between valid commands and invalid commands used in the voice commands.

[0012] In combination with the first aspect above, in one possible implementation, the data processing layer includes edge computing nodes and central computing nodes; the edge computing nodes are deployed on the sensor devices of the vehicle and are used to pre-process the sensor data generated by the edge sensor devices; the central computing nodes are used to integrate multimodal sensor data to obtain processed data.

[0013] In combination with the first aspect above, in one possible implementation, the smart cockpit system also includes a system maintenance layer for optimizing and maintaining the smart cockpit system; the system maintenance layer includes: a federated learning module, a remote upgrade module and a fault redundancy module; the federated learning module is used to optimize the smart cockpit system based on a federated learning optimization model, wherein the federated learning optimization model is trained based on shared vehicle data in the cloud; the remote upgrade module is used to upgrade the version of the smart cockpit system based on a differential update method; the fault redundancy module is used to detect the operating status of the main functional nodes in the smart cockpit system, and when an operating failure is detected in the main functional node, replace the main functional node to implement the corresponding function.

[0014] In combination with the first aspect above, in one possible implementation, the smart cockpit system also includes a secure communication layer, which is used to provide full-link security protection for the smart cockpit system and provide low-latency communication; the secure communication layer includes: a data security module and a communication module; the data security module is used to encrypt the data transmission link of the smart cockpit system based on TLS and biometric technology; the communication module is used to transmit data based on a low-latency communication protocol.

[0015] In a second aspect, the present application provides a vehicle control method, which is applied to the smart cockpit system provided in the first aspect above, and the method includes: obtaining multimodal sensor data; determining the current driving scenario of the vehicle based on the multimodal sensor data; allocating resources based on the current driving scenario of the vehicle; analyzing user needs based on the current driving scenario of the vehicle and the user's historical behavior data, and performing tasks related to the user needs.

[0016] The vehicle control method provided in the embodiment of the present application integrates multimodal sensor data, predicts driving scenarios, adaptively completes resource allocation, analyzes user needs based on multi-source data such as driving scenarios, and completes corresponding task execution. It can meet complex driving needs and enhance the user experience of vehicle driving.

[0017] In a third aspect, the present application provides a vehicle control device for implementing the vehicle control method provided in the second aspect above, the vehicle control device comprising: an acquisition module and a processing module; the acquisition module is used to acquire multimodal sensor data; the processing module is used to determine the current driving scenario of the vehicle based on the multimodal sensor data; the processing module is also used to allocate resources based on the current driving scenario of the vehicle; the processing module is also used to analyze user needs based on the current driving scenario of the vehicle and the user's historical behavior data, and to perform tasks related to the user needs.

[0018] In a fourth aspect, the present application provides an electronic device comprising: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method of the second aspect above.

[0019] In a fifth aspect, the present application provides a computer-readable storage medium, which includes: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device implements the method of the second aspect above.

[0020] In a sixth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the steps of the related method described in the fifth aspect to implement the method of the second aspect.

[0021] In a seventh aspect, the present application provides a vehicle comprising the smart cockpit system provided in the first aspect above; or, the electronic device provided in the fourth aspect above.

[0022] The beneficial effects of the third to seventh aspects mentioned above can be referred to the corresponding description of the first or second aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 A schematic diagram of the composition of an intelligent cockpit system provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of another intelligent cockpit system provided in an embodiment of the present application;

[0026] Figure 3 A flow chart of a vehicle control method provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the composition of a vehicle control device provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in the embodiments of this application as "exemplarily" or "for example" should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0031] In the embodiments of the present application, the terms "first," "second," "third," "fourth," "fifth," and "sixth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, a feature specified as "first," "second," "third," "fourth," "fifth," and "sixth" may explicitly or implicitly include one or more of the features.

[0032] In the embodiments of the present application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0033] “A and / or B” includes the following three combinations: A only, B only, and a combination of A and B.

[0034] As described in the background technology, the smart cockpit system has gradually become the core carrier for the evolution of vehicles from "travel tools" to "third living space". However, most of the existing smart cockpits have problems such as simple design and insufficient response, and cannot meet complex driving needs.

[0035] In response to the above technical problems, an embodiment of the present application provides an intelligent cockpit system that can collect and respond to multi-source data in real time, analyze user driving needs and provide corresponding services to improve the vehicle driving experience. The system includes: a data acquisition layer, a data processing layer, and an intelligent decision-making layer; the data acquisition layer is used to collect multimodal sensor data; the multimodal sensor data includes: physiological data, behavioral data and environmental data; the data processing layer is used to pre-process and integrate the multimodal sensor data to obtain processed data; and then identify the current driving scene of the vehicle based on the processed data; the intelligent decision-making layer is used to allocate resources based on the current driving scene of the vehicle, and analyze user needs based on the current driving scene of the vehicle and the user's historical behavior data, and perform tasks related to user needs.

[0036] Figure 1 A schematic diagram of the composition of an intelligent cockpit system provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the intelligent cockpit system 100 includes: a data collection layer 101, a data processing layer 102, and an intelligent decision-making layer 103.

[0037] In some embodiments, the data acquisition layer 101 is used to collect multi-modal sensor data.

[0038] Among them, multimodal sensor data includes: physiological data, behavioral data and environmental data.

[0039] For example, physiological data include heart rate, body temperature, EEG signals, etc., which are collected through on-board physiological sensors; behavioral data include steering wheel operation, pedal force, gesture movements, etc., which are collected through cameras and pressure sensors; environmental data include temperature and humidity inside the vehicle, road conditions outside the vehicle, light intensity, etc., which are collected through temperature and humidity sensors, millimeter-wave radars, and cameras.

[0040] Exemplarily, the data acquisition layer 101 includes a sensor fusion array, an infrared thermal imaging camera, a semantic microphone array matrix, a bioelectric sensor, and an environmental sensor.

[0041] In some embodiments, the data collection layer 101 is also used to obtain multimodal input from the user.

[0042] Exemplarily, multimodal input includes voice commands, gesture commands, touch commands, etc.

[0043] Exemplarily, voice commands may be acquired through a semantic microphone matrix, gesture commands may be acquired through a gesture sensor, and touch commands may be acquired through a touch screen.

[0044] In some embodiments, the data collection layer 101 includes a semantic microphone matrix. When the user's multimodal input includes voice commands, the semantic microphone matrix is ​​used to obtain the user's voice commands and distinguish valid commands from invalid commands in the voice commands.

[0045] For example, the semantic microphone matrix has a built-in semantic recognition algorithm, which extracts command features through the semantic recognition algorithm and combines contextual semantic analysis to distinguish valid commands (such as "turn on the air conditioner") and invalid commands (such as chatting, noise, etc.).

[0046] In some embodiments, the data collection layer 101 is also used to detect the driver's driving state, and to issue a reminder to the driver when it is detected that the driver is in the target driving state.

[0047] Exemplarily, the data acquisition layer 101 includes a camera and a steering wheel sensor. The camera detects data such as the driver's eye blinking frequency and head posture, and the steering wheel sensor detects the driver's operating stability, thereby identifying whether the driver is in a target driving state (such as fatigue, distraction, drunk driving, etc.), and issues reminders such as sound and light alarms, seat vibrations, etc. when the driver is detected to be in the target state.

[0048] The data acquisition layer 101 comprehensively collects physiological, behavioral, environmental and other data through multimodal sensors, and obtains multimodal inputs such as voice and gestures. Its semantic microphone matrix can accurately distinguish between valid and invalid commands, and can also use cameras and sensors to monitor the driver's status and provide timely reminders. This not only provides the system with rich and accurate original information, but also actively ensures driving safety, greatly improving the accuracy of human-computer interaction and driving safety.

[0049] In some embodiments, the data processing layer 102 is used to pre-process and integrate multimodal sensor data to obtain processed data, and then identify the current driving scene of the vehicle based on the processed data.

[0050] In some embodiments, the data processing layer 102 includes edge computing nodes and central computing nodes.

[0051] In some embodiments, the edge computing node is deployed on the edge sensor device of the vehicle to pre-process the sensor data generated by the edge sensor device.

[0052] Exemplarily, edge sensors include cameras, radars, etc.

[0053] Exemplarily, the edge computing node performs preliminary processing on the edge sensor data, such as filtering outliers, data noise reduction, format conversion, etc.

[0054] In some embodiments, a simple data processing model is preset in the edge computing node, such as an abnormal heart rate threshold judgment model, a distraction behavior recognition model, etc. The edge computing node marks and filters the collected data in real time to reduce data redundancy and provide accurate data for the data processing layer 102.

[0055] In some embodiments, the central computing node is used to integrate multimodal sensor data to obtain processed data.

[0056] In some embodiments, the central computing node is deployed in the vehicle-mounted central control system, receives pre-processed data from each edge node, integrates multimodal data through a data fusion algorithm, and obtains structured processed data.

[0057] For example, the structured data may be in the following form: "driver's heart rate 80 beats / minute + vehicle speed 100 km / h + rain outside the car".

[0058] In some embodiments, the data processing layer 102 is specifically configured to determine the current driving scene of the vehicle based on the processed data and the driving scene recognition model.

[0059] Among them, the driving scene recognition model is used to determine the vehicle's current driving scene based on processed data and preset scene matching rules.

[0060] Exemplarily, the driving scenario of a vehicle may include the following elements: time, weather, road conditions, vehicle speed, etc.

[0061] For example, "vehicle speed > 90km / h + highway sign + no pedestrians on the road" matches the "high-speed driving scene", and "vehicle speed < 5km / h + shopping mall nearby + time 6pm" matches the "parking and shopping scene".

[0062] Data processing layer 102 pre-processes sensor data through edge computing nodes to optimize data quality. Central computing nodes then integrate multimodal data into structured information, enabling accurate recognition of driving scenarios. This process effectively reduces data errors and improves processing efficiency, providing accurate data tailored to actual scenarios for subsequent intelligent decision-making, ensuring the system can quickly respond to the needs of diverse driving scenarios.

[0063] In some embodiments, the intelligent decision-making layer 103 is used to allocate resources based on the current driving scenario of the vehicle, analyze user needs based on the current driving scenario of the vehicle and the user's historical behavior data, and perform tasks related to the user needs.

[0064] In some embodiments, the intelligent decision-making layer 103 dynamically adjusts the cockpit system's computing power, power, network bandwidth and other resources based on the vehicle's current driving scenario.

[0065] For example, for high-speed driving scenarios, resources are allocated first to ensure vehicle driving functions, such as lane keeping and collision warning.

[0066] For example, for parking and rest scenarios, priority is given to allocating resources to ensure vehicle leisure functions, such as video playback, gaming, Bluetooth, etc.

[0067] In some embodiments, the intelligent decision-making layer 103 is specifically used to determine user needs based on the vehicle's current driving scenario, the user's historical behavior data and the demand analysis model.

[0068] Among them, the demand analysis model is used to analyze the user's historical behavior data through federated learning, and determine user needs in combination with the vehicle's current driving scenario.

[0069] In some embodiments, multiple vehicle terminals upload local vehicle data to the cloud, which aggregates multiple vehicle data to form a demand analysis model and transmits it to the vehicle terminals. The intelligent decision-making layer 103 inputs the vehicle's current driving scenario and the user's historical behavior data into the demand analysis model to determine user needs.

[0070] For example, the current scenario is input as "commuting + Monday", and the demand analysis model outputs the user demand "playing morning news".

[0071] In some embodiments, the intelligent decision-making layer 103 is also used to predict the destination based on the vehicle's current driving scenario, the user's historical behavior data and environmental data, and to plan a route based on the destination.

[0072] In some embodiments, the intelligent decision-making layer 103 uses a path planning algorithm to predict the destination and generate the optimal path based on the current driving scenario, user historical behavior data and environmental data.

[0073] For example, the user's historical behavior data may include that the user goes to the gym every Friday at 6 pm.

[0074] For example, environmental data include real-time traffic data, weather data, etc.

[0075] For example, the intelligent decision-making layer 103 detects that the current driving scenario is "time Friday afternoon + vehicle is located near the company", and inputs the data into the path planning algorithm. The path planning algorithm predicts that the user's destination is the gym, and plans the optimal path for the vehicle based on the environmental data.

[0076] In some embodiments, the intelligent decision-making layer 103 obtains environmental data in real time and dynamically adjusts the planned path through a path planning algorithm.

[0077] The intelligent decision-making layer 103 dynamically allocates computing power, electricity and other resources according to the driving scenario to ensure the stable operation of core functions; it analyzes needs based on the scenario and user historical behavior and performs related tasks to provide personalized services; it can also predict the destination and plan the optimal route, and dynamically adjust to improve travel efficiency, comprehensively improving the intelligence and personalized service level of the cockpit system and enhancing the user's driving experience.

[0078] In some embodiments, as Figure 2 As shown, the smart cockpit system 100 further includes a system maintenance layer 104 for optimizing and maintaining the smart cockpit system 100 .

[0079] In some embodiments, the system maintenance layer 104 includes: a federated learning module, a remote upgrade module, and a fault redundancy module.

[0080] In some embodiments, the federated learning module is used to optimize the smart cockpit system 100 based on a federated learning optimization model.

[0081] Among them, the federated learning optimization model is trained based on shared vehicle data in the cloud.

[0082] In some embodiments, the federated learning module trains a federated learning optimization model based on vehicle data shared in the cloud, and regularly updates the algorithms and models included in the smart cockpit system 100, such as path planning algorithms, scene recognition models, demand analysis models, etc.

[0083] In some embodiments, the remote upgrade module is used to upgrade the version of the smart cockpit system 100 based on a differential update method.

[0084] Exemplarily, the remote upgrade module regularly reports the current version of the smart cockpit system 100 to the server. The server generates a differential update file after comparing the new and old versions using a differential algorithm. After compression, encryption and signing, the file is transmitted to the remote upgrade module for differential update.

[0085] For example, version 2.0 is an upgraded version of version 1.0. If only 10% of the code in version 1.0 is modified in version 2.0, then the differential update file only contains update information about this 10% of the code.

[0086] In some embodiments, the fault redundancy module is used to detect the operating status of the main functional node in the smart cockpit system 100, and when an operating failure is detected in the main functional node, replace the main functional node to implement the corresponding function.

[0087] Exemplarily, the main function nodes include navigation function nodes, voice interaction function nodes, etc.

[0088] Illustratively, the fault redundancy module monitors the operating status of the navigation function node in real time, and when the navigation signal of the navigation function node is lost, the navigation function node is switched to a backup function node to ensure that the navigation function is not interrupted.

[0089] The system maintenance layer 104 uses the federated learning module to optimize system algorithms and models based on multi-vehicle data, allowing the system to continuously evolve; the remote upgrade module uses differential updates to achieve efficient version upgrades and reduce maintenance costs; the fault redundancy module switches to a backup node when the main functional node fails to ensure uninterrupted functions, effectively improving the system's adaptability, maintenance convenience, and operational reliability.

[0090] In some embodiments, as Figure 2 As shown, the smart cockpit system 100 also includes a secure communication layer 105, which is used to provide full-link security protection for the smart cockpit system 100 and provide low-latency communication.

[0091] In some embodiments, the secure communication layer 105 includes: a data security module and a communication module.

[0092] In some embodiments, the data security module is used to encrypt the data transmission link of the smart cockpit system 100 based on TLS and biometric technology.

[0093] In some embodiments, the data security module encrypts the transmission link of sensor data and command data through the TLS encryption protocol to prevent the data from being tampered with or stolen, and combines biometric technologies such as driver facial recognition technology and fingerprint verification technology to confirm user identity and only perform operations for authorized users.

[0094] In some embodiments, the communication module is used to transmit data based on a low-latency communication protocol.

[0095] Exemplarily, the communication module uses a combination of 5G and V2X protocols to transmit data between various layers of the intelligent cockpit system 100, as well as from the vehicle to other traffic communication objects. Exemplarily, other traffic communication objects include vehicles, the cloud, and road test equipment.

[0096] The data security module of the secure communication layer 105 uses TLS encryption and biometric technology to protect data transmission security and user identity authentication. The communication module adopts a low-latency protocol to achieve efficient data transmission, which not only ensures that data is not tampered with, stolen, or subjected to unauthorized operations, but also meets the real-time requirements of each layer of the system and communications with other traffic objects, providing security and communication guarantees for the entire smart cockpit system 100.

[0097] The vehicle control method provided by the embodiment of the present application is described below with reference to specific embodiments and accompanying drawings:

[0098] Figure 3 A flow chart of a vehicle control method provided in an embodiment of the present application is applied to an intelligent cockpit system, such as Figure 3 As shown, the method includes the following:

[0099] S101: Acquire multimodal sensor data.

[0100] In some embodiments, multimodal sensor data is acquired through the data acquisition layer of the smart cockpit system.

[0101] In some embodiments, the multimodal sensor data includes physiological data, behavioral data, and environmental data.

[0102] For example, physiological data include heart rate, body temperature, EEG signals, etc., behavioral data include steering wheel operation, pedal force, gesture movements, etc., and environmental data include temperature and humidity inside the car, road conditions outside the car, light intensity, etc.

[0103] In some embodiments, the multimodal sensor data also includes a user's multimodal input. For example, the multimodal input includes voice commands, gesture commands, touch commands, etc.

[0104] Exemplarily, voice commands may be acquired through a semantic microphone matrix, gesture commands may be acquired through a gesture sensor, and touch commands may be acquired through a touch screen.

[0105] S102: Determine the current driving scenario of the vehicle based on multimodal sensor data.

[0106] In some embodiments, the smart cockpit system includes a data processing layer, and the data acquisition layer sends the collected multimodal data to the data processing layer. The data processing layer processes the multimodal data to determine the current driving scenario of the vehicle.

[0107] In some embodiments, the acquired multimodal sensor data is preprocessed, such as format normalization processing, feature extraction processing, etc.

[0108] Exemplarily, the feature extraction process may be to extract distraction behavior features, where the distraction behavior features are composed of a user's head rotation frequency and a user's achieved deviation angle.

[0109] In some embodiments, the data processing layer determines the vehicle's current driving scene based on the processed data and a driving scene recognition model, wherein the driving scene recognition model is used to determine the vehicle's current driving scene based on the processed data and preset scene matching rules.

[0110] Exemplarily, the driving scenario of a vehicle may include the following elements: time, weather, road conditions, vehicle speed, etc.

[0111] For example, "vehicle speed > 90km / h + highway sign + no pedestrians on the road" matches the "high-speed driving scene", and "vehicle speed < 5km / h + shopping mall nearby + time 6pm" matches the "parking and shopping scene".

[0112] S103: Allocate resources based on the current driving scenario of the vehicle.

[0113] In some embodiments, the smart cockpit system includes an intelligent decision-making layer, which allocates resources based on the vehicle's current driving scenario. Exemplarily, the allocable resources include the system's computing power, power, network bandwidth, etc.

[0114] For example, in high-speed driving scenarios, resources are allocated first to ensure vehicle driving functions, such as lane keeping and collision warning. For example, 80% of the GPU computing power is allocated to the visual perception module (for lane line detection and preceding vehicle identification), and the remaining 20% ​​is reserved for cockpit UI rendering. 70% of the battery output power is prioritized to the vehicle driving domain controller (to ensure full load operation of millimeter-wave radar and lidar), and the cockpit domain controller is limited to 30%. The bandwidth of the on-board 5G module is prioritized to transmit high-precision map update packages (e.g., 60% bandwidth is reserved) and driving assistance algorithm models (e.g., 30% is reserved), with only 10% used for passenger hotspot sharing.

[0115] For example, for parking and rest scenarios, priority is given to allocating resources to ensure vehicle leisure functions, such as video playback, gaming, Bluetooth, etc. For example, 60% of the GPU computing power is switched to the entertainment decoding module (supporting 4K video hardware decoding), 30% is used for game rendering, and only 10% of the computing power is retained for basic vehicle monitoring (such as tire pressure monitoring). 50% of the battery output power is allocated to the cabin entertainment system (such as audio amplifier, display), 30% is used for air conditioning to maintain a comfortable temperature, and the remaining 20% ​​is used for background silent updates. 40% of the bandwidth is used for high-definition video caching (such as preloading user-subscribed streaming content), 30% is used for game data synchronization, 20% of the bandwidth is used for Bluetooth audio streaming, and the remaining 10% of the bandwidth is used for vehicle health data reporting.

[0116] In some embodiments, a dynamic expert scheduling mechanism can be used to dynamically allocate resources according to different driving scenarios.

[0117] For example, we first assign feature labels to different driving scenarios, including the event confidence vector e (e.g., overtaking confidence 0.9), the intention weight vector i (e.g., navigation priority weight 0.8), the normalized vehicle speed v, and the weather embedding vector w. These feature labels are then concatenated to form a driving scene feature vector, which is represented as:

[0118] x=[e,i,v,w]

[0119] Next, a lightweight gating network is used to map the driving scene features to expert weights. The activation weight of each expert sub-network is calculated based on the driving scene feature vector x, which in turn determines which experts to activate and how much computing power to allocate. The activation weight of the expert sub-network satisfies the following formula:

[0120] z=ReLU(W1x+b1),

[0121] zw i =softmax(W2z+b1),

[0122] Among them, W1 and W2 are weight matrices, w i is the activation weight of the i-th expert.

[0123] Then, based on the activation weight w i , using the Top-K selection mechanism to activate N / 8 experts (N is the number of activation weights). The activated experts form a set S, and the computing power ratio of each activated expert satisfies the following formula:

[0124] score i =w i c i ,

[0125]

[0126] Where T is the total computing power budget for the driving scenario.

[0127] For example, when the two experts of “perception + prediction” are activated, if w1=0.6, c1=1.2, w2=0.4, c2=1.0, then we can calculate: score1=0.72, score2=0.4, and then we get α1≈45.8%, α2≈24.2%

[0128] S104: Analyze user needs based on the vehicle's current driving scenario and the user's historical behavior data, and execute tasks related to the user needs.

[0129] In some embodiments, the intelligent decision-making layer of the smart cockpit system uses a demand analysis model to analyze the user's historical behavior data through federated learning, and determines the user's needs in combination with the vehicle's current driving scenario.

[0130] In some embodiments, the demand analysis model is trained based on data from multiple vehicle terminals. Multiple vehicle terminals upload their local vehicle data to the cloud, which aggregates the data to form a demand analysis model and transmits it to the vehicle terminals. The intelligent decision-making layer then inputs the demand analysis model into the vehicle's current driving scenario and the user's historical behavior data to determine user needs.

[0131] Exemplarily, user requirements include route planning requirements, driving mode requirements, etc.

[0132] In some embodiments, city-level congestion information, regional meteorological data, user commuting route library and vehicle sensor data are accessed in real time through the cloud platform to generate a global dynamic road condition map, and then the pre-trained path planning model is called to combine user historical data with real-time road conditions to dynamically output the optimal path.

[0133] In some embodiments, a global dynamic traffic map is generated to construct a graph structure with spatiotemporal dependencies:

[0134] G=(V,E,A t )

[0135] Among them, the node V represents the intersection, including the location, number of lanes, signal cycle and other characteristics, the edge E represents the road section, including the length, speed limit, congestion index and other characteristics, the spatiotemporal adjacency matrix A t Represents the connectivity and travel cost between nodes at time t, and then uses the spatiotemporal convolutional network to update the node features. The following formula is used:

[0136]

[0137] in, is the adjacency matrix, is the degree matrix, W (l) and U (l) is a learnable parameter, is the time transfer matrix.

[0138] In some embodiments, the preset driving mode in the vehicle includes a driver behavior intervention mode, and the driver behavior intervention mode includes functions such as seat vibration reminder, automatic vehicle deceleration, etc.

[0139] Exemplarily, the process of the vehicle control method is introduced below by taking the driver's fatigue driving as an example.

[0140] First, the camera in the data acquisition layer collects the driver's facial data. After detection and calculation, if the proportion of the driver's eye closure time reaches or exceeds the threshold (for example, the threshold is 0.8 and the actual proportion is 0.85), the driver is determined to be in the target driving state of "fatigue" and an audio-visual reminder is immediately issued (such as the dashboard displays "Please take a break" accompanied by a buzzer alarm, etc.), and the driver's fatigue status data is transmitted to the data processing layer.

[0141] Next, the data processing layer integrates the fatigue status data and vehicle data to obtain the structured data "fatigue status + vehicle speed 70km / h + current road section 5km away from the service area", identifying the current situation as a "rest-required scenario".

[0142] Then, the intelligent decision-making layer combines historical data of users choosing to rest at the nearest service area when they are tired, analyzes user needs through a demand analysis model, and predicts that the destination is a service area 5 km ahead.

[0143] Finally, the intelligent decision-making layer issues instructions to control each vehicle module to perform the corresponding tasks: the navigation module plans the route to the service area and broadcasts the planning information by voice, while the driving assistance module enhances the lane keeping function to reduce the risk of vehicle deviation.

[0144] The vehicle control method provided in the embodiment of the present application can enhance the user experience of vehicle driving by fusing multimodal sensor data, predicting driving scenarios, adaptively allocating resources according to driving scenarios, analyzing user needs based on multi-source data such as driving scenarios, and completing corresponding task execution.

[0145] In some embodiments, the vehicle control method provided in the embodiments of the present application can also realize real-time update optimization of the vehicle system or version. Multiple vehicle terminals are configured with a federated learning model. After the model completes one or more iterative training locally, the model data is uploaded to the cloud server. The cloud server uses the differential update mechanism to weightedly aggregate the parameters from multiple vehicle terminals to generate a new global model, and transmits the differential update package of the global model back to the vehicle terminal through the incremental distribution channel. The vehicle terminal completes the update using the score query update package.

[0146] In some embodiments, the vehicle control method provided by the embodiments of the present application can also detect the operating status of a primary functional node in real time and, if a primary functional node fails, control a redundant design to replace the primary functional node to perform the corresponding function. Exemplary primary functional nodes include navigation functional nodes, voice interaction functional nodes, and the like.

[0147] In some embodiments, the vehicle control method provided by the embodiments of the present application can also implement full-link encryption. The data acquisition layer encrypts the collected data and transmits it through a two-way certificate channel using the TLS encryption method. During the transmission process, it verifies the data using a preset public key and performs dual authentication of face and voiceprint.

[0148] In some embodiments, the vehicle control method provided in the embodiments of the present application can also achieve low-latency communication, using the 5G-V2X protocol to establish a direct link with the roadside unit, and receive real-time information such as traffic light phase, construction warnings, speed limit updates, etc.

[0149] It can be seen that the above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, the embodiment of the present application provides hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the modules and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0150] In the embodiment of the present application, the vehicle control device can be divided into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, other division methods can be used.

[0151] Figure 4 This is a schematic diagram of the composition of a vehicle control device provided in this application. Figure 4 As shown, the vehicle control device 400 includes: an acquisition module 401 and a processing module 402. Acquisition module 401 is used to acquire multimodal sensor data; processing module 402 is used to determine the vehicle's current driving scenario based on the multimodal sensor data; processing module 402 is also used to allocate resources based on the vehicle's current driving scenario; processing module 402 is also used to analyze user needs based on the vehicle's current driving scenario and the user's historical behavior data, and perform tasks related to the user needs.

[0152] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the embodiment of the present invention provides a possible structural diagram of the electronic device involved in the above-mentioned embodiment. Figure 5 As shown, the electronic device 500 includes: a processor 502 , a communication interface 503 , and a bus 504 . Optionally, the electronic device 500 may further include a memory 501 .

[0153] Processor 502 may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 502 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 502 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0154] The communication interface 503 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0155] The memory 501 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0156] As a possible implementation, the memory 501 may exist independently of the processor 502. The memory 501 may be connected to the processor 502 via a bus 504 and used to store instructions or program codes. When the processor 502 calls and executes the instructions or program codes stored in the memory 501, the model parameter compression method or model parameter decoding method provided in the embodiment of the present invention can be implemented.

[0157] In another possible implementation, the memory 501 may also be integrated with the processor 502 .

[0158] The bus 504 may be an extended industry standard architecture (EISA) bus, etc. The bus 504 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0159] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0160] The present application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments may be performed by computer program instructions directed to the relevant hardware. The program may be stored in the above computer-readable storage medium. When the computer program instructions are executed on a computer, the computer executes the model parameter compression method or model parameter decoding method described in any of the above embodiments.

[0161] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0162] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program product is run on a computer, it enables the computer to execute any one of the model parameter compression methods or model parameter decoding methods provided in the above embodiments.

[0163] In the description of the embodiments of the present application, specific features, structures, materials or characteristics may be combined in an appropriate manner in any one or more embodiments or examples.

[0164] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An intelligent cockpit system, characterized in that: The intelligent cockpit system includes: a data acquisition layer, a data processing layer, and an intelligent decision-making layer; The data acquisition layer is used to collect multimodal sensor data; the multimodal sensor data includes: physiological data, behavioral data and environmental data; The data processing layer is used to pre-process and integrate the multimodal sensor data to obtain processed data; and then identify the current driving scene of the vehicle based on the processed data; The intelligent decision-making layer is used to allocate resources based on the current driving scenario of the vehicle, analyze user needs based on the current driving scenario of the vehicle and the user's historical behavior data, and perform tasks related to the user needs.

2. The intelligent cockpit system according to claim 1, characterized in that: The intelligent decision-making layer is specifically used to determine the user needs based on the current driving scenario of the vehicle, the user's historical behavior data and the demand analysis model; wherein the demand analysis model is used to analyze the user's historical behavior data through federated learning, and determine the user needs in combination with the current driving scenario of the vehicle.

3. The intelligent cockpit system according to claim 1, characterized in that: The intelligent decision-making layer is also used to predict the destination based on the current driving scenario of the vehicle, the user's historical behavior data and environmental data, and to plan a route based on the destination.

4. The intelligent cockpit system according to claim 1, characterized in that: The data processing layer is specifically used to determine the current driving scene of the vehicle based on the processed data and the driving scene recognition model; wherein the driving scene recognition model is used to determine the current driving scene of the vehicle based on the processed data and preset scene matching rules.

5. The intelligent cockpit system according to claim 1, characterized in that: The data collection layer is also used to obtain multimodal input from users; The data collection layer is further used to detect the driver's driving state and, when it is detected that the driver is in the target driving state, issue a reminder to the driver.

6. The intelligent cockpit system according to claim 5, characterized in that: The data acquisition layer includes a semantic microphone matrix; In the case where the user's multimodal input includes voice instructions, the semantic microphone matrix is ​​used to obtain the user's voice instructions and distinguish between valid instructions and invalid instructions in the voice instructions.

7. The intelligent cockpit system according to claim 1, characterized in that: The data processing layer includes edge computing nodes and central computing nodes; The edge computing node is deployed on the sensor device of the vehicle and is used to pre-process the sensor data generated by the edge sensor device; The central computing node is used to perform data integration on the multimodal sensor data to obtain the processed data.

8. The intelligent cockpit system according to claim 1, characterized in that: The smart cockpit system also includes a system maintenance layer for optimizing and maintaining the smart cockpit system; The system maintenance layer includes: a federated learning module, a remote upgrade module and a fault redundancy module; a federated learning module, configured to optimize the intelligent cockpit system based on a federated learning optimization model, wherein the federated learning optimization model is trained based on shared vehicle data in the cloud; The remote upgrade module is used to upgrade the version of the smart cockpit system based on a differential update method; The fault redundancy module is used to detect the operating status of the main function node in the intelligent cockpit system, and when an operating failure is detected in the main function node, replace the main function node to implement the corresponding function.

9. The intelligent cockpit system according to claim 1, characterized in that: The smart cockpit system also includes a secure communication layer for providing full-link security protection for the smart cockpit system and providing low-latency communication; The secure communication layer includes: a data security module and a communication module; The data security module is used to encrypt the data transmission link of the smart cockpit system based on TLS and biometric technology; The communication module is used to transmit data based on a low-latency communication protocol.

10. A vehicle control method, characterized in that: The method comprises: Acquire multimodal sensor data; determining a current driving scenario of the vehicle based on the multimodal sensor data; Allocating resources based on the current driving scenario of the vehicle; Based on the current driving scenario of the vehicle and the user's historical behavior data, user needs are analyzed and tasks related to the user needs are performed.

11. An electronic device, characterized in that: The electronic device includes: a processor and a memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the vehicle control method according to claim 10 .

12. A vehicle, characterized in that: The vehicle includes the smart cockpit system according to any one of claims 1 to 9; or the electronic device according to claim 11.

Images