Control method and device for cabin in vehicle, processor and electronic equipment

Abstract

The invention discloses a control method and device for a cabin in a vehicle, a processor and electronic equipment. The method comprises the steps that multiple kinds of initial modal data in a vehicle are obtained, space-time alignment processing is conducted on the multiple kinds of initial modal data, multiple kinds of target modal data are obtained, the initial modal data are used for representing sensing data, obtained through sensors, of the vehicle in the running process, and different initial modal data correspond to different sensors; the multiple target modal data are in one-to-one correspondence with the multiple initial modal data; inputting the multiple target modal data into a digital twin model for fusion to obtain fused modal data; inputting the fusion modal data into a graph neural network model for analysis to obtain knowledge graph information, and analyzing the knowledge graph information to obtain various control instructions; and controlling the cabin environment in response to a target control instruction in the plurality of control instructions. The technical problem that the cabin in the vehicle cannot be effectively controlled is solved.

Description

Technical Field

[0001] This invention relates to the field of vehicle control technology, and more specifically, to a control method, device, processor, and electronic equipment for a vehicle cabin. Background Technology

[0002] Currently, vehicle cabin environment control methods often employ single-modal data acquisition. For example, they rely solely on ambient temperature and humidity sensors or simple multimodal data overlay methods, and data transmission primarily uses fixed sampling triggering methods and basic verification mechanisms, lacking a unified spatiotemporal synchronization architecture. Furthermore, the decision-making process for the vehicle cabin environment often depends on monotonous, preset, and fixed rules, such as setting a fixed temperature threshold to trigger air conditioning adjustments.

[0003] In summary, simple multimodal data overlay is prone to data distortion due to inconsistencies in spatiotemporal references and lack of quantitative control over transmission error rates, leading to subsequent decisions deviating from actual needs. Therefore, technical challenges remain in effectively controlling the vehicle's cockpit.

[0004] There is currently no effective solution to the aforementioned technical problems. Summary of the Invention

[0005] This invention provides a method, apparatus, processor, and electronic device for controlling the cockpit of a vehicle, to at least solve the technical problem of the inability to effectively control the cockpit of a vehicle.

[0006] According to one aspect of the present invention, a method for effectively controlling the cabin in a vehicle is provided. The method may include: acquiring multiple initial modal data in a vehicle, and performing spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data, wherein the initial modal data is used to represent the perception data of the vehicle during driving obtained by sensors, different initial modal data correspond to different sensors, and the multiple target modal data correspond one-to-one with the multiple initial modal data; inputting the multiple target modal data into a digital twin model for fusion to obtain fused modal data, wherein the digital twin model is used to construct the mapping relationship between the fused modal data and the cabin environment in the vehicle; inputting the fused modal data into a graph neural network model for analysis to obtain knowledge graph information, and analyzing the knowledge graph information to obtain multiple control commands, wherein the graph neural network model is trained using fused modal data samples corresponding to the fused modal data, the knowledge graph information is used to represent the mapping relationship between multiple entity information in the fused modal data, and the control commands are used to control the cabin environment of the vehicle; and controlling the cabin environment in response to a target control command among the multiple control commands, wherein the priority of the target control command is higher than a priority threshold.

[0007] Optionally, the multiple initial modal data include at least visual data, biological data, speech data, and environmental data. Spatiotemporal alignment processing is performed on the multiple initial modal data to obtain multiple target modal data, including: extracting keypoint data from the visual data, where the keypoint data represents the motion trajectory of keypoints of the target object in the visual data; determining the spectral information corresponding to the biological data, where the spectral information represents the spectrum of the biological data; constructing a gradient field model based on the environmental data, and generating environmental distribution information using the gradient field model, where the environmental distribution information represents the temperature and humidity distribution of the environmental data; and performing spatiotemporal alignment processing on the keypoint data, spectral information, environmental distribution information, and speech data using a spatiotemporal synchronization bus to obtain multiple target modal data, where the spatiotemporal synchronization bus is a hierarchical clock tree architecture.

[0008] Optionally, multiple target modal data are input into the digital twin model for fusion to obtain fused modal data, including: using a cross-modal spatiotemporal encoder to input multiple target modal data into the digital twin model for fusion to obtain fused modal data.

[0009] Optionally, a cross-modal spatiotemporal encoder is used to input multiple target modal data into a digital twin model for fusion to obtain fused modal data, including: using a cross-modal spatiotemporal encoder to input multiple target modal data into a digital twin model for fusion to obtain tensor information; and inputting the tensor information into the state matrix of the digital twin model to obtain fused modal data.

[0010] Optionally, the method further includes: performing dual error correction processing on multiple target modal data to obtain the bit error rate of the dual error correction processing multiple target modal data, wherein the bit error rate is less than the bit error rate threshold.

[0011] Optionally, the fused modal data is input into a graph neural network model for analysis to obtain knowledge graph information, and the knowledge graph information is analyzed to obtain various control commands, including: inputting the fused modal data into the graph neural network model; using the fused modal data to update the edge weight parameters of the graph neural network model to obtain knowledge graph information; determining the size relationship between various entity information and entity information thresholds in the knowledge graph information; and generating control commands according to the size relationship.

[0012] Optionally, in response to a target control command among multiple control commands, the cockpit environment is controlled, including: determining the vehicle's bus load rate trigger threshold; prioritizing multiple control commands based on the load rate trigger threshold and the vehicle's driving scenario to obtain a ranking result; identifying the control command with a priority greater than the priority threshold in the ranking result as the target control command; and controlling the cockpit environment according to the target control command.

[0013] According to another aspect of the present invention, a control device for a vehicle cabin is also provided. The device may include: an acquisition unit, configured to acquire multiple initial modal data in the vehicle, and to perform spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data, wherein the initial modal data represents perception data of the vehicle during driving obtained by sensors, different initial modal data correspond to different sensors, and the multiple target modal data correspond one-to-one with the multiple initial modal data; a fusion unit, configured to input the multiple target modal data into a digital twin model for fusion to obtain fused modal data, wherein the digital twin model is used to construct a mapping relationship between the fused modal data and the cabin environment in the vehicle; and a confirmation unit. The fixed unit is used to input the fused modal data into the graph neural network model for analysis to obtain knowledge graph information, and to analyze the knowledge graph information to obtain various control commands. The graph neural network model is trained using fused modal data samples corresponding to the fused modal data. The knowledge graph information is used to represent the mapping relationship between various entity information in the fused modal data. The control commands are used to control the vehicle's cabin environment. The control unit is used to respond to the target control command among the various control commands to control the cabin environment. The target control command has a higher priority than the priority threshold.

[0014] According to another aspect of the present invention, a processor is also provided. The processor is used to run a program, wherein the program executes the methods of the embodiments of the present invention during runtime.

[0015] According to another aspect of the present invention, an electronic device is also provided, wherein a processor is configured to run a program, wherein the program executes the method of the present invention during runtime.

[0016] According to another aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method of the embodiments of the present invention.

[0017] According to another aspect of the present invention, a vehicle is also provided. The vehicle includes a memory and a processor. The memory stores an executable program; the processor runs the program, which, when executed, implements the methods described in the embodiments of the present invention.

[0018] In this embodiment of the invention, multiple initial modal data from the vehicle are acquired and spatiotemporally aligned to obtain multiple target modal data, ensuring precise synchronization of these target modal data in time and space. Then, the multiple target modal data are input into a digital twin model for fusion. The digital twin model can reflect various states of the vehicle's cabin environment in real time and construct a mapping relationship between the fused modal data and the vehicle's cabin environment. The fused modal data is further input into a graph neural network model for in-depth analysis. Because the graph neural network model learns and trains on a large number of fused modal data samples, it possesses the ability to understand the complex mapping relationships between multiple entity information in the cabin environment. By analyzing the fused modal data, knowledge graph information can be output, reflecting the mapping relationships between multiple entity information in the fused modal data. Using the knowledge graph information, multiple control commands can be generated to specifically improve the cabin environment. Then, based on priority evaluation, target control commands can be determined to control the cabin environment, thereby solving the technical problem of ineffective control of the vehicle cabin and achieving the technical effect of effective control of the vehicle cabin. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a flowchart of a vehicle cabin control method according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of an intelligent automotive cockpit environment adaptive control system that supports multimodal interaction according to an embodiment of the present invention.

[0022] Figure 3 This is a flowchart of a multimodal data transmission verification method according to an embodiment of the present invention;

[0023] Figure 4 This is a flowchart of a digital twin construction method according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of a control device for a vehicle cabin according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to an embodiment of the present invention, an embodiment of a control method for a vehicle cabin is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] Figure 1 This is a flowchart of a vehicle cabin control method according to an embodiment of the present invention, such as... Figure 1 As shown, the method may include the following steps.

[0029] Step S102: Acquire multiple initial modal data from the vehicle, and perform spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data.

[0030] In the technical solution provided in step S102 of the present invention, the initial modal data can be used to represent the perception data of the vehicle during driving obtained by the sensor. Different initial modal data can correspond to different sensors, and multiple target modal data can correspond one-to-one with multiple initial modal data.

[0031] In this embodiment, the various initial modal data can be visual data, speech data, biological data, and environmental data.

[0032] Optionally, visual data within the vehicle's cabin environment can be acquired using visual sensors. For example, visual sensors can capture the eyelid closure and gesture spatial coordinates of a target object (e.g., the driver) at a high-frequency sampling rate to determine the driver's attention state and interaction intentions.

[0033] Optionally, voice data within the vehicle's cabin environment can be acquired via a voice sensor to recognize the driver's voice commands or alarm sounds, thereby responding to the driver's needs or warning of potential hazards. The voice sensor's sampling rate is moderate, balancing clarity and processing load.

[0034] Optionally, biological data within the vehicle's cabin environment can be acquired through biosensors. For example, biosensors can monitor steering wheel grip strength and heart rate variability at a higher sampling rate, reflecting the driver's physiological state (such as anxiety, fatigue, etc.) and operational stability in real time.

[0035] Optionally, environmental data within the vehicle's cabin environment can be acquired through environmental sensors. For example, environmental sensors can acquire data such as temperature, humidity, light intensity, and air quality index to comprehensively assess cabin comfort and the impact of the external environment on the cabin.

[0036] Optionally, to ensure consistency in the time dimension and accuracy in the spatial dimension of various initial modal data, a GPS-disciplined clock chip can be used as the master clock source. The GPS-disciplined clock chip can provide a highly stable and low-drift time reference signal, i.e., generate a highly stable master clock signal. Subsequently, the master clock signal can be distributed to each sensor node through a star topology, reducing attenuation and interference during master clock signal propagation, ensuring clock reference consistency across nodes, and improving the accuracy and reliability of timestamps for various initial modal data.

[0037] In this embodiment, multiple initial modal data from the vehicle are acquired, and spatiotemporal alignment processing is performed on these initial modal data to obtain multiple target modal data, achieving efficient and accurate collection of multiple initial modal data. Spatiotemporal alignment processing provides high-quality input data for subsequent digital twin model construction and graph neural network model analysis, thereby improving the overall efficiency of vehicle cabin environment control and user experience.

[0038] Step S104: Input multiple target modal data into the digital twin model for fusion to obtain fused modal data.

[0039] In the technical solution provided in step S104 of the present invention, the digital twin model can be used to construct the mapping relationship between fused modal data and the cabin environment in the vehicle.

[0040] In this embodiment, after acquiring multiple initial modal data in the vehicle and performing spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data, the multiple target modal data can be input into a digital twin model (e.g., a cockpit environment digital twin, a digital twin) for fusion to obtain fused modal data, which maps the physical cockpit state in real time.

[0041] Optionally, by extracting key features from multiple target modal data and fusing these key features across time, space, modality, and eigenvalue dimensions using a cross-modal spatiotemporal encoder, a high-dimensional four-dimensional tensor can be generated. This fused four-dimensional tensor can then be injected into the state matrix of the digital twin to reflect the actual state of the cockpit in real time. The updating of the state matrix is ​​a dynamic process that can be adjusted based on the real-time collection and processing of multiple target modal data, ensuring consistency and synchronization between the digital twin and the physical cockpit environment.

[0042] In this embodiment, by extracting key features from multiple target modal data and integrating these key features into a structured digital twin model, a comprehensive and real-time state description can be provided for subsequent intelligent decision-making and control. The construction and dynamic updating of the digital twin model (e.g., a digital twin entity) ensures that the output of various subsequent control commands can be based on the real-time state of the cockpit environment, thereby improving the accuracy and responsiveness of vehicle cockpit environment control.

[0043] Step S106: Input the fused modal data into the graph neural network model for analysis to obtain knowledge graph information, and analyze the knowledge graph information to obtain various control commands.

[0044] In the technical solution of step S106 of the present invention, the graph neural network model can be obtained by training the neural network model using fusion modal data samples corresponding to the fusion modal data. Knowledge graph information can be used to represent the mapping relationships between various entity information in the fusion modal data. Control commands can be used to control the vehicle's cabin environment.

[0045] In this embodiment, after multiple target modal data are input into a digital twin model for fusion to obtain fused modal data, the fused modal data can be input into a graph neural network model for analysis to obtain knowledge graph information. Furthermore, the knowledge graph information can be analyzed to obtain various control commands. The nodes of the graph neural network model (Graph Neural Network) can be used to represent entity information in the cockpit environment, such as the driver's physiological state, vehicle motion parameters, and environmental comfort zones, while the edges of the graph neural network can be used to represent the mapping relationships or interactions between these entity information.

[0046] Alternatively, through reinforcement learning algorithms, graph neural networks can learn how to predict future states based on current fused modal data and how to select target control commands to respond to the current cockpit environment state, thereby forming edge weight parameters (e.g., edge weights) based on historical operational data.

[0047] Optionally, after inputting the fused modal data into the graph neural network model, a series of knowledge graph information can be output. Knowledge graph information is a high-level abstraction of various entity information and the mapping relationships between entities in the fused modal data, containing information needed for a deep understanding of the cockpit environment and intelligent decision-making. Nodes in the knowledge graph information can be used to represent entity information, while edges can be used to represent the mapping relationships between entity information.

[0048] For example, if a driver exhibits signs of fatigue (such as an increased heart rate variability) and the outside temperature drops sharply, the graph neural network model can identify a mapping indicating that the driver needs a warmer cabin environment and more rest in this situation. Through this mapping, the graph neural network model can identify the impact of changes in the cabin environment on the driver's state, thus providing a basis for generating various control commands.

[0049] Optionally, based on the knowledge graph information output by the graph neural network model, various control commands can be generated to adjust the cabin environment to suit the driver's current state and needs. These control commands can include adjusting the air conditioning temperature, changing the seat tilt angle, and playing soothing music, thereby improving cabin comfort and safety.

[0050] In this embodiment, by inputting fused modal data into a graph neural network model, and utilizing the graph neural network model and edge weights generated by reinforcement learning, a deep analysis of entity information and mapping relationships in the fused modal data is achieved, thereby generating a series of control commands aimed at optimizing the cockpit environment. This intelligent analysis and decision-making method based on graph neural network models and knowledge graph information provides support for adaptive control of the intelligent cockpit environment, improving vehicle comfort and safety.

[0051] Step S108: In response to the target control command among various control commands, control the cockpit environment.

[0052] In the technical solution of step S108 of the present invention, the priority of the target control command is higher than the priority threshold.

[0053] In this embodiment, after inputting the fused modal data into a graph neural network model for analysis to obtain knowledge graph information, and analyzing the knowledge graph information to obtain multiple control commands, a target control command can be determined from the multiple control commands according to their priority to control the cabin environment. These multiple control commands can include zoned air conditioning adjustment, seat posture correction, emergency warning signals, etc., which are merely illustrative examples and not specifically limited here.

[0054] Optionally, a priority threshold can be set, and the priority threshold can be dynamically adjusted according to the driver's safety level, comfort requirements, and the urgency of the cabin environment to ensure the priority execution of target control commands.

[0055] Optionally, to ensure the rapid transmission and execution of target control commands, an intelligent bus management mechanism can be adopted, such as dynamic topology command distribution, which intelligently adjusts the transmission path and resource allocation of control commands based on real-time monitoring of the vehicle's bus load rate and the priority of various control commands.

[0056] For example, if the cabin environment detects mild signs of heatstroke in the driver, the air conditioning cooling capacity can be automatically increased. This control command can be transmitted via a high-priority bus, such as a Controller Area Network with Flexible Data Rate (CAN-FD) bus, ensuring that the control command reaches the vehicle's air conditioning control unit quickly. If the bus load rate is too high, for example, exceeding the bus load rate trigger threshold, the command distribution strategy can be dynamically adjusted, migrating non-emergency control commands to buses with lower load rates, such as the Local Interconnect Network (LIN) bus. This avoids delays or loss of target control commands, ensuring the timeliness and effectiveness of cabin environment control.

[0057] In steps S102 to S108 of this application, multiple initial modal data from the vehicle are acquired, and spatiotemporal alignment processing is performed on these initial modal data to obtain multiple target modal data, ensuring precise synchronization of the multiple target modal data in time and space. Then, the multiple target modal data are input into a digital twin model for fusion. The digital twin model can reflect various states of the cabin environment in the vehicle in real time and construct a mapping relationship between the fused modal data and the cabin environment. The fused modal data is further input into a graph neural network model for in-depth analysis. Because the graph neural network model learns and trains on a large number of fused modal data samples, it has the ability to understand the complex mapping relationships between multiple entity information in the cabin environment. By analyzing the fused modal data, knowledge graph information can be output, reflecting the mapping relationships between multiple entity information in the fused modal data. Using the knowledge graph information, various control commands can be generated, which can specifically improve the cabin environment. Subsequently, based on priority evaluation, target control commands can be determined to control the cabin environment, thereby solving the technical problem of ineffective control of the cabin in a vehicle and achieving the technical effect of effective control of the cabin in a vehicle.

[0058] The method described in this embodiment will be further described below.

[0059] As an optional embodiment, the multiple initial modal data include at least visual data, biological data, speech data, and environmental data. Step S102 involves performing spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data, including: extracting keypoint data from the visual data, wherein the keypoint data is used to represent the motion trajectory of key points of the target object in the visual data; determining the spectral information corresponding to the biological data, wherein the spectral information is used to represent the spectrum of the biological data; constructing a gradient field model based on the environmental data, and generating environmental distribution information using the gradient field model, wherein the environmental distribution information is used to represent the temperature and humidity distribution of the environmental data; and performing spatiotemporal alignment processing on the keypoint data, spectral information, environmental distribution information, and speech data using a spatiotemporal synchronization bus to obtain multiple target modal data, wherein the spatiotemporal synchronization bus is a hierarchical clock tree architecture.

[0060] In this embodiment, spatiotemporal alignment of multiple initial modal data (i.e., spatiotemporal alignment of multimodal data) can ensure that initial modal data from different sensors can be accurately analyzed and fused within a unified temporal and spatial framework.

[0061] Optionally, keypoint data can be extracted from visual data, such as extracting the motion trajectory of 68 key points on the driver's face from visual data. Spectral information corresponding to the biological data can be determined, for example, generating a heart rate-respiration coupled spectrum from the biological data. Based on environmental data, a gradient field model (e.g., a temperature and humidity gradient field model) can be constructed, and environmental distribution information can be generated using the gradient field model. Then, the above data and information can be fused into a four-dimensional tensor (time × space × modality × eigenvalue) using a cross-modal spatiotemporal encoder and injected into the state matrix.

[0062] Optionally, key point information may include, but is not limited to, facial feature points such as the eyes, nose, mouth, and ears of the target object (e.g., the driver). By tracking and analyzing the motion trajectory of these key point information, the driver's attention status and possible emotional changes can be monitored in real time, such as whether the driver is fatigued or distracted.

[0063] Optionally, spectral information can reflect the driver's physiological state, such as whether they are in a state of tension or relaxation. For example, by monitoring the coefficient of heart rate variability and breathing patterns, the driver's physical and mental health can be assessed, providing data support for adjusting the cabin environment.

[0064] Optionally, environmental distribution information, such as temperature and humidity distribution in different areas of the cabin, can provide more precise environmental control decisions, such as local temperature or humidity regulation.

[0065] Optionally, to ensure temporal alignment and spatial accuracy of multiple initial modal data, a hierarchical clock tree architecture spatiotemporal synchronization bus can be employed. The master clock signal can be generated by a GPS-disciplined clock chip and evenly distributed to each sensor node through a star topology, ensuring the stability and consistency of the clock signal. Under a unified time reference, the motion trajectories of key points in visual data, the spectral information of biological data, the distribution information of environmental data, and the timestamps of speech data can all be aligned through the spatiotemporal synchronization bus, thereby ensuring the consistency of multiple initial modal data on the time axis and the accuracy in spatial coordinates, providing a reliable foundation for subsequent fusion analysis.

[0066] In this embodiment, through the above steps, multiple initial modal data from visual sensors, biosensors, voice sensors, and environmental sensors can be converted into structured multiple target modal data. The multiple target modal data are not only synchronized in time, but also consistent with the actual state of the cabin in space.

[0067] As an optional embodiment, step S104 involves inputting multiple target modal data into a digital twin model for fusion to obtain fused modal data, including: using a cross-modal spatiotemporal encoder to input multiple target modal data into a digital twin model for fusion to obtain fused modal data.

[0068] In this embodiment, a cross-modal spatiotemporal encoder can be used to input multiple target modal data into a digital twin model for fusion to obtain fused modal data.

[0069] Optionally, the cross-modal spatiotemporal encoder can process multi-variable heterogeneous data (multiple target modal data) from different sensors and convert these multiple target modal data into a unified and easy-to-process representation, i.e., fused modal data.

[0070] As an optional implementation, a cross-modal spatiotemporal encoder is used to input multiple target modal data into a digital twin model for fusion to obtain fused modal data. This includes: using a cross-modal spatiotemporal encoder to input multiple target modal data into a digital twin model for fusion to obtain tensor information; and inputting the tensor information into the state matrix of the digital twin model to obtain fused modal data.

[0071] In this embodiment, multiple target modal data are fused into tensor information (time × space × modality × eigenvalue) using a cross-modal spatiotemporal encoder. This tensor information can then be input into the state matrix of the digital twin model to obtain the fused modal data. The following describes the dynamic construction and application process of the digital twin.

[0072] Optionally, to ensure the accuracy and robustness of multi-target modal data fusion, the cross-modal feature fusion weight calculation formula can be defined as follows:

[0073]

[0074] in, Can be used to represent time Spatial location Modal fusion weights; Can be used to represent time Spatial location Modal information entropy is used to measure the amount of effective information in the data of that modality; Can be used to represent time Spatial location The correlation coefficient between a mode and other modes is used to reflect the degree of cooperation between modes; , It can be used to represent weighting coefficients, and .

[0075] Optionally, the purpose of the above formula is to dynamically allocate fusion weights based on the information value and synergy of different modal data, avoiding the impact of single-modal data bias on the fusion result. The fusion weights calculated by the above formula enable tensor information (e.g., a four-dimensional tensor) to more accurately reflect the overall state of the cockpit environment. By updating multiple initial modal data and dynamically correcting the state matrix, the consistency between the virtual cockpit state of the digital twin and the physical (real) cockpit state can be ensured.

[0076] Optionally, the state matrix It is a digital mapping carrier of the cockpit's physical state, which can be used to integrate spatiotemporally aligned data from multiple target modalities and serve as input to graph neural network models. It can be used to represent the length of the sampling sequence in the time dimension, and its value is determined by the sampling frequency of each sensor to ensure coverage of the dynamic changes in the cockpit status; It can be used to represent the spatial dimension of the cabin, corresponding to the spatial layout of actuators such as zoned air conditioning and seats, and can be refined into spatial coordinate nodes of different physical areas within the cabin; It can be used to represent the data source type of modality dimension, such as including three core modalities: visual, biological, and environmental, which is consistent with the modality type in the cross-modal compensation mechanism; The number of parameters can be used to represent the feature dimension, corresponding to the key features extracted under each modality. The number of parameters can be determined by the accuracy requirements of the single-modality feature extraction algorithm.

[0077] Optionally, matrix elements It can then be used to represent in time, Spatial location, Modal 1 The specific values ​​of each feature can be generated through fusion using a cross-modal spatiotemporal encoder. For valid modal data, data after step calibration can be used. Specifically, the payload area stores compressed multimodal data streams. The compression process is based on data redundancy analysis, reducing transmission bandwidth usage while ensuring that valid data information is not lost.

[0078] Alternatively, failure mode data can be calculated using a cross-modal compensation mechanism.

[0079] In this embodiment, the problem of fusing multiple target modal data is effectively solved by using a cross-modal spatiotemporal encoder, generating fused modal data, which provides a more complete and detailed description of the environmental state for subsequent decision-making stages.

[0080] As an optional embodiment, the method further includes: performing dual error correction processing on multiple target modal data to obtain the bit error rate of the dual error correction processing multiple target modal data, wherein the bit error rate is less than the bit error rate threshold.

[0081] In this embodiment, since the control of the cockpit environment relies on high-precision sensor data, errors in the data transmission process can lead to deviations in control decisions and even endanger the driver's safety. Before fusing multiple target modal data, dual error correction processing can be performed on the multiple target modal data to reduce the bit error rate of data transmission and ensure that the bit error rate meets the set bit error rate threshold.

[0082] Optionally, the quality verification parameters of various target modal data can be controlled by the overall bit error rate P through a dual error correction mechanism of 32-bit polynomial cyclic redundancy check (CRC-32) and Hamming code, as well as the compressed data stream obtained based on data redundancy analysis, to ensure that effective information is not lost when extracting features.

[0083] Optionally, the check area can employ a dual error correction mechanism with CRC-32 and Hamming codes, with an error rate of 100%. The data frame length is The error correction capability satisfies the following formula:

[0084]

[0085] in, It can be used to represent the length of a data frame, indicating the total number of bits in a data frame; It can be used to represent the bit error rate per bit; It can be used to represent the upper limit of acceptable bit error rate. This formula quantifies the impact of single-bit bit error rate and data frame length on the overall bit error rate, which can ensure that the bit error rate of multiple target mode data transmission is controlled at an extremely low level in the vehicle electromagnetic interference environment, thus ensuring data integrity.

[0086] In this embodiment, by performing a dual error correction mechanism before fusing multiple target modal data, the reliability of data transmission of multiple target modal data is effectively enhanced, the risk of cabin environment control is reduced, and the decisions made are made to reflect the real environmental state of the cabin to the greatest extent, thereby improving the riding experience and safety.

[0087] As an optional embodiment, step S106 involves inputting the fused modal data into a graph neural network model for analysis to obtain knowledge graph information, and analyzing the knowledge graph information to obtain various control commands, including: inputting the fused modal data into the graph neural network model; using the fused modal data to update the edge weight parameters of the graph neural network model to obtain knowledge graph information; determining the size relationship between various entity information and entity information thresholds in the knowledge graph information; and generating control commands according to the size relationship.

[0088] In this embodiment, the fused modal data is input into a graph neural network model for analysis. This not only generates knowledge graph information but also dynamically updates the edge weight parameters (edge ​​weights) of the graph neural network model to more accurately reflect entity information and the relationships between entities in the cockpit environment. Based on this relationship analysis, various control commands are generated. The following is the process of collaborative decision-making based on the knowledge graph.

[0089] Optionally, the state matrix is ​​input into a pre-trained graph neural network model. The nodes of the graph neural network model can contain entity information such as the driver's physiological state, vehicle motion parameters, and environmental comfort range. The edge weights are generated by reinforcement learning from historical operation data.

[0090] Optionally, to accurately achieve the activation determination and control command output optimization of the emergency comfort subgraph, two correlation calculation models can be constructed to adapt to the dynamic interaction logic of multi-dimensional entity information in the state matrix. The first core calculation can be the dynamic update formula for edge weights of the graph neural network model, used to optimize the correlation strength between entity information nodes such as driver physiological state, vehicle motion parameters, and environmental comfort range in real time. The dynamic update of edge weights can be described by the following formula:

[0091]

[0092] in, It can be used to represent entity nodes at time t. (e.g., "heart rate variability coefficient") and nodes (e.g., edge weights between "emergency comfort subgraph activation thresholds"); Can be used to represent Historical edge weights at each moment ensure the continuity of weight updates; It can be used to represent the learning rate, and its value is dynamically adjusted by the convergence speed of historical operation data to avoid overshooting of weight updates or slow convergence. It can be used to represent the feedback reward value after the command is executed at time t. For example, if the heart rate variability coefficient falls back to the normal range after the "zone climate control pressurization and air supply + seat side wing tightening" control command is executed, it is assigned a positive reward; otherwise, it is assigned a negative reward. Can be used to represent Time Node and The expected associated revenue is calculated from the average revenue of similar historical scenarios in the cockpit environment knowledge graph; This can be used to represent the partial derivative of the loss function L with respect to historical edge weights. The loss function L can be defined as the Euclidean distance between the current digital twin state and the comfort baseline state. The partial derivative reflects the contribution of weight adjustments to reducing state bias. The purpose of this formula is to allow edge weights to dynamically iterate with real-time scenario feedback and historical experience, making the relationships between nodes more aligned with actual decision-making needs. For example, when multiple instances of "increased heart rate variability + decreased grip strength" are detected, the edge weights of the "physiological state node" and the "emergency comfort subgraph node" will continuously increase, improving the accuracy of subgraph activation.

[0093] Optionally, the second core calculation can be an emergency comfort subgraph activation comprehensive decision formula, used to quantify the impact of multi-entity state deviations on activation decisions, which can be described by the following formula:

[0094]

[0095] Optionally, Γ can be used to represent the comprehensive judgment value. When Γ exceeds the preset activation threshold, the "emergency comfort" subgraph is triggered. n can be used to represent the number of key entity features involved in the judgment. It can include four core features: heart rate variability coefficient, steering wheel grip force, vehicle yaw rate, and cabin temperature deviation. The weight coefficients, which can be used to represent the k-th feature, are obtained by normalizing the edge weights of the graph neural network. For example, the "heart rate variability coefficient"... The temperature deviation is higher than the "cabin temperature deviation," reflecting the principle of prioritizing physiological safety. It can be used to represent the real-time value of the k-th feature at time t, and can be taken from the corresponding entity node data in the twin state matrix; The comfort baseline value that can be used to represent the k-th feature can be determined by the historical average of normal states of drivers of the same type in a cloud-based knowledge graph; The relative deviation rate of the k-th feature can be used to quantify the degree of deviation between the current state and the baseline state; This can be used to represent the deviation attenuation coefficient of the k-th feature. If the duration of the feature deviation exceeds a set threshold, It can increase linearly, avoiding accidental triggering of subgraphs due to instantaneous fluctuations; It can be used to represent the validity coefficient of the k-th feature, when the reliability of sensor data is lower than the set standard (such as when a biosensor is subject to electromagnetic interference). The reduced value weakens the impact of unreliable data on the judgment result. This formula, through weighted fusion of multi-dimensional features, avoids erroneous decisions caused by single feature bias, and accurately captures scenarios of multi-feature synergistic anomalies such as "increased heart rate variability + decreased grip strength", ensuring the accuracy of the activation timing and command output of the emergency comfort subgraph, and generating control commands such as zoned air conditioning and seat adjustment.

[0096] Optionally, the state matrix can provide full-dimensional entity data input for the graph neural network. The nodes of the graph neural network can contain entity information such as "driver's physiological state, vehicle motion parameters, and environmental comfort range," and the real-time data for this entity information can be directly derived from the state matrix. For example, the "heart rate variability coefficient and steering wheel grip strength" data required by the "driver's physiological state" node correspond to the data in the state matrix. The eigenvalues; the "cabin temperature and humidity deviation" data required for the "environmental comfort zone" node, corresponding to the matrix in... The eigenvalues; data such as "yaw rate" required by the "vehicle motion parameters" node are also associated with the real-time sampling sequence through the spatial dimension S and the time dimension T of the matrix.

[0097] Optionally, the state matrix can ensure the accuracy of edge weight updates and subgraph activation. The "instruction execution feedback reward value R(t) at time t" required in the edge weight update formula needs to be retrieved in real-time from the matrix and compared with historical comfort baseline data. The "real-time feature value" in the emergency comfort subgraph activation formula... (e.g., the coefficient of variation of heart rate at time t, cabin temperature), which are also directly taken from the elements of the corresponding dimension of the matrix. This matrix transforms multimodal data into a structured form of "time × space × modality × feature value," enabling graph neural networks to quickly locate and access real-time data of each entity. This ensures the dynamic nature of edge weight updates and the accuracy of subgraph activation determination, avoiding decision delays or misjudgments caused by inconsistent data formats.

[0098] In this embodiment, by fusing knowledge graph information from modal data and dynamically updating the edge weights of the network, control commands are generated in a more intelligent and precise manner. This not only improves the response speed of cockpit environment control but also enhances its adaptive and learning capabilities, realizing intelligent cockpit environment adaptive control based on multimodal interaction.

[0099] As an optional embodiment, step S108 involves controlling the cabin environment in response to a target control command among multiple control commands, including: determining the vehicle's bus load rate trigger threshold; sorting the priorities of multiple control commands based on the load rate trigger threshold and the vehicle's driving scenario to obtain a sorting result; determining the control command with a priority greater than the priority threshold in the sorting result as the target control command; and controlling the cabin environment according to the target control command.

[0100] In this embodiment, three types of core parameters can be extracted from the graph neural network model: entity edge weights, emergency subgraph activation states, and historical feedback reward values.

[0101] Optionally, the entity edge weights may include the real-time edge weights of CAN-FD bus load capacity node A and command priority node B. And the real-time edge weights of CAN-FD bus load capacity node A and driver demand urgency node C. The activation status of the emergency subgraph can include the activation criteria for the emergency comfort subgraph. and corresponding thresholds Historical feedback reward values ​​can include the feedback reward sequence following recent instruction executions. The above parameters can be transmitted in real time via the internal data bus of the edge computing layer.

[0102] Optionally, after extracting the three types of core parameters, the bus load rate trigger threshold of the vehicle can be determined. That is, calculating the dynamic load factor threshold. . It can be determined using the following formula:

[0103]

[0104] in, It can be used to represent the CAN-FD bus load rate trigger threshold in the current scenario. When the actual load rate of the CAN-FD bus exceeds the bus load rate trigger threshold, it can trigger the control command migration to the LIN bus and other related control command distribution strategy adjustment operations. It can be used to represent dynamic baseline thresholds. Based on various parameters such as driving scenarios (e.g., current scenario type, such as the vehicle being at high speed or in congested traffic), historical bus load data, and control command type distribution, the baseline load rate is calculated in real time through a graph neural network to adapt to the reasonable baseline requirements of bus load under different driving scenarios. It can be used to represent the real-time edge weights of "CAN-FD bus load capacity node A" and "instruction priority node B" in a graph neural network, reflecting the change in the correlation strength between the two entities "CAN-FD bus load capacity" and "instruction priority" over time t. This can be used to represent the real-time edge weights between "CAN-FD bus load capacity node A" and "driver demand urgency node C" in a graph neural network, reflecting the state of the correlation strength between the two entities, "CAN-FD bus load capacity" and "driver demand urgency," at time t. α and β can be used to represent weighting coefficients, satisfying α + β = 1, for weighing the relationships. and For the final trigger threshold The extent of the impact The value can be determined by training with historical scene data from the knowledge graph information.

[0105] Optionally, when determining the vehicle's bus load rate trigger threshold, the LIN bus idle threshold can be calculated simultaneously. This serves as a condition for determining the receiving end of the control command migration.

[0106] Optionally, the received emergency subgraph activation state can be subject to the following priority determination. If Γ≥ (In emergency scenarios) Commands related to safety, such as "zone climate control adjustment," "seat posture correction," and "seatbelt pretensioning," can be automatically marked as "highest priority P0." These control commands are forcibly transmitted via the CAN-FD bus and are not subject to load rate threshold limitations. If Γ < (In typical scenarios) rewards can be based on historical feedback values. Calculate the priority coefficients for various control orders, whereby the priority coefficients can be determined using the following formula:

[0107]

[0108] in, It can be used to represent the weight of control instruction types, according to Control commands can be divided into "high priority P1" and "low priority P2".

[0109] Optionally, dynamic topology instruction distribution can be performed to monitor the current load rate L(t) of the CAN-FD bus in real time. When At this time, all P0, P1, and P2 level control commands can be transmitted via the CAN-FD bus, recording current transmission efficiency parameters (delay, packet loss rate) and providing feedback. And LIN bus idle time When necessary, P2-level control commands can be migrated to the LIN bus, while P0 and P1-level control commands are retained for transmission on the CAN-FD bus. An integrity verification mechanism (i.e., a dual error correction mechanism) is then activated for the migrated control commands. And LIN bus idle time < At this time, the graph neural network is triggered to make an emergency adjustment, recalculate the edge weights, and feed back the new weights. value.

[0110] Optionally, three types of key data can be collected after control commands are distributed: bus performance data, actuator response data, and driver feedback data. Bus performance data may include the actual load rate, transmission delay, and bit error rate of the CAN-FD / LIN bus. Actuator response data may include the execution completion time of each control command and the deviation from the expected effect. Driver feedback data may be changes in comfort indirectly obtained through biosensors (e.g., heart rate variability recovery rate).

[0111] Optionally, the formatted key data can be input into a graph neural network to update the edge weights between entity information nodes and correct the activation formula of the emergency comfort subgraph. and Coefficients, optimize the weighting coefficients of α and β.

[0112] In this embodiment of the invention, multiple initial modal data from the vehicle are acquired and spatiotemporally aligned to obtain multiple target modal data, ensuring precise synchronization of these target modal data in time and space. Then, the multiple target modal data are input into a digital twin model for fusion. The digital twin model can reflect various states of the vehicle's cabin environment in real time and construct a mapping relationship between the fused modal data and the vehicle's cabin environment. The fused modal data is further input into a graph neural network model for in-depth analysis. Because the graph neural network model learns and trains on a large number of fused modal data samples, it possesses the ability to understand the complex mapping relationships between multiple entity information in the cabin environment. By analyzing the fused modal data, knowledge graph information can be output, reflecting the mapping relationships between multiple entity information in the fused modal data. Using the knowledge graph information, multiple control commands can be generated to specifically improve the cabin environment. Then, based on priority evaluation, target control commands can be determined to control the cabin environment, thereby solving the technical problem of ineffective control of the vehicle cabin and achieving the technical effect of effective control of the vehicle cabin.

[0113] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0114] Currently, vehicle cabin environment control methods mostly employ single-modal data acquisition (such as relying solely on ambient temperature and humidity sensors) or simple multimodal data overlay schemes. Data transmission primarily relies on fixed sampling triggering methods and basic verification mechanisms, lacking a unified spatiotemporal synchronization architecture and failing to establish a deep mapping relationship between the physical cabin and the digital model. Decision-making processes largely depend on preset fixed rules (such as setting a fixed temperature threshold to trigger air conditioning adjustment), failing to incorporate multi-dimensional entities such as the physiological state of the target object (e.g., the driver) and vehicle motion parameters to construct collaborative decision-making logic. Command distribution is based on fixed bus load thresholds (e.g., migrating commands when the CAN-FD bus load rate exceeds a fixed value), lacking the ability for scenario-based dynamic adjustment.

[0115] In summary, the relevant technologies suffer from several drawbacks. Multimodal data is prone to distortion due to inconsistencies in spatiotemporal references and lack of quantitative control over transmission error rates, leading to subsequent decisions deviating from actual needs. Furthermore, the large discrepancy between the physical cockpit status and the digital model makes it impossible to provide accurate structured data support for decision-making.

[0116] This invention proposes an intelligent automotive cockpit environment adaptive control system supporting multimodal interaction to address the following problems in vehicle cockpit environment control: First, data distortion caused by inconsistencies in spatiotemporal references and transmission interference in multimodal data (visual, biological, environmental, etc.). This is addressed by using GPS-disciplined clock synchronization, a dual error correction mechanism, and a bit error rate control formula to ensure the integrity and accuracy of data from acquisition to transmission. Second, the problem of large deviations between the physical cockpit state and the digital model is addressed by using dynamic digital twin construction technology, multimodal feature extraction, cross-modal fusion weight formulas, and a digital twin state matrix. Dynamic correction enables precise synchronization between the physical cockpit and the digital twin. Thirdly, addressing the lack of multi-dimensional entity collaboration logic in cockpit decision-making and inaccurate emergency response, a graph neural network edge weight dynamic update model and an emergency comfort subgraph activation determination model are constructed to adapt to multi-dimensional entity data interaction logic, ensuring accurate subgraph activation and optimized command output in emergency scenarios. Fourthly, to address resource waste or high-priority command delays caused by fixed bus load thresholds in command distribution, dynamic load rate thresholds are calculated using decision parameters to achieve dynamic command priority determination and dynamic bus topology distribution, improving command execution efficiency and system adaptability.

[0117] The embodiments of the present invention will be further described below.

[0118] Figure 2 This is a schematic diagram of an intelligent automotive cockpit environment adaptive control system supporting multimodal interaction according to an embodiment of the present invention, as shown below. Figure 2 As shown, the system may include: a hardware execution layer 201, an edge computing layer 202, and a cloud optimization layer 203.

[0119] The hardware execution layer 201 includes a zoned air conditioning controller, a multi-degree-of-freedom seat motor, a LIN bus, etc. Each actuator can be interconnected via a CAN-FD bus and supports millisecond-level command response.

[0120] Edge computing layer 202 is deployed on the vehicle domain controller and integrates a multi-source heterogeneous data processing unit, a digital twin mapping engine, and a real-time decision-making module.

[0121] The cloud optimization layer 203 is used to build a cockpit environment knowledge graph and incremental learning platform. It can communicate bidirectionally with the edge computing layer through the 5th Generation Vehicle-to-Everything (5G-V2X) module.

[0122] Figure 3 This is a flowchart of a multimodal data transmission verification method according to an embodiment of the present invention, such as... Figure 3 As shown, the method includes the following steps.

[0123] Step S301: Send the high-stability master clock signal to the sensor node.

[0124] In this embodiment, a highly stable master clock signal can be sent to the sensor node.

[0125] Step S302: Perform GPS taming clock synchronization.

[0126] In this embodiment, the spatiotemporal synchronization bus adopts a hierarchical clock tree architecture. A highly stable master clock signal is generated by a GPS disciplined clock chip and distributed to each sensor node through a star topology. This architecture can effectively reduce the attenuation and interference of the clock signal during transmission and ensure the consistency of the clock reference of each node.

[0127] Step S303: Define a three-level data frame.

[0128] In this embodiment, the sampling triggering mechanism uses rising edge triggering for visual sensors and dual-edge triggering for biosensors. The choice between the two triggering methods depends on the data characteristics of the two types of sensors. Visual data needs to avoid signal overlap within adjacent sampling periods, while biosensors require higher time density to capture subtle changes in physiological signals. Simultaneously, phase-locked loop technology is used to eliminate sampling time jitter, further improving the accuracy of sampling timing. The bus transmission protocol defines a three-level data frame structure. The frame header contains a sensor identifier (ID) and an absolute timestamp, where the absolute timestamp is directly associated with the master clock signal, ensuring the accuracy of data traceability. The payload area stores the compressed multimodal data stream, i.e., the compressed data stream. The compression processing is based on data redundancy analysis, reducing transmission bandwidth usage while ensuring no loss of valid data information. The verification area can employ a dual error correction mechanism with CRC-32 and Hamming codes.

[0129] Step S304: Transmission via CAN-FD bus.

[0130] In this embodiment, the three-level data frames can be transmitted via the CAN-FD bus.

[0131] Step S305: Perform a data integrity check.

[0132] In this embodiment, data integrity can be determined. If the data is invalid, step 306 is executed; if the data is valid, step 307 is executed.

[0133] Step 306: Invalid data, triggering the compensation mechanism.

[0134] In this embodiment, the failure mode data is calculated through a cross-modal compensation mechanism.

[0135] Step 307, valid data, input to the edge computing layer.

[0136] In this embodiment, valid data can be input into the edge computing layer, and the calibrated data can be used directly.

[0137] Figure 4 This is a flowchart of a digital twin construction method according to an embodiment of the present invention, such as... Figure 4 As shown, the method includes the following steps.

[0138] Step S401: Input multimodal data.

[0139] In this embodiment, multimodal data (e.g., multiple target modal data) is input into the digital twin (digital twin model).

[0140] Step S402: Process visual data.

[0141] In this embodiment, visual data can be extracted.

[0142] Step S403: Process biological data.

[0143] In this embodiment, biological data can be processed to generate a heart rate-respiration coupled spectrum.

[0144] Step S404: Process environmental data.

[0145] In this embodiment, environmental data can be processed to construct a temperature and humidity gradient field model.

[0146] Step S405: Extract facial key point trajectories.

[0147] In this embodiment, a digital twin of the cockpit environment can be created at the edge computing layer to map the physical cockpit state in real time. Motion trajectories of 68 key facial points are extracted from visual data.

[0148] Step S406: Generate the heart rhythm and respiratory coupling frequency.

[0149] In this embodiment, a heart rate-respiration coupled spectrum can be generated from biological data.

[0150] Step S407: Construct a temperature and humidity gradient field model.

[0151] In this embodiment, a temperature and humidity gradient field model can be constructed from environmental data.

[0152] Step S408: Generate a four-dimensional tensor.

[0153] In this embodiment, the above features are fused into a four-dimensional tensor (time × space × mode × feature value) by a cross-modal spatiotemporal encoder.

[0154] Step S409: Inject the twin state matrix.

[0155] In this embodiment, the above features are fused into a four-dimensional tensor (time × space × mode × eigenvalue) by a cross-modal spatiotemporal encoder and injected into the twin state matrix.

[0156] Step S410: Receive sensor status feedback.

[0157] In this embodiment, sensor status feedback can be received in real time.

[0158] Step S411: Dynamically correct the twin.

[0159] In this embodiment, the digital twin mapping engine also needs to receive state update data from the hardware execution layer sensors in real time. After that, the twin state matrix can be dynamically corrected to ensure the consistency between the twin and the physical cockpit state.

[0160] Step S412: Output structured state data.

[0161] In this embodiment, structured state data can be output.

[0162] It should be noted that the above-mentioned multimodal data spatiotemporal alignment, dynamic construction of digital twins, knowledge graph-based collaborative decision-making, and dynamic topology instruction distribution have been described in the preceding sections and will not be repeated here.

[0163] According to embodiments of the present invention, a control device for a vehicle's passenger compartment is also provided. It should be noted that this control device for a vehicle's passenger compartment can be used to execute the control method for the vehicle's passenger compartment described in the embodiments.

[0164] Figure 5 This is a schematic diagram of a control device for a vehicle cabin according to an embodiment of the present invention. Figure 5 As shown, the control device 500 in the vehicle's cockpit may include: an acquisition unit 502, a fusion unit 504, a determination unit 506, and a control unit 508.

[0165] The acquisition unit 502 is used to acquire multiple initial modal data in the vehicle and perform spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data. The initial modal data is used to represent the perception data of the vehicle during driving obtained by the sensors. Different initial modal data correspond to different sensors, and the multiple target modal data correspond one-to-one with the multiple initial modal data.

[0166] The fusion unit 504 is used to input multiple target modal data into the digital twin model for fusion to obtain fused modal data. The digital twin model is used to construct the mapping relationship between the fused modal data and the cabin environment in the vehicle.

[0167] The determining unit 506 is used to input the fused modal data into the graph neural network model for analysis to obtain knowledge graph information, and to analyze the knowledge graph information to obtain various control commands. The graph neural network model is obtained by training the neural network model using the fused modal data samples corresponding to the fused modal data. The knowledge graph information is used to represent the mapping relationship between various entity information in the fused modal data, and the control commands are used to control the vehicle's cabin environment.

[0168] Control unit 508 is used to control the cockpit environment in response to a target control command among a variety of control commands, wherein the target control command has a higher priority than a priority threshold.

[0169] In this embodiment of the invention, the acquisition unit 502 acquires multiple initial modal data from the vehicle and performs spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data. The initial modal data represents the perception data of the vehicle during driving obtained by sensors, and different initial modal data correspond to different sensors. The multiple target modal data and the multiple initial modal data are in one-to-one correspondence. The fusion unit 504 inputs the multiple target modal data into a digital twin model for fusion to obtain fused modal data. The digital twin model is used to construct the mapping relationship between the fused modal data and the cabin environment in the vehicle. The determination unit 506 inputs the fused modal data into a graph neural network. The system analyzes the network model to obtain knowledge graph information, and analyzes the knowledge graph information to obtain various control commands. The graph neural network model is trained using fusion modal data samples corresponding to the fusion modal data. The knowledge graph information is used to represent the mapping relationship between various entity information in the fusion modal data. The control commands are used to control the vehicle's cabin environment. The control unit 508 responds to the target control command among the various control commands to control the cabin environment. The target control command has a higher priority than the priority threshold, thus solving the technical problem of ineffective control of the vehicle cabin and achieving the technical effect of effective control of the vehicle cabin.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0171] According to another aspect of the present invention, a processor is also provided. The processor is used to run a program, wherein the program executes the methods of the embodiments of the present invention during runtime.

[0172] According to another aspect of the present invention, an electronic device is also provided, wherein a processor is configured to run a program, wherein the program executes the method of the present invention during runtime.

[0173] According to another aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method of the embodiments of the present invention.

[0174] According to another aspect of the present invention, a vehicle is also provided. The vehicle includes a memory and a processor. The memory stores an executable program; the processor runs the program, which, when executed, implements the methods described in the embodiments of the present invention.

[0175] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0179] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0180] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for controlling a vehicle's cockpit, characterized in that, include: Multiple initial modal data in the vehicle are acquired, and the multiple initial modal data are spatiotemporally aligned to obtain multiple target modal data. The initial modal data is used to represent the perception data of the vehicle during driving obtained by the sensors. Different initial modal data correspond to different sensors, and the multiple target modal data correspond one-to-one with the multiple initial modal data. Multiple target modal data are input into a digital twin model for fusion to obtain fused modal data, wherein the digital twin model is used to construct a mapping relationship between the fused modal data and the cabin environment in the vehicle; The fused modal data is input into a graph neural network model for analysis to obtain knowledge graph information, and the knowledge graph information is analyzed to obtain various control commands. The graph neural network model is trained using fused modal data samples corresponding to the fused modal data. The knowledge graph information is used to represent the mapping relationship between various entity information in the fused modal data, and the control commands are used to control the cabin environment of the vehicle. The cockpit environment is controlled in response to a target control command among a plurality of control commands, wherein the target control command has a higher priority than a priority threshold.

2. The method according to claim 1, characterized in that, The initial modal data includes at least visual data, biological data, speech data, and environmental data. Spatiotemporal alignment processing is performed on the initial modal data to obtain multiple target modal data, including: The visual data is extracted to obtain key point data, wherein the key point data is used to represent the motion trajectory of key points of the target object in the visual data; Determine the spectral information corresponding to the biological data, wherein the spectral information is used to represent the spectrum of the biological data; Based on the environmental data, a gradient field model is constructed, and environmental distribution information is generated using the gradient field model, wherein the environmental distribution information is used to represent the temperature distribution and humidity distribution of the environmental data; The key point data, the spectrum information, the environmental distribution information, and the voice data are spatiotemporally aligned using a spatiotemporal synchronization bus to obtain multiple target modal data, wherein the spatiotemporal synchronization bus is a hierarchical clock tree architecture.

3. The method according to claim 2, characterized in that, Multiple target modal data are input into a digital twin model for fusion to obtain fused modal data, including: Using a cross-modal spatiotemporal encoder, multiple target modal data are input into the digital twin model for fusion to obtain the fused modal data.

4. The method according to claim 3, characterized in that, Using a cross-modal spatiotemporal encoder, multiple target modal data are input into the digital twin model for fusion to obtain the fused modal data, including: Using the cross-modal spatiotemporal encoder, multiple target modal data are input into the digital twin model for fusion to obtain tensor information; The tensor information is input into the state matrix of the digital twin model to obtain the fused modal data.

5. The method according to claim 4, characterized in that, The method further includes: Double error correction processing is performed on multiple target modal data to obtain the bit error rate of the multiple target modal data with double error correction processing, wherein the bit error rate is less than the bit error rate threshold.

6. The method according to claim 5, characterized in that, The fused modal data is input into a graph neural network model for analysis to obtain knowledge graph information. The knowledge graph information is then analyzed to obtain various control commands, including: The fused modal data is input into the graph neural network model; Using the fused modal data, the edge weight parameters of the graph neural network model are updated to obtain the knowledge graph information; Determine the magnitude relationship between the various entity information and entity information thresholds in the knowledge graph information; The control instructions are generated according to the stated size relationship.

7. The method according to claim 6, characterized in that, Controlling the cockpit environment in response to a target control command among a plurality of said control commands, including: Determine the bus load rate trigger threshold for the vehicle; Based on the load rate trigger threshold and the vehicle's driving scenario, the priorities of various control commands are sorted to obtain a sorting result; The control instructions whose priority is greater than the priority threshold in the sorting results are determined as the target control instructions; The cockpit environment is controlled according to the target control command.

8. A control device for a vehicle's passenger compartment, characterized in that, include: The acquisition unit is used to acquire multiple initial modal data in the vehicle and perform spatiotemporal alignment processing on the multiple initial modal data to obtain multiple target modal data. The initial modal data is used to represent the perception data of the vehicle during driving obtained by the sensor. Different initial modal data correspond to different sensors. The multiple target modal data correspond one-to-one with the multiple initial modal data. The fusion unit is used to input multiple target modal data into a digital twin model for fusion to obtain fused modal data, wherein the digital twin model is used to construct a mapping relationship between the fused modal data and the cabin environment in the vehicle; The determining unit is used to input the fused modal data into a graph neural network model for analysis to obtain knowledge graph information, and to analyze the knowledge graph information to obtain various control commands. The graph neural network model is obtained by training the neural network model using fused modal data samples corresponding to the fused modal data. The knowledge graph information is used to represent the mapping relationship between various entity information in the fused modal data. The control commands are used to control the cabin environment of the vehicle. A control unit is configured to control the cockpit environment in response to a target control command among a plurality of control commands, wherein the target control command has a higher priority than a priority threshold.

9. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 7 when it runs.

10. An electronic device, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 7 when it runs.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

12. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 7.

Images