Multi-dimensional safety sensing method and system of multi-modal vehicle information physical system

Abstract

The invention provides a multi-dimensional safety perception method and system for a multi-modal vehicle information physical system, and the method comprises the steps: mapping a multi-dimensional safety demand in the vehicle information physical system to multi-modal information source data, and building a mapping relation; according to the mapping relationship, querying and collecting multi-modal information source data corresponding to the user security demand; inputting the collected multi-modal information source data into a multi-modal fusion perception model to obtain a security perception result output by the multi-modal fusion perception model, the multi-modal fusion perception model being obtained through unsupervised learning; the safety perception result is input into a risk judgment module to obtain a risk score output by the risk judgment module, the risk score is input into a risk response module based on a strategy network to generate an optimal risk response strategy, and the risk response module is obtained through reinforcement learning. The comprehensive safety guarantee capability of the vehicle in a complex environment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle safety technology, and in particular to a multi-modal vehicle information physical system multi-dimensional safety perception method and system. BACKGROUND

[0002] With the rapid development of intelligent networked automobile technology, the automobile system is gradually evolving into a highly complex cyber-physical system (CPS), and its wide application in the fields of traffic safety, automatic driving and vehicle-road cooperation puts forward higher requirements for the multi-dimensional safety of the system.

[0003] Cyber-physical systems integrate information space and physical space, embodying the deep integration of control systems and computing systems. In the current field of intelligent vehicles, the safety system mainly includes automobile functional safety, automobile information safety and automobile expected functional safety. To some extent, the three are mutually intersected and independent of each other, but in terms of multi-dimensional safety perception, risk linkage analysis and multi-modal information fusion, the existing technology still lacks systematic solutions, making it difficult to comprehensively cope with the safety challenges faced by intelligent networked vehicles. SUMMARY

[0004] The present application provides a multi-modal vehicle information physical system multi-dimensional safety perception method and system to solve the defects in the prior art that it is difficult to comprehensively cope with the safety challenges faced by intelligent networked vehicles, and to realize multi-modal, multi-dimensional safety perception of intelligent networked vehicles from the overall perspective of vehicle information physical system safety, in order to improve the comprehensive safety protection capability of vehicles in complex environments.

[0005] The present application provides a multi-modal vehicle information physical system multi-dimensional safety perception method, comprising:

[0006] Mapping the multi-dimensional safety requirements in the vehicle information physical system to the multi-modal information source data, and establishing a mapping relationship;

[0007] According to the mapping relationship, querying and collecting multi-modal information source data corresponding to user safety requirements;

[0008] Inputting the collected multi-modal information source data into a multi-modal fusion perception model to obtain a safety perception result output by the multi-modal fusion perception model, the multi-modal fusion perception model being obtained through unsupervised learning;

[0009] Inputting the safety perception result into a risk judgment module to obtain a risk score output by the risk judgment module, and inputting the risk score into a risk response module based on a policy network to generate an optimal risk response strategy, the risk response module being obtained through reinforcement learning.

[0010] The application provides a multi-modal vehicle information physical system multi-dimensional safety perception method.

[0011] The application provides a multi-modal vehicle information physical system multi-dimensional safety perception method, wherein the multi-modal information source data comprises multiple data in a physical layer data, an information layer data, a function layer data and a human-computer interaction layer data.

[0012] The application provides a multi-modal vehicle information physical system multi-dimensional safety perception method, wherein the multi-modal fusion perception model comprises an embedding layer, an encoding layer and an output layer connected in sequence.

[0013] The application provides a multi-modal vehicle information physical system multi-dimensional safety perception method, wherein the multi-modal fusion perception model is unsupervised learning through the following loss function:

[0014] L = λ1·L recon + λ2·L align + λ3·L contrast

[0015] Wherein, L is a loss function, L recon , L align and L contrast respectively represent the auto-encoding reconstruction loss of the multi-modal information source data, the feature alignment loss between the multi-modal information source data and the contrastive perception loss between the multi-modal information source data, λ1, λ2 and λ3 are corresponding weight coefficients.

[0016] The application provides a multi-modal vehicle information physical system multi-dimensional safety perception method, wherein the auto-encoding reconstruction loss L recon of the multi-modal information source data is calculated according to the following formula:

[0017]

[0018] Wherein, x i represents the information source data of the i-th mode, represents the output of the i-th mode information source data after being reconstructed by the encoding layer, and N is the number of modes.

[0019] The feature alignment loss L align between the multi-modal information source data is calculated according to the following formula:

[0020]

[0021] Wherein, f i , f j respectively represent the shared representation vectors output after the i-th and j-th mode information source data pass through the embedding layer.

[0022] Contrastive perception loss L between multi-modal information source data contrast The calculation formula is:

[0023]

[0024] Wherein, z i represents the output of the multi-modal information source data of the i th sample after the embedding layer, represents the similar sample of z i , sim(·) represents the vector similarity function, τ represents the temperature coefficient, and M represents the number of samples used by the unsupervised learning.

[0025] According to the multi-modal vehicle cyber-physical system multi-dimensional safety perception method provided by the application, the risk response module is reinforced learning through the following reward function:

[0026] r t int = alpha * r t rep + beta * r t anomaly + gamma * r t info

[0027] Wherein, r t int represents the final reward function, r t rep represents the error driven reward, r t anomaly represents the anomaly monitoring driven reward, r t info represents the state confidence difference reward, alpha, beta and gamma represent the corresponding weight coefficients respectively.

[0028] According to the multi-modal vehicle cyber-physical system multi-dimensional safety perception method provided by the application, the error driven reward r t rep The calculation formula is:

[0029]

[0030] Wherein, f θ (·) represents the representation of the current state s t in the risk response module by the encoder, represents the reconstruction result of the current state s t in the risk response module by the prediction module;

[0031] The anomaly monitoring driven reward r t anomalyThe calculation formula is:

[0032]

[0033] Where, x t This represents the input multimodal information source data. This represents the reconstruction result of the prediction module in the risk response module on the multimodal information source data, where ζ is the anomaly threshold and σ(·) is the ReLU activation function;

[0034] State confidence difference reward r t info The calculation formula is:

[0035]

[0036] Among them, D KL Denotes KL divergence, P represents the next state predicted by the prediction module in the risk response module, and P represents the actual environmental state transition distribution, that is, given the current state S t and action a t The next state S that the environment actually produces. t+1 The probability distribution, a t This indicates the action performed at time step t, which is based on the current policy on state S. t The decision made, S t Indicates the current state.

[0037] This invention also provides a multi-modal vehicle cyber-physical system multi-dimensional safety perception system, comprising:

[0038] The mapping module is used to map the multidimensional security requirements of the vehicle cyber-physical system onto multimodal information source data and establish mapping relationships.

[0039] The data acquisition module is used to query and acquire multimodal information source data corresponding to user security needs based on the mapping relationship.

[0040] The perception module is used to input the collected multimodal information source data into the multimodal fusion perception model to obtain the security perception result output by the multimodal fusion perception model, which is obtained through unsupervised learning;

[0041] The response module is used to input the security perception result into the risk determination module to obtain the risk score output by the risk determination module, and input the risk score into the risk response module based on the policy network to generate the optimal risk response strategy. The risk response module is obtained through reinforcement learning.

[0042] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-modal vehicle cyber-physical system multi-dimensional safety perception method according to any one of the above when executing the program.

[0043] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the multi-modal vehicle cyber-physical system multi-dimensional safety perception method according to any one of the above.

[0044] The application further provides a computer program product, which includes a computer program, wherein the computer program is executable on a processor to implement the multi-modal vehicle cyber-physical system multi-dimensional safety perception method according to any one of the above.

[0045] The multi-modal vehicle cyber-physical system multi-dimensional safety perception method and system provided by the application can enhance the precision of safety perception and improve the comprehensive safety guarantee capability of vehicles in complex environments by performing multi-modal and multi-dimensional safety perception for intelligent connected vehicles through multi-dimensional safety fusion mapping, multi-modal data acquisition, multi-modal unsupervised learning and reinforcement learning from the overall perspective of vehicle cyber-physical system safety. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0047] Figure 1 is one of the flowcharts of the multi-modal vehicle cyber-physical system multi-dimensional safety perception method provided by the application;

[0048] Figure 2 is the system framework diagram of the multi-modal vehicle cyber-physical system multi-dimensional safety perception method provided by the application;

[0049] Figure 3 is the second flowchart of the multi-modal vehicle cyber-physical system multi-dimensional safety perception method provided by the application;

[0050] Figure 4 is the structural diagram of the multi-modal fusion perception model in the multi-modal vehicle cyber-physical system multi-dimensional safety perception method provided by the application;

[0051] Figure 5 is the structural diagram of the multi-modal vehicle cyber-physical system multi-dimensional safety perception system provided by the application. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of, rather than all of, the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0053] In view of the above problems in the prior art, the present application provides a multi-modal vehicle information physical system multi-dimensional safety perception method and system. The method comprehensively considers three types of core safety requirements, i.e. functional safety, information safety and expected functional safety, constructs a unified safety requirement representation through the intersection thereof, and solves the problems such as single data source, lack of multi-modal collaboration and unsupervised learning capability in the current safety perception system.

[0054] To this end, the present application mainly includes the following four parts:

[0055] (I) Multi-dimensional safety fusion mapping: the present application first analyzes the three types of safety requirements in the vehicle information physical system, i.e. functional safety, information safety and expected functional safety, identifies the internal relationship and risk intersection thereof, and realizes the unified modeling and expression of the three types of safety requirements through the construction of a multi-dimensional safety fusion mapping mechanism. The mapping mechanism maps the safety requirements at different levels to a unified feature space based on a knowledge graph, a risk ontology or a vector embedding method, forms a structured multi-dimensional safety requirement vector, and uses the vector as a guide index for the subsequent data acquisition and learning system to provide input for the perception and fusion of multi-modal data.

[0056] (II) Multi-modal data acquisition: a distributed acquisition device is used to acquire multi-source heterogeneous data such as safety-related sensor data, communication information, operation logs, control commands and the like in the vehicle system, and to perform standardized preprocessing, including the following data:

[0057] ① Physical layer data: vehicle speed, acceleration, steering wheel angle, lane deviation, radar / camera data, etc.

[0058] ② Information layer data: vehicle Ethernet communication logs, CAN bus communication data, control commands, diagnostic data, etc.

[0059] ③ Functional layer data: system operation state, control strategy log, function module health state, etc.

[0060] ④ Human-machine interaction layer data: driver behavior data, language and touch data, system interaction feedback data, driver style data, etc.

[0061] The collected data is from multiple sensor nodes, has the characteristics of modal heterogeneity, dimension inconsistency, and time sequence asynchronization.

[0062] (iii) Multimodal unsupervised learning: to realize the safe perception fusion of multimodal data under the condition of no label, the application proposes a multimodal unsupervised learning framework. The framework uses a large model architecture to jointly model different modal data, and designs a joint Loss function to realize the collaborative representation and anomaly perception of different modal data in the feature space.

[0063] (iv) Reinforcement learning system: the application further introduces a reinforcement learning mechanism to construct an intelligent safety perception system with decision-making ability. In the absence of explicit label data, the system uses self-supervised signals to construct an intrinsic reward function, and learns the optimal response strategy under different safety states through model-based reinforcement learning method.

[0064] The application provides a multimodal vehicle cyber-physical system multi-dimensional safety perception method. Figure 1 The application provides a multimodal vehicle cyber-physical system multi-dimensional safety perception method.

[0065] Step 101, mapping the multi-dimensional safety requirements in the vehicle cyber-physical system to the multimodal information source data, and establishing a mapping relationship;

[0066] Step 102, querying and collecting the multimodal information source data corresponding to the user safety requirements according to the mapping relationship;

[0067] Step 103, inputting the collected multimodal information source data into a multimodal fusion perception model to obtain a safety perception result output by the multimodal fusion perception model, wherein the multimodal fusion perception model is obtained through unsupervised learning;

[0068] Step 104, inputting the safety perception result into a risk judgment module to obtain a risk score output by the risk judgment module, and inputting the risk score into a risk response module based on a policy network to generate an optimal risk response strategy, wherein the risk response module is obtained through reinforcement learning.

[0069] Preferably, the risk response module adopts a policy network construction based on deep reinforcement learning, and can be selected from algorithms such as Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), or Soft Actor-Critic (SAC). The module is trained through the interaction process with the system environment, and learns the optimal response strategy to be taken under different risk score conditions.

[0070] The policy network takes the risk score, safety perception results, and historical response information as input, and outputs risk mitigation actions for the current system state. The actions include but are not limited to communication channel closure, service isolation, controller restart, security alarm triggering, traffic speed limit, and other policy measures. The training goal of the policy network is to achieve the optimal risk control effect under the premise of ensuring system stability, thereby improving the response efficiency and intelligent level of the vehicle system when facing multi-dimensional safety threats.

[0071] As shown in Figure 2 The system framework of the multi-modal vehicle cyber-physical system multi-dimensional safety perception method generally includes four main modules: multi-dimensional safety fusion mapping module, data acquisition module, multi-modal unsupervised learning module, and unsupervised learning system module, which together constitute a multi-dimensional safety perception method system for intelligent connected vehicles. The complete flowchart is shown in Figure 3 .

[0072] The multi-dimensional safety fusion mapping module faces the functional safety (ISO 26262), information security (ISO 21434), and intended functional safety (ISO 21448) three-dimensional safety standards, establishes a unified safety modeling structure, and mainly includes the following steps:

[0073] 1.1 Extract the core demand elements in the three types of safety standards, and perform semantic normalization and reconstruction to form a general safety index set aligned across standards.

[0074] 1.2 Based on the vehicle cyber-physical system architecture, establish mapping relationships for data in the functional domain (controller, sensor, actuator), information domain (vehicle network, vehicle-cloud communication), and interaction domain (human-machine interface), and form a safety event influence causal chain model.

[0075] 1.3 Label and map safety requirements at each information source data level, such as associating CAN message abnormalities with functional safety failures, and associating network traffic changes with information attacks.

[0076] The output of this module is a structured multi-dimensional safety representation model, serving as the semantic basis for subsequent perception system modeling and analysis.

[0077] The data acquisition module is used for synchronous acquisition and preprocessing of multi-source and multi-modal data involved in vehicle operation, mainly including the following data types:

[0078] 2.1 Physical data: including vehicle speed, braking state, steering angle, acceleration, inertial navigation information, radar point cloud, camera image, etc.; the acquisition method depends on the sensor interface standard (such as CAN, Ethernet, video stream, etc.).

[0079] 2.2 Information data: including T-Box communication data, Ethernet / 5G vehicle communication data packet, OTA update log, in-vehicle network topology change, etc.

[0080] 2.3 Safety data: including intrusion detection system (IDS) log, security policy response record, abnormal alarm record, etc.

[0081] 2.4 Human-machine interaction data: including HMI operation log, driver behavior information, voice interaction instruction, etc.

[0082] All data are aligned using a unified timestamp and unique identifier, and support caching, denoising and compression mechanisms to facilitate subsequent fusion analysis.

[0083] The multi-modal unsupervised learning module is based on large-scale neural network models (such as Transformer, Autoencoder, GraphNet, etc.) to realize deep feature extraction and fusion of multi-modal data, and combines unsupervised learning methods to complete potential risk perception.

[0084] The self-learning mechanism does not rely on manual annotation during training, and is optimized and updated through internal self-supervised signals, to mine potential anomalies or implicit risk factors in data.

[0085] The intermediate perception results output by the multi-modal fusion perception model are: such as abnormal risk features, modal fusion feature maps, multi-dimensional abnormal scores, etc., for use by the subsequent strategy risk judgment module.

[0086] The reinforcement learning system module, as the output end of the system, is used to combine unsupervised learning results with risk assessment mechanisms to form safety level scores and auxiliary decision-making basis.

[0087] The front-end multi-modal fusion perception result is input to a risk judgment module, and a risk score or level label is output based on a preset risk level system (such as ASIL levels defined in ISO26262, or functional safety levels S1-S4). The system can normalize the abnormality degree of each modal data using an abnormality score function or entropy calculation method to generate a unified risk score.

[0088] A risk response module based on a policy network is constructed, and an unsupervised reinforcement learning mechanism is introduced for policy optimization and response generation. In the absence of explicit labels or reward signals, the system constructs an intrinsic reward function through self-supervised methods to drive the agent to learn the optimal risk response strategy.

[0089] The embodiment enhances the accuracy of safety perception and improves the comprehensive safety protection capability of vehicles in complex environments by performing multi-modal, multi-dimensional safety perception for intelligent connected vehicles through multi-dimensional safety fusion mapping, multi-modal data acquisition, multi-modal unsupervised learning, and reinforcement learning from the overall perspective of vehicle information physical system safety.

[0090] On the basis of the above embodiment, the multi-dimensional safety requirements in the embodiment include multiple requirements in functional safety, information security, and expected functional safety.

[0091] On the basis of the above embodiment, the multi-modal information source data in the embodiment includes multiple data in physical layer data, information layer data, functional layer data, and human-computer interaction layer data.

[0092] On the basis of the above embodiment, as shown in Figure 4 The multi-modal fusion perception model in the embodiment includes an embedding layer, an encoding layer, and an output layer connected in sequence.

[0093] On the basis of the above embodiment, the multi-modal fusion perception model in the embodiment performs unsupervised learning through the following loss function:

[0094] L=λ1·L recon +λ2·L align +λ3·L contrast

[0095] Wherein, L is a loss function, L recon , L align and L contrast represent the auto-encoding reconstruction loss of multi-modal information source data, the feature alignment loss between multi-modal information source data, and the contrast perception loss between multi-modal information source data, respectively, λ1, λ2 and λ3 are corresponding weight coefficients, respectively weighing the contribution of the three types of Loss to the total target, and the parameter specification is shown in Table 1.

[0096] Table 1: Explanation of basic parameters

[0097]

[0098] The multi-modal unsupervised learning module includes large model construction and loss function design. The large model construction designs a multi-modal input structure, supports parallel processing of images, time series, and structured data, and fuses a graph neural network and an attention mechanism to improve the interaction ability between modalities. The loss function design designs various unsupervised loss functions according to the consistency between features of each modality, spatial-temporal context dependency, and self-encoding reconstruction error.

[0099] On the basis of the above embodiment, in the embodiment, each modality data enters the corresponding self-encoder, and outputs the reconstructed data. The self-encoding reconstruction loss L recon is the mean square error function between the original input and the reconstructed output, and the calculation formula is:

[0100]

[0101] wherein x i represents the information source data of the i-th modality of the original input (such as the number of physical layers

[0102] According to the encoding layer, represents the output of the information source data of the i-th modality after being reconstructed by the encoding layer, and N is the number of modalities. This loss is used to maintain the integrity of the representation of each modality and ensure that the model does not ignore any modality features.

[0103] The feature alignment loss L align between the multi-modal information source data is used to pull the vector embedding results of different modalities in the common feature space closer, and uses the maximum mean difference loss, and the calculation formula is:

[0104]

[0105] wherein f i and f j respectively represent the shared representation vectors output by the i-th and j-th modalities of the information source data after the embedding layer. This term encourages different modalities to have consistent understanding of the same safety state and improves the interoperability of the representation.

[0106] The contrast perception loss L contrast between the multi-modal information source data is used to enhance the recognition ability of the model to the abnormal points. In the unsupervised scene, the “normal- abnormal” distinguishing ability can be enhanced through the contrast learning method. The InfoNCE contrast loss in the SimCLR style can be used, and the calculation formula is:

[0107]

[0108] where z i represents the output of the i-th sample after the embedding layer, represents the output of the i-th sample after the embedding layer, i similar samples, sim(·) represents a vector similarity function, usually cosine similarity, τ represents a temperature coefficient, and M represents the number of samples used in the unsupervised learning. In the unsupervised scenario, there is usually no abnormal type label, so the similar samples are obtained by pre-computing feature similarity. This term can improve the system's ability to distinguish feature distributions under different security states, which is beneficial to subsequent security level evaluation.

[0109] On the basis of the above embodiment, the risk response module in this embodiment performs reinforcement learning through the following reward function:

[0110] r t int = α·r t rep + β·r t anomaly + γ·r t info

[0111] where r t int represents the final reward function, r t rep represents the error-driven reward, r t anomaly represents the anomaly monitoring-driven reward, r t info represents the state confidence difference reward, α, β, and γ represent the corresponding weight coefficients, and the default values are 0.5, 0.3, and 0.2, respectively. This function comprehensively considers the state representation confidence, the abnormal degree, and the model prediction deviation, and can effectively guide the system to explore high-value states and optimize strategies.

[0112] On the basis of the above embodiment, the error-driven reward r t rep in this embodiment is calculated according to the following formula:

[0113]

[0114] where f θ (·) represents the representation of the current state s t in the risk response module by the encoder, represents the reconstruction result of the current state s t in the risk response module by the prediction module, and the current state s t is the current risk score.

[0115] The error-driven reward function is to measure the deviation between the current observation data and the learned representation inside the system, reflecting the uncertainty of the system to the current state. When the system has high uncertainty to the current state (difficult to reconstruct), a higher exploration reward is given.

[0116] The abnormality monitoring-driven reward r t anomaly The calculation formula is:

[0117]

[0118] Wherein, x t represents the input multi-modal information source data, represents the reconstruction result of the multi-modal information source data by the prediction module in the risk response module, ζ is an abnormal threshold, and σ(·) is a ReLU activation function;

[0119] The abnormality monitoring-driven reward function is to use an unsupervised detector to evaluate whether the current state is abnormal, and the higher the reward is, the more abnormal it is, driving the system to actively explore potential risk areas. Encourage the system to focus on high-risk (difficult to predict) areas and improve the ability to identify abnormalities.

[0120] The state confidence difference reward r t info The calculation formula is:

[0121]

[0122] Wherein, D KL represents the KL divergence, represents the next state predicted by the prediction module in the risk response module, P represents the real environment state transition distribution, that is, the probability distribution of the actual next state S t generated by the environment given the current state S t and action a t+1 , a t represents the action performed at time step t, which is the decision made by the current policy for state S t , and S t represents the current state.

[0123] The state confidence difference reward function is to quantify the uncertainty of the system in predicting state transitions, improving the state modeling capability. The more uncertain the state prediction is, the higher the reward is, driving the model to improve the generalization ability.

[0124] The system has a real-time closed-loop feedback mechanism, which can dynamically correct the perception model and response strategy based on vehicle operation data and actual risk conditions. By introducing self-supervised evaluation indicators and reward feedback mechanisms, the model perception ability and strategy behavior are continuously optimized to support long-term evolution and robustness improvement of the system, and an adaptive closed-loop control architecture of "perception - decision - feedback" is constructed.

[0125] In summary, the present application realizes multi-modal and multi-dimensional safety perception for intelligent connected vehicles through the system cooperation of the four modules of "safe modeling - data collection - feature learning - risk assessment". The technical path covers the whole process of data - model - strategy, and has high scalability, universality and engineering implementation value.

[0126] The multi-modal vehicle cyber-physical system multi-dimensional safety perception system provided by the present application is described below, and the multi-modal vehicle cyber-physical system multi-dimensional safety perception system described below can be correspondingly referred to the multi-modal vehicle cyber-physical system multi-dimensional safety perception method described above.

[0127] As shown in Figure 5 The system includes a mapping module 501, a collection module 502, a perception module 503 and a response module 504, wherein:

[0128] The mapping module 501 is used to map the multi-dimensional safety requirements in the vehicle cyber-physical system to the multi-modal information source data, and establish a mapping relationship;

[0129] The collection module 502 is used to query and collect multi-modal information source data corresponding to user safety requirements according to the mapping relationship;

[0130] The perception module 503 is used to input the collected multi-modal information source data into the multi-modal fusion perception model to obtain the safety perception result output by the multi-modal fusion perception model, and the multi-modal fusion perception model is obtained by unsupervised learning;

[0131] The response module 504 is used to input the safety perception result into the risk judgment module to obtain the risk score output by the risk judgment module, and input the risk score into the risk response module based on the strategy network to generate the optimal risk response strategy, and the risk response module is obtained by reinforcement learning.

[0132] This embodiment realizes multi-modal and multi-dimensional safety perception for intelligent connected vehicles through multi-dimensional safety fusion mapping, multi-modal data collection, multi-modal unsupervised learning and reinforcement learning from the overall perspective of vehicle cyber-physical system safety, enhances the accuracy of safety perception, and improves the comprehensive safety protection capability of vehicles in complex environments.

[0133] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal vehicular cyber-physical system multi-dimensional safety perception method, characterized in that, The method comprises the following steps: mapping multi-dimensional safety requirements in a vehicle cyber-physical system to multi-modal information source data to establish a mapping relationship; querying and collecting multi-modal information source data corresponding to user safety requirements according to the mapping relationship; inputting the collected multi-modal information source data into a multi-modal fusion perception model to obtain safety perception results output by the multi-modal fusion perception model, the multi-modal fusion perception model being obtained through unsupervised learning; inputting the safety perception results into a risk judgment module to obtain risk scores output by the risk judgment module, and inputting the risk scores into a risk response module based on a policy network to generate an optimal risk response strategy, the risk response module being obtained through reinforcement learning.

2. The multi-modal vehicle cyber-physical system multi-dimensional safety perception method of claim 1, wherein, The multi-dimensional safety requirements include multiple requirements in functional safety, information safety and expected functional safety.

3. The multi-modal vehicle cyber-physical system multi-dimensional safety perception method of claim 1, wherein, The multi-modal information source data includes multiple data in physical layer data, information layer data, functional layer data and human-computer interaction layer data.

4. The multi-modal vehicle cyber-physical system multi-dimensional safety perception method of claim 1, wherein, The multi-modal fusion perception model comprises an embedding layer, an encoding layer and an output layer connected in sequence.

5. The multi-modal vehicle cyber-physical system multi-dimensional safety perception method of claim 4, wherein, The multi-modal fusion perception model is unsupervised learned through the following loss function: L = λ1•L recon + λ2•L align + λ3•L contrast wherein L is a loss function, L recon , L align and L contrast respectively represent the auto-encoding reconstruction loss of the multi-modal information source data, the feature alignment loss between the multi-modal information source data, and the contrast perception loss between the multi-modal information source data, and λ1, λ2 and λ3 are corresponding weight coefficients.

6. The multi-modal vehicle cyber-physical system multi-dimensional safety perception method of claim 5, wherein, Self-encoding reconstruction loss L of multi-modal information source data recon The calculation formula is: wherein x i represents the information source data of the i-th modality, represents the output of the information source data of the i-th modality after reconstruction by the encoding layer, and N is the number of modalities; feature alignment loss L between the multi-modal information source data align The calculation formula is: wherein f i , f j respectively represent shared representation vectors output by the embedding layer from the i-th and j-th modal information source data; Comparative perception loss L between multi-modal information source data contrast The calculation formula is: where z i denotes the output of the i-th sample after passing through the embedding layer, denotes the output of the i-th sample after passing through the embedding layer, i similar samples, sim(·) denotes a vector similarity function, τ denotes a temperature coefficient, and M denotes the number of samples used for the unsupervised learning.

7. The multi-modal vehicle cyber-physical system multi-dimensional safety perception method of claim 1, wherein, The risk response module is reinforced learned through the following reward function: r t int = a · r t rep + β · r t anomaly + γ · r t info where r t int represents the final reward function, r t rep represents the error-driven reward, r t anomaly represents the anomaly monitoring-driven reward, r t info represents the state confidence difference reward, and α, β, γ represent the corresponding weight coefficients, respectively.

8. The multi-modal vehicle cyber-physical system multi-dimensional safety perception method of claim 7, wherein, Error-driven reward r t rep The calculation formula is: wherein f θ (·) represents a reconstruction result of the current state s t by an encoder in the risk response module, represents a reconstruction result of the current state s t by a prediction module in the risk response module. Abnormality monitoring drive reward r t anomaly The calculation formula is: wherein x t represents the input multimodal information source data, represents the reconstruction result of the multimodal information source data by the prediction module in the risk response module, ζ is an abnormal threshold, and σ(·) is a ReLU activation function. State confidence difference reward r t info The calculation formula is: where D KL represents the KL divergence, represents the next state predicted by a prediction module in the risk response module, P represents the real environment state transition distribution, and is the probability distribution of the next state S t actually generated by the environment given the current state S t and action a t+1 at time step t, a t represents the action performed at time step t, which is a decision made according to the current policy for the state S t , and S t represents the current state.

9. A multi-modal vehicular cyber-physical system multi-dimensional safety perception system, comprising: The method comprises the following steps: a mapping module for mapping multi-dimensional safety requirements in a vehicle cyber-physical system to multi-modal information source data to establish a mapping relationship; a collection module for querying and collecting multi-modal information source data corresponding to user safety requirements according to the mapping relationship; a perception module for inputting the collected multi-modal information source data into a multi-modal fusion perception model to obtain safety perception results output by the multi-modal fusion perception model, the multi-modal fusion perception model being obtained through unsupervised learning; a response module for inputting the safety perception results into a risk judgment module to obtain risk scores output by the risk judgment module, and inputting the risk scores into a risk response module based on a policy network to generate an optimal risk response strategy, the risk response module being obtained through reinforcement learning.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the multi-modal vehicle cyber-physical system multi-dimensional safety perception method according to any one of claims 1 to 8 when executing the program.

Images